Changelog¶
Unreleased¶
Added¶
- Per-contig QC columns. Every command that stores the contact-quality
block —
create pairs,create cool,collect,qc --store,reassign— now also writesqc_intra_contig_shortandqc_intra_contig_longto/contigs(int64, orfloat64on a map with float counts), from the same tally: read positions forcreate pairs, bin midpoints for a pixel tally. They are the two categories that do not depend on the genome assignment, counted contig by contig; the/genomescolumns of the same names are their per-genome sums. The block'sversionstays2: no existing key changes meaning. Like the/genomesones, the names are reserved (a user table column of that name is stored asuser_qc_intra_contig_short),binnifyandmergedrop them, andannotatecannot drop or replace them. Python:ContactQC(..., n_contigs=),ContactQC.contig_table(),ContactQC.set_intra_contig(),metacooler.ops.qc.tally_map()(the pixel pass ofqc, with the bins' genomes as an option), and aper_contigargument tostore_qc. annotate --unmatched warn|ignore|error(API:annotate(..., unmatched=)): what to do with the rows of a genomes or contigs table that name no genome or contig of the map.warn, the default, keeps the warningannotatealways gave;ignoresays nothing about them (a debug line);errorrefuses the tables before anything is written. A table that matches nothing at all is still refused in every case.keep_partialon every writer that stages its output —from_pairs,from_cool,collect,reassignandannotate. When the write fails,keep_partial=Falseremoves<output>.partial, whatever stopped the run (KeyboardInterruptincluded), and logs that at INFO, where the default,True, leaves it for inspection and logs at ERROR as before. For callers that stage their own output; see A failed write.from_pairsnow stages its.ps.tsvtable with the map and moves it into place just before it, the previous table set aside until the map is in place and put back if the map cannot be moved: written after the map was in place, a failure there left the new map behind. Once the map is in place the set-aside table is removed, and one that cannot be is left, with a warning, not put back: the new map and table match, and the run succeeds unless it was interrupted. A<name>.ps.tsv.previousleft behind, by that failure or by a run killed during the moves, makesfrom_pairsrefuse to start, before it reads anything, until someone checks which table matches the map (a file of the user's own of that name is refused alike: rename it).reassignandannotateare now in the I/O API reference.MetaCooler.iter_pixels(chunksize=, contig_ids=False)yields the pixel table in consecutive chunks of at mostchunksizerows, read with cooler's selector, so a caller can stream a large map withoutpixels()reading all of it at once.contig_ids=Trueaddscontig1_id/contig2_id, looked up throughindexes/chrom_offset. See Reading pixels in chunks.
Changed¶
reassignkeeps the QC block instead of recomputing it from the pixels. The per-contig intra-contig counts are carried over and summed per new genome, only the two inter-contig categories are re-measured from the pixels (exact at any bin size), and every other key —source,distance_threshold,binsize,n_pairs_read,n_pairs_discarded— is kept. Socreate pairswith a first table (or none), thenreassignwith a second, now stores the very block,/genomesand/contigsqc_*columns thatcreate pairswith the second table gives. It used to recompute the block from bin midpoints: on a map at one bin per contig that turned every intra-contig contact intointra_contig_short, and it lostn_pairs_read/n_pairs_discardedandsource="pairs"— on every map binned after creation, which is how MetaTOR works. A source without the per-contig columns (written by 0.2.0 or earlier), or whose columns no longer add up to its block, is still recomputed from the pixels, with a warning naming what is lost when the block was measured on read positions (an INFO line otherwise).--no-qcstill drops the block, and its two/contigscolumns with it. The new block is measured from the source's pixels before anything is written, and written with the rest of the map, so a run stopped meanwhile —--inplaceincluded — leaves the source's block and per-contig counts as they were; it used to be stored after the relabelling.- In
/contigs, only the two QC columns go with the QC block.mergedrops them (it copied the first input's/contigsverbatim), asbinnifydoes, andreassignreplaces or drops them with the block, whether it moves the contigs or not. A user column that merely starts withqc_(qc_pass, from a--contigstable) is contig metadata, kept by every writer:binnifyand a re-groupingreassignused to drop everyqc_*column of/contigs.annotate --dropnow accepts it, andmergereports it when inputs differ, as for any metadata column; both treated everyqc_*column of/contigsas metacooler's. mergerefuses the same map given twice.merge A Asummed every contact ofAwith itself and exited 0. Inputs are now identified by file (device and inode) and group, so an alias or a hard link of a listed input is refused too; two different entries of one.mmetacoolstill merge.create pairs' refusal of a pairs file given twice is keyed the same way, so a hard link no longer slips past it.- Faster genome and name selections on large maps. On a map whose genomes
are not stored contiguously (
--contig-order alphaorfasta, or areassign --layout keep), each of a genome's bin ranges was read with cooler's selector naming every contig of the map: at 200 k contigs,bins_for_genomeof a 97-contig genome took 7.7 s (now 0.3 s) andextract --genomes15 s (now 1 s). Lists of names (dump/extract--contigsand--genomes,show) scanned the contigs table once per name: 5 000 contigs took 7 s, now 0.16 s. Output is unchanged.
Fixed¶
reassignno longer crashes on a non-objectqcmetadata value when it recomputes the block; it measures at the default distance threshold, asqc --storedoes without--threshold.- Type checkers see what every top-level name is.
metacooler.from_pairs,metacooler.MetaCoolerand the rest of__all__are loaded lazily through the module's__getattr__, so pyright and ty typed each of them asobjectand flagged every call in client code as not callable, while mypy typed each asAny(import-untyped). Anif TYPE_CHECKING:block inmetacooler/__init__.pynow names each one at its source for type checkers only: nothing more is imported at run time, soimport metacoolercosts what it did. The package also ships apy.typedmarker, without which mypy does not read an installed metacooler at all (import-untyped). reassignno longer claims that only provenance differs when no contig moves although the output differs in more: a--genomestable, a re-grouped layout or a newcontig-ordering,--no-qc,--metadatakeys, or the QC block it recomputes. The message names each, and is an INFO line rather than a warning when the caller asked for one of them; a table that moves nothing, and nothing else asked for, still warns.- A left-over
<output>.partialgiven as the input is no longer deleted. Every writer that stages its output removes a stale<output>.partialbefore it starts, and it did so even when that file was its input:reassign out.metacool.partial -o out.metacool, rescuing a partial run, deleted the map before reading it (as didannotate,collect,create coolandcreate pairsgiven such an input). It is now refused before anything is read, as an output that is an input always was. - A
"in a user table no longer swallows rows. Genomes and contigs tables were read with CSV quote handling, so"drafton one row andcurated"a few rows later became one field spanning every row between them: those rows vanished without a warning (increate, their contigs went to__unbinned__). Every user table, and the bins BED, is now read with quoting off; a quote is stored verbatim. networkno longer wraps large integer weights. Edge weights were summed in the storedint32, so a node pair with more than 2³¹ contacts (the__unbinned__self-edge of a deep library) printed a negative weight. They are summed inint64.collect -fcan no longer delete one of its own inputs, nor the previous output when the run then fails:collect A.cool B.cool --genomes g.tsv -o g.tsv -fdeletedg.tsvbefore reading it. Every command now names its inputs to the output guard (checked by a test over everycmd_*.py).- A user's
qc_*genome column survives. Everyqc_*column of/genomesis metacooler's, but only the five exact QC names were renamed on the way in, so aqc_passfrom a--genomestable was stored as is and then dropped by the nextreassign,binnifyormerge. It is now stored asuser_qc_pass, with the usual warning. - A column named
.or..(which HDF5 cannot store) is stored ascolNwith a warning, instead of failing aftercreate's whole pairs pass. annotate --inplaceis all or nothing. It swapped each rebuilt table in as soon as it was built, so a failure on/contigsleft/genomesalready edited, no history recorded and the scratch group behind. Every table is now built before any is swapped in, and a failure leaves the map as it was.- Pairs files: upper-case suffixes and blank header lines. The codec was
chosen by pyarrow's case-sensitive suffix match, so
R1.PAIRS.GZwas parsed as raw gzip bytes into an empty map; suffixes are now matched case-insensitively, and.zstdis read as.zst. A blank line inside the#header made the read fail on the first pair (and hid the#chromsizelines after it); blank lines are now part of the header. - One contigs table gives one column type, whichever command stores it.
createtypes a metadata column over every row of the table, so oneN/A— even on a row naming a contig the map lacks — keeps it text;collectandannotatetyped only the matched rows and stored the same column as numbers, andreassignre-typed a stored text column the same way. The table is now typed once, as read, by every command. collectaccepts an input whose every contig is a single bin (a phage, a few short plasmids). cooler infers no bin size for it and calls it variable, but its bins are the grid of any bin size at least as long as its longest contig, so it joins such a collection instead of being refused.- Docs:
metacooler validate(which never existed) ismetacooler check;contact_qc,contact_probabilityandplot_psare in the API reference; thereassignandannotatehistory blocks of themetadataattribute are in the format specification;networkandaggregate--chunksizedefaults read 50 000 000, as they are.
0.2.0 - 2026-11-16¶
Breaking¶
- The default
--contigs-format(contigs) findscontig_nameandgenome_nameby name, wherever they are, instead of requiring them to be columns 1 and 2; likewise--genomes'genome_nameandcollect --contigs'contig_name. --contigs-format contig_genomekeeps a table's extra columns, ascol3,col4, …, like every other headerless spec; it used to drop them.- A genome named
-1in a contig table means__unbinned__, asNAandNDalready did: it is SemiBin2's label for unbinned contigs. mergecompares only what makes two maps the same map — contig names and lengths, the contig-to-genome assignment, genome sizes. Metadata columns may now differ between inputs: the output keeps the first input's, and each difference is a warning. Integer width no longer matters either (int32lengths fromcreate coolagainstint64ones).showtakes a positional input path. The required-f/--fileoption is gone;metacooler show FILE.metacool -o out.pngreplacesmetacooler show -f FILE.metacool -o out.png.-fnow means--force, as in every other command.binnifyrejects a target bin size that is not an exact multiple of the source bin size. A non-multiple target produced a systematic ±14 % checkerboard while advertising a uniform bin size. For variable-bin sources the median source bin width is used as the effective resolution and a target at or below it is refused.zoomifyrejects a resolution ladder that is not divisible by the base bin size, and stops early instead of emitting duplicate levels.dumperrors instead of silently ignoring incompatible combinations:--weight/--joinwith a non-pixels table,--genomes/--contigswith-t indexes,-owith-t all,--prefixwith a non-alltable.- An explicitly empty
--genomes ''/--contigs ''is rejected rather than silently disabling the filter. - A bare
.mmetacoolpath is rejected by every command that needs a single map, with a message listing the entry URIs.ls,tree, andattrsstill accept a collection. metadata add/get/remove/loadon a collection root is rejected; they used to write to the root, invisibly from every entry.create pairs --contig-order fastarequires a FASTA or a#chromsize:pairs header. It used to fall back to alphabetical order while still recordingcontig-ordering="fasta".--metadatafiles may not contain acommandskey (reserved for the provenance record).- The assembly name is stored in the standard
genome-assemblyHDF5 attribute, notassembly-name.infonow reports it, soassemblyreads"unknown"where it previously readnull. - Declining a
cp/mv/rmconfirmation prompt exits non-zero (it used to exit 0). attrsrejects a group given twice (URI group andGROUP_PATH); the URI group used to be silently discarded.show --genomesis a comma-separated/repeatable list, notnargs=2.--genomes MAG_01,MAG_02(or--genomes MAG_01 --genomes MAG_02) replaces--genomes MAG_01 MAG_02, bringing it in line with--genomeseverywhere else. The old space-separated form now fails with a usage error, because the second name would fall through to the positionalMETACOOL_PATHslot. An explicitly empty--genomes ''is rejected, as on every other command.show --genomesis symmetric for any number of names, including two: every name goes on both axes, so the figure holds all the intra- and inter-genome blocks between them.--genomes A,Bused to mean "A on the rows, B on the columns" — for that, use-r A -r2 B, now the only asymmetric form.show --genome(singular) is gone.--genomes MAG_01with one name is exactly the same thing, and the near-identical pair meant--genome MAG_01,MAG_02looked for a genome literally namedMAG_01,MAG_02.showwas also the only command in the CLI with a singular form.- Contig and genome names must be ASCII.
createrefuses a non-ASCII name before it bins anything,validate()refuses one on read, andcheckreports it./contigsis cooler's/chroms, and cooler decodes those columns with an ASCII.astype("U"), so an accented contig name — or an accented value in any extra/contigscolumn, which cooler reads just the same — produced a file that passed every metacooler check and then raisedUnicodeDecodeErrorincooler.Cooler(), and therefore inMetaCooler(). Genome names hit the mirror image of the same assumption on the way out, whereaggregate --level genomewrites them into a.coolas chrom names (UnicodeEncodeError). Extra/genomesmetadata columns are unaffected and still take any UTF-8: cooler never reads that group. The limitation is upstream, but writing a file metacooler itself cannot open breaks the one invariant the format is built on, so it is now a named refusal instead of a crash three commands later. -
A
--contigsor--genomestable whose header and data rows disagree on the number of fields is rejected.pandas.read_csvused to resolve the mismatch silently, and in opposite directions depending on which side was wider — a wider header pads the trailing columns, wider rows absorb the surplus leading fields into the row index — so in both cases every value was stored under the name of a neighbouring column. A trailing tab on the header line is still accepted, and no longer becomes an emptyUnnamed: Ncolumn in/genomesor/contigs. -
psfrom a binned map counts opportunities in position pairs. Each bin pair of widthsw_i,w_jcontributesw_i × w_j(the diagonalw (w + 1) / 2) — the unit the--pairspath already used — son_bin_pairs,avg_contactsandprobchange on the pixel path and now mean the same thing on both. Counting bin pairs weighed the diagonal as a full opportunity though it holds half as many position pairs, halving the first point.probalso no longer depends on--bins-per-decade: it was normalised over the full width of every log bin, though a binned map only realises separations at bin-distance steps, so the pixel/pairs ratio at 10 kb drifted from 1.07 to 1.52 as the grid got finer.distnow sits on the separations a distance bin can actually hold ([1, 2)is plotted at 1, not 1.41). - QC counts a contact between two different
__unbinned__contigs asinter_genome. The unbinned bucket is not a genome, and treating it as one inflatedintra_genome_fractionby every contact among unassigned contigs. Themetadata.qcblock is nowversion: 2. create coolrefuses asquarecooler. Both triangles were folded into the upper one and summed, doubling every off-diagonal count.create pairsdrops pairs at or past their contig's end instead of folding them into the last bin (ascooler cloaddoes); they are counted in the discard report and inqc.n_pairs_discarded.- Zero-width bins in
--bins, and a--drop-unassignedthat removes every contig, are refused before binning rather than failing after the whole pairs pass. create pairsandcreate coolrefuse a--contigstable that names no contig of the assembly, or has no row, before anything is binned. Every contig used to land in__unbinned__, and the command exited 0 with nothing logged above INFO — the signature of the wrong table, or of the wrong--contigs-format(a leading ID column read as1,2takes the IDs for contig names). The error shows a few names from each side and points at--contigs-format. A table naming contigs the assembly does not have is still accepted, now with a warning naming up to 10 of them. See The contig table.--metadatafiles may not contain aqckey, whichcreateoverwrote.validate()is stricter: it refuses a repeated(bin1_id, bin2_id)pixel, a/chromslink that does not resolve to the map's own/contigs, and a table member that is a group rather than a column.stack,create_mmetacoolandadd_entryvalidate their sources, so a plain.coolor a whole.mmetacoolis refused, and entry names containing/are rejected (also bycp/mv).- An output that is one of the command's inputs is refused, even with
-f.binnify a.metacool -o a.metacool -fdeleted the input, then failed. - Distance options must be finite, non-negative and at most 2^53 bp
(
qc --threshold,ps --min-distance/--max-distance,create --qc-threshold);infand1e30used to escape as tracebacks.
Added¶
metacooler annotateadds, replaces or drops metadata columns of/genomesand/contigsin an existing map (APImetacooler.annotate): CheckM2, GTDB-Tk or depth tables that arrive after the map was built. Rows are matched by name; the assignment, bins and pixels are never touched, so it is cheap enough to run--inplace. Columns get the names, values and types the same table would have givencreate. A changed column needs--replace(--missing keep|nachooses between a patch and an authoritative table); derived columns cannot be dropped or replaced; a contig table'sgenome_nameis checked, never applied; non-ASCII/contigsvalues are refused. A bare.mmetacoolpath annotates every entry.- One grammar for table columns.
--contigs-format CONTIG,GENOME[,noheader]takes 1-based positions or header names (Name,Final_bin,2,5), and aliases for common binners:metator,dastool,binette,semibin,vamb(besidescontigsandcontig_genome).collectgets a--contigs-formatof its own, for its per-contig metadata table (see its section). - A map with no binning yet:
--contigsis optional oncreate pairsandcreate cool. Without it the map is the cooler plus the genome layer, every contig in__unbinned__, ready forreassign(create cool --contig-order coolthen keeps the cool file's contigs, bins and pixels unchanged). To keep a table's other columns as contig metadata meanwhile,--contigs-format CONTIG,NAsays the table has no genome column; withreassign, the same spec unbins the contigs the table lists. --genomes-formatoncreate pairs,create cool,collect,reassignandannotate: which column of--genomesholds the genome names, so a CheckM2 (Name) or GTDB-Tk (user_genome) report needs no editing.-
create pairsreads several pairs files in one pass — one per Hi-C library of the same assembly: repeat--pairs(not comma-split, since a path may contain a comma), or pass a list asfrom_pairs(pairs=[...]); a single path works as before. The files are streamed back to back in the given order, so the map, the QC block and/genomesqc_*columns, the.ps.tsvP(s) table and the dropped-pairs report (n_pairs_read,n_pairs_discarded) are exactly those of the files' concatenation — which per-library maps joined bymergecannot give, sincemergedrops the QC block. Chunk folding, the sort strategy,--nproc(output identical to--nproc 1) and memory behaviour are unchanged; a chunk may span two files. The files'#chromsize:headers are united in file order; a file without one still works with--fastaor another file's header, and one contig given two different lengths is an error naming both files. The same file given twice, and a missing file, are refused before any pairs file is read. Thecommandsprovenance lists every--pairs. -
--nproconcreate pairsandzoomify(default 1, which keeps the serial behaviour).create pairs --nproc Nbins pairs batches on N threads while the file is read and decompressed; per-batch drop tallies, QC and P(s) are folded in file order, so every output is identical to--nproc 1(_PairsDropStats.unknown_nameskeeps insertion order for that). On a 200 k-contig, 30 M-pair library: 23.9 s → 20.3 s with 2 threads, ~20 s with 4 — one gzip stream is then the critical path.zoomify --nproc Nbuilds independent levels in N processes, each as soon as the level it is coarsened from exists, and appends them in ladder order:-r 1000,2000,5000,10000on the same map goes from 24.9 s to 15.6 s with 2 processes (peak memory per process unchanged, so the total roughly doubles). -
collect: zip single-genome cool files — one per species, strain, or MAG — into one.metacoolfile, one genome per input, with no contig-to-genome table to write. Inputs are.coolfiles or one resolution of an.mcooleach (E_coli.mcool::resolutions/1000); a genome is named after its file stem, or by--names. Every contig is renamed<genome>__<contig>, so thechromosomeorcontig_1that many assemblies share cannot collide — without stacking a second prefix on a contig that already carries its genome's, socollect → extract → collectis stable.--genomesdecorates the genome rows as increate;--contigsattaches per-contig metadata, naming a contig as in its input or as collected, with an optionalgenome_namecolumn that disambiguates a shared name and is checked rather than used to re-assign a contig;--metadata,--filter(includinggenomes.n_bins, since the bins come with the inputs),--assembly-name, and--qcwork as increate cool. Inputs must share one bin size, or all be variable-binned; a metacool, an.mcoolroot, and asquarecooler are refused with a message saying what to use instead. Every input occupies one contiguous block of bins, so under the default--contig-order cool(each input keeps its own contig order) the pixels are streamed into place with no sort at all, andextract --genomes Xgives back input X bin for bin and pixel for pixel.genome-then-lengthandgenome-then-alphare-sort only the inputs whose contigs they move, one at a time, falling back to cooler's external sort past 100 M pixels. The inter-genome blocks of a collected map are empty by construction — no contact between two inputs was ever observed — which the docs spell out. Exposed asmetacooler.collect. Seecollectand the new guide, Collecting single-genome cools. reassign: re-attribute the contigs of an existing map to genomes from a new contig-to-genome table — a re-binning, a curation pass, a merge of two MAGs — without re-binning a pair. Every count is kept;/genomesis rebuilt from the new assignment (genomes created and emptied, everygenome_idrenumbered,length/n_contigs/n_binsrecomputed). Under a genome-groupedcontig-orderinga moved contig breaks the layoutgenome_offsetandgenome_extentrely on, so--layout autore-groups such a file (bins permuted, pixels remapped and re-sorted, balancing weights carried) and only relabels any other, skipping the rewrite when the new grouping reproduces the old order;--layout keepnever moves a row and recordscontig-ordering="custom", a new valuevalidate()does not hold to a grouping. A relabel copies the file and rewrites four datasets, so it can also run--inplace.--missing keep|unbinnedmakes the table an incremental edit or authoritative; neither drops a contig. The QC block is recomputed (or, with--no-qc, dropped), per-genome metadata is carried with a warning naming the genomes whose contig set changed, and a table matching no contig is refused. Asquaremap is only ever relabelled, and acollected map's genomes, contiguous althoughcooldoes not promise it, are named in a warning if the new assignment scatters them. Exposed asmetacooler.reassign. Seereassign.qc: contact-quality statistics — where a library's contacts actually land. Every contact is classified into one of four categories (intra_contig_short/intra_contig_long, split at--threshold, default 10 kb;inter_contig_intra_genome;inter_genome), from which the headline 3D ratio (trans_contig_ratio, the share of contacts joining two different contigs) and the intra-/inter-MAG breakdown follow.create pairsandcreate coolnow compute the block during creation at essentially no cost — they are already streaming every pair or pixel — and store it inmetadata.qcplus five per-genomeqc_*columns of/genomes, includingqc_intra_genome_ratio, a per-MAG "how self-contained is this bin?" score. Pass--no-qcto skip it. The standalonemetacooler qccommand recomputes the same block from any existing file's pixel table (--table,--per-genome FILE.tsv,--store). The two sources agree on every total; only the intra-contig short/long split differs, because binning discards the sub-bin distances — which is whycreate pairsmeasures it on raw read positions and recordssource="pairs".binnifyandmergedrop the block and the columns rather than carry stale numbers onto a different pixel table, andmergeignoresqc_*when comparing input/genomestables, so two libraries over one assembly still merge. Seeqc.ps: P(s), contact probability as a function of genomic distance, as a TSV (-o) and/or a log-log figure (--plot), pooled or one curve per MAG (--by genome), with the usual--genomes/--contigs/--filterselection. Columns aregenome,dist,n_contacts,n_bin_pairs,avg_contacts,prob. Each log-spaced distance bin reports the contacts observed and the number of opportunities that could have produced one, so the decay is not confounded by the shrinking number of long-range pairs a contig offers, and a normalisedprobcolumn that integrates to 1 makes libraries of different depth comparable. Only intra-contig contacts take part — the ones a genomic distance is defined for. The bin-pair denominator is counted with a single globalsearchsortedper distance edge over a contig-folded key, and the position-pair one in closed form per contig, so neither costs a per-contig Python loop on a metagenome. Seeps.- P(s) at base-pair resolution, measured during creation. A curve read off
a binned map cannot resolve anything below one bin, which is precisely where
a metaHi-C library shows its undigested fragments and self-religation. So
create pairsnow accumulates a per-MAG curve from the raw read positions during the pass it already makes over the pairs, and writes it beside the output as<output>.ps.tsv(--no-psopts out,--ps-bins-per-decadesets the grid). No plot is drawn. Measured cost on 4 M pairs: none outside run-to-run noise — onesearchsortedand onebincountper chunk over the intra-contig rows, and a closed-form denominator computed once. ps --pairsrecomputes that fine-grained curve for a file created earlier, with a different grid or selection. The pairs file goes through the same readercreate pairsuses, so compression and column handling are identical by construction, and the output is byte-for-byte the tablecreatewrote. The map still supplies the contig→genome assignment, the contig lengths, and the selection.--filter, a repeatable predicate selector oncreate pairs,create cool,extract,dump, andshow, for choosing genomes and contigs by property rather than by name:--filter "genomes.length > 1Mb",--filter "contigs.length >= 10kb",--filter "genomes.Completeness > 90". One comparison per flag —> >= < <= == != =~ !~, withbp/kb/Mb/Gbsuffixes on numbers — and repeating the flag ANDs them. Thegenomes./contigs.prefix is mandatory, since both tables have alengthand aname. Any column of either table can be used, including the ones a--contigs/--genomesmetadata table added, plus a derivedcontigs.genomeholding the genome name. Names choose the candidates and filters narrow them, so--genomes MAG_01 --filter "contigs.length > 50kb"is an intersection, not a union. Because the column names in real MetaTOR tables contain spaces and dashes (Present Markers,GC-content), the first operator terminates the column name and no quoting or backticking is needed; the cost is that there is noor. Selecting nothing is an error that reports what each filter would have kept on its own, and an unknown column suggests the closest one. Increatethe filter runs before anything is tiled, so a dropped contig costs no bin, no pixel, and nocooler.createoverhead — see Filtering with--filter.check: an integrity report over the existing validators. Prints one line per check —readable,format,structure,dtypes,cooler-compat— instead of raising on the first problem, so every failure is visible in one pass, and repeats the per-map checks for each entry of an.mmetacool. Two checks go beyond what runs on every file open: the on-disk dtype kind of the 16 core columns (validatecasts on read, so a float64bins/startused to pass it and fail later inside cooler), and, under--deep, a full O(nnz) pixel scan that also reconcilesattrs['sum']with the stored counts. Exposed asmetacooler.check/metacooler.CheckResult.show --genomesaccepts any number of genomes, with one uniform rule: every name goes on both axes, so N names give all N against all N — intra and inter blocks alike — laid out in the order the names were given, with a heavier boundary line between blocks.show --contigs, the same selector in contig space: bare contig names, comma-separated or repeated, on both axes. It closes the one gap-rcould not — more than one contig on the same axis — and upgrades--level genome(where a contig has no row of its own) to--level contig. Mutually exclusive with--genomes.showfigures are titled and their axes labelled. The title names the selection and the level (MAG_01 vs MAG_02 — contig level), truncating a long genome list to(+N more); axis labels name the selection and its size (MAG_01 (12 contigs)).show --level binticks the axes in base pairs, not bin indices. A bin index starts at 0 even when the selection does not, so-r contig:20000-50000used to read as though it began at the start of the contig. A selection spanning several contigs has no single coordinate system, so there the ticks name the contigs, one at the first bin of each.-q/--quieton every command: suppresses INFO logging, keeps warnings and errors.-f/--forceondump,network, andshow, which previously clobbered their output silently.binnify --chunksizeto bound peak memory during pixel remapping.zoomify --temp-dirfor the intermediate per-level files; peak scratch use is now one level rather than a second copy of the pyramid.MetaCooler(..., validate=False)fast path for very large, known-good files.- Top-level exports:
metacooler.extract,.flatten,.merge,.binnify,.zoomify,.aggregate,.network,.dump_table,.validate, and.validate_mmetacooler. stackaccepts entry URIs and rejects duplicate--names.
Fixed¶
metadata addandmetadata removerefuse the keys metacooler writes itself —commands,qc, and thereassignandannotatehistories.metadata add -k commands --value '[]'wiped the provenance record, andmetadata remove -k commandsdeleted it, both exiting 0, although a--metadatafile was already refusedcommandsandqc.--metadatafiles andmetadata loadnow refuse the two history keys as well.create pairs --sort-strategy externalworks on large inputs with cooler < 0.12. Its pixel chunks reached cooler's unordered writer unsorted, and cooler before 0.12 indexes each chunk as if it were sorted. Once the merge took more than onemergebufepoch (20 M records), the output had out-of-order and duplicated pixels, andvalidate()rejected it at the very end of the run (bin1_offset disagrees with pixels/bin1_id), as it did on 30 M pairs over 200 k contigs in the benchmarks.create pairspast its in-memory budget underauto,create coolwith the external sort, andreassign's regroup path streamed unsorted chunks in the same way. Every unordered write now sorts each chunk that is not sorted already (one vectorised check for those that are).MetaCooler.pixels(join=True)names the contigs.chrom1andchrom2held contig ids (0,1, …):/bins/chromis a plain integer column in a metacool file, and cooler's join reports such a column as ids wherecooler.Cooler.bins()converts it to names. The coordinates are now joined fromMetaCooler.bins(). Passconvert_enum=Falseto get the ids.dump … | headno longer reports a failure. Closing the pipe early is the normal end of a pipeline, but the resultingBrokenPipeErroris anOSErrorand so was rendered in the same redError: [Errno 32] Broken pipebox as a bad input — after the rows had already been printed, which read as though the dump had failed. Every command now exits silently on a closed stdout with status141(128 + SIGPIPE), the same asyes | head, and no longer leaks anException ignored in: …from the interpreter's final flush.- Answering
yto acp/mvoverwrite prompt now actually overwrites;-fused to be the only way through. show --genome/--genomesare honoured at every--level, and the dense-matrix size guard applies at--level bintoo.- Counts are preserved as
int64throughextract,flatten, andmerge, so re-binned sources no longer truncate. .bgzpairs files are decompressed correctly instead of being parsed as raw binary into an empty file; pairs dropped for an unknown contig or a missing position are now counted and reported.mvrenames the HDF5 link in place instead of copying and deleting.- A
--binsBED with more than three columns (BED4/BED6) is read correctly instead of shiftingchrom/start/endone column to the left. Because contig lengths are derived from the bins frame itself, the shifted table was self-consistent and creation went ahead with contigs named after coordinates. -
createreports when the--contigsand--genomestables cover different sets of genomes: names assigned contigs but absent from the genomes table (kept, with derivedlength/n_contigs/n_binsand empty metadata) and names listed in the genomes table with no contigs (absent from/genomes). Both used to be silent, which made a/genomestable larger than the input look as though genomes had been invented. -
cooler ≥ 0.11 no longer brings back the quadratic per-contig write cost. cooler moved
write_binsintocooler.create._writer; metacooler's shims looked in the old module and silently did nothing, so on cooler 0.12 creating an 80 k-contig map took 13.9 s instead of 2.1 s, growing about 3× per doubling of contigs. mergeran cooler's writer without those shims (65 s → 3.1 s at 150 k contigs), floored float counts, lostbin-sizewhen every contig was a single bin, crashed on.mmetacoolentry URIs, and — in the CLI — put back the stalemetadata.qcblock it had just removed.create --forcekeeps the previous map (and its.ps.tsv) until the new one has been written and validated; it used to delete them before any work, so a failed run left nothing.- A contig table with its own
genome_idcolumn no longer sends every contig to__unbinned__, and the same collision in--genomesno longer crashes. - A
/in a--contigs/--genomescolumn name (e.g.reads/kb) is stored as_; it used to create a nested HDF5 group that neither cooler norMetaCooler()could open. --contigs-format I,J,noheaderwith extra columns no longer crashes after the whole pairs pass (they are stored ascol3,col4, …)..zstpairs (and.bz2with an integer--bins) are read; the header scans used their own openers and failed.- An empty pixel stream no longer crashes
createunder--sort-strategy external, norbinnify,zoomifyoraggregateon a zero-pixel map. - Fragment-level (
--bins file.bed) binning is one vectorised lookup instead of a loop over contigs (13.1 s → 4.7 s at 80 k contigs; identical output). aggregate --level genomerefuses a genome longer than 2^31 bp instead of writing a negative chromosome length.showkeeps float counts at contig/genome level, andshow --transform log10islog10(1 + count)as documented (single-count pixels used to render as empty).--filter!=/!~no longer match a missing text value.extract --splitstrips only a trailing.coolfrom-o, so-o sample.rep1and-o sample.rep2no longer collide.create coolkeeps the source'sgenome-assembly.MetaCooler.rootis the entry path for an.mmetacoolentry,info["sum"]keeps a fractional sum, andMetaCoolerhas arepr.- A
G:start-endrange on abinnify --level genomefile (where every contig is also a genome) selects the contig instead of being refused. qc --storeno longer crashes on a non-objectqcmetadata value.
Documentation¶
- The CLI reference gives editing its own section. "Create and edit" is
split into Create (
create,collect), Edit an existing map (reassign,annotateandmetadata, moved from Inspect, with a table of which one to use) and Combine and re-bin (merge,binnify,zoomify);metacooler --helpgroups the commands the same way. - The documentation site is restyled after doc.chromosomix.app. The
zensical theme keeps its layout but takes a new header, an in-column page
footer with previous/next cards (also reached with the ← / → keys), a
light/dark palette, framed code blocks and Starlight-style asides and tables
(
overrides/,docs/stylesheets/metacooler.css,docs/javascripts/metacooler.js). The home page is now the introduction and four entry tiles; its motivation, quick look, compatibility table and documentation map moved to a new Why metacooler? page, the on-disk layout to Concepts, and theqc/pscommands to the Quickstart. COMMANDS.mdremoved;docs/cli.mdis the single CLI reference._write_rebinned_metacool: wrongchrom_offsetcaused by callingnp.searchsortedon an unsortedcontig_namesarray. Replaced with an explicit contig-name → index mapping._write_rebinned_metacool: missingvalidate()call after write; corrupt files were not caught before being returned to the caller.
Changed¶
ops/aggregate.pycentralises the accumulation logic in module-level helpers (_aggregate_sparse/_aggregate_pixel_stream) instead of duplicating it across call sites.
Performance¶
- Genome and contig retrieval goes through the offset indexes. A genome
resolves to its contigs through
indexes/genome_offset(or, under afasta/alpha/cool/customordering,/contigs/genome_id), a contig to its bins throughindexes/chrom_offset, and a bin range to its pixel rows throughindexes/bin1_offset; those rows are then read with cooler's own bin and pixel selectors, and nothing scans the bin table or the pixels outside them.pixels_for_genome_pair,genome_bin_ids,genome_extent,contig_extent,bins_for_genome,bins_for_contig,dump -r/-r2(long and wide),dump -t pixels --genomes/--contigs/--filterandextractall use it. On a 200 k-contig map,dump -t pixels -r <contig>went from 3.0 s to 0.8 s (cooler: 1.0 s) andpixels_for_genome_pair(g, g)from 119 ms to 3.5 ms.genome_extentstill raises for a genome that is not contiguous.extractof asymmetric-uppermap now writes in order, without cooler's merge. New publicMetaCooler.genome_contig_ids()andMetaCooler.genome_bin_ranges(). - Opening a file runs only the structural checks.
MetaCooler()used to run the whole ofschema.validate(), O(n_bins), on every open (2.8 s on a 2 M-contig map). It now checks format attributes, required groups and columns, column lengths, the/chromslink, the count attributes and the length and endpoints of every offset index — O(n_columns) reads.schema.validate(uri, full=False)runs that subset;metacooler validate,schema.validate()and every writer still run the full check. A file cooler cannot decode (non-ASCII names) still fails the open with the namedMetaCoolerFormatError.pixels_for_genome_pairnow returns every pixel column, typed, when nothing matches (it returned an untyped three-column frame). - Pixels are aggregated on one
int64key. Every pixel metacooler builds itself is summed throughio/_utils.aggregate_pixels:(bin1, bin2)folds intobin1 << 32 | bin2, which is sorted (a stable, run-detecting sort when the inputs are already sorted frames) and reduced withnp.add.reduceat, instead of a pandasgroupbyover two columns.create pairsbins each chunk by sorting its keys and counting runs, and its in-memory sort frees each frame as soon as it is keyed. Count dtypes are kept exactly as before.aggregateandnetworkaggregate their chunks the same way, and every caller letsaggregate_pixelschoose the sort: a stable sort (which merges existing runs) when the keys come mostly in order, as a genome-grouped remap does, quicksort when they are scrambled. - Pair contig names are dictionary-encoded.
create pairsandps --pairsask pyarrow for the two chrom columns asdictionary<int32, string>, so the names are never materialised per pair and are mapped to contig ids once per distinct name per batch rather than once per row. The drop report (unknown contigs with examples, missing positions, out-of-bounds pairs) is unchanged. Together with the key-based aggregation, on a 200 k-contig, 30 M-pair library:create pairs76.0 s / 3.54 GB → 28.1 s / 2.56 GB,--sort-strategy external78.8 s / 3.67 GB → 39.9 s / 3.50 GB; outputs identical. - metacooler's own tables are compressed like cooler's (gzip level 6,
shuffle, chunked):
/genomes,/contigs,/bins/chrom,indexes/chrom_offset,indexes/genome_offsetand the QC columns, through oneio/_utils.write_column, increate,binnify,merge,reassignand QC storage. They were written uncompressed: on a 200 k-contig map they took 11.8 of 47.5 MB (bins/chromalone 3.4 MB, against 0.15 MB now), and the file shrinks from 48.1 to 38.9 MB. A storage change only: filters are transparent to readers, and existing files read as before (see Storage filters). binnifyandzoomifystream re-binned pixels in order. At--level contigthe source-to-target bin map is monotonic, so a source pixel's new row is the new row of itsbin1and the upper triangle is preserved: the source is cut at output-row boundaries of itsbin1_offsetinto blocks of about--chunksizepixels, each block is read with cooler's pixel selector, mapped, sorted on its pixel key and reduced, and the blocks are written in order (ordered=True) — no temporary coolers, no merge passes, memory proportional to one block.--level genome, which reorders contigs, andsquaresources keep the unordered path (now also key-aggregated).zoomifycoarsens each level from the coarsest level already written whose bin size divides it, as cooler's does (10 kb from 5 kb; 2 kb and 5 kb from 1 kb); each level's counts are those of re-binning the base, and its provenance still names the base. On a 200 k-contig map,zoomify -r 1000,2000,5000,10000: 82.7 s / 4.23 GB → 27.5 s / 2.62 GB, every level identical to before.create coolremaps in NumPy and sorts on the pixel key. The remap of each source chunk (_stream_remapped_pixels, shared withreassign's regroup path) no longer copies the chunk and swaps through.loc; and since contigs are only reordered, the in-memory sort is one argsort on the pixel key whose reduction is a no-op — kept because a cooler written without cooler's duplicate check can repeat a pixel, which is summed. On a 200 k-contig cool: 25.4 s / 2.99 GB → 16.7 s / 2.25 GB (together with the key-based aggregation above), output identical.- Per-contig overhead cut at metagenome scale. With two million contigs,
create pairsspent ~25 s on its contig and genome tables alone, before a single pair (measured with a pairs file of headers only). The table builders (_build_contig_order,_build_genomes_table,_build_contigs_table) now resolve each contig's genome once per builder and do the grouping, sorting and summing in NumPy (onelexsort, onenp.add.at) instead of a Python sort per genome with a per-contig key function; the contig-table parser zips object arrays rather than pandas' Arrow-backed columns (4× faster) and normalises each distinct genome name once;validate()detects repeated names with a sort instead of a hash-basednp.unique(4× faster on byte strings); the ASCII checks and_str_to_hdf5are vectorised forstrcolumns (falling back to the per-value path for non-ASCII text); andbinnifyhands cooler contig ids rather than two million names, which cooler iterated as Arrow strings several times per zoom level. Outputs are unchanged. MetaCooler()opens cooler lazily. Building acooler.Coolerreads every contig name into a dict (1.4 s at two million contigs), which the open,info,metadata,binsize,shape,rootand the genome and contig lookups (genome_extent,contig_extent,genome_bin_ranges, ...) never need: they read the attributes with cooler's owninfo()and the offset indexes. It is now created on first use of anything that reads bins, pixels or chroms, or resolves a region (bins(),pixels(),matrix(),extent(),chromnames,bins_for_*,pixels_for_genome_pair,dump, ...), all of which go through it. A contig name is found by one vectorised comparison over the stored/contigsnames (or cooler's dict, once open). An open still refuses a map cooler could not decode (non-ASCII bytes in/contigs) with the named error. At XL: opening 1.37 s → 0.055 s. Also, the pairs header scans read bytes, andcreate pairsmaps each batch's name dictionary through an object array, halving that step.- The contact-QC tally sums each block with one
bincount. Every contact gets the index of its category, and onebincountkeyed on(genome, category)sums every category of every genome at once, where five masked passes used to select each category's contacts first. A chunk is classified in blocks of 2¹⁸ contacts, whose temporaries stay in cache, rather than 50 M at a time (a float-weighted tally, whose sums do not associate, still takes each chunk in one pass, so its bits are unchanged). Tallying the 73 M pixels of a 2 M-contig map: 6.4 s → 2.2 s, with a few MB of scratch memory instead of ~3 GB per 50 M-pixel chunk (the peak ofcreate cool);metacooler qcon that map: 25.8 s / 5.5 GB → 16.6 s / 3.6 GB; outputs identical.
Packaging and CI¶
- The MIT license text ships as
LICENSE(PEP 639 metadata). - Corrected lower bounds:
pyfaidx>=0.7.2.2(older releases importpkg_resources) andrich-click>=1.9(older releases cannot import the CLI). Dropped the unusedbioframedependency andplotlyextra. - The development toolchain is a
devdependency group, souv syncinstalls it; as an extra, the documenteduv sync --devinstalled nothing. - Python 3.13 and 3.14 are supported and tested. A
dependency-boundsCI job runs the suite at the lowest and the newest versions every dependency range allows, andtests/data/cli_tests.shruns in CI. - Benchmarks against cooler 0.12.1:
benchmarks/benchmarks.qmdtimes both tools from the command line on four reproducible synthetic metagenomes (10³ to 2 × 10⁶ contigs): creating a map from pairs, converting, zoomifying, querying a contig, a genome or a genome pair, and genome-level aggregation. Thebenchmarksworkflow renders it for every release tag and on demand, attaches the page and its TSV tables to the release, publishes them on the documentation site at/benchmarks/(the docs build fetches them from thebenchmark-pagerelease), and keeps the datasets in abenchmark-data-v<N>release. The page says outright that every dataset is simulated (no accessions), and reports, beside wall clock: peak memory (for cooler'sdump | awkpipelines, both stages added up), in the summary figure too, against the machine's RAM; the size on disk of the.metacoolagainst the.coolwith its binning tables and against one.coolper MAG (with what those hold); and 3 to 5 runs per command, with their spread drawn and tabulated. A run stopped at the time limit is drawn where it was stopped, as a lower bound, and no longer joined to the line.cooler cload pairsis not run on XL, where it has never finished within the time limit (BENCH_SKIP_COOLER_CREATE).
0.1.0¶
Initial alpha release.
Added¶
.metacoolHDF5 format:genomes/,contigs/,bins/,pixels/,indexes/tables with full cooler backward compatibility via/chromssoft link..mmetacoolcollection format for grouping multiple metacooler entries in a single file.MetaCoolerPython class: genome- and contig-aware accessors, aggregation, range selectors, andto_cooler()export.from_pairs(): streaming creation from 4DN pairs files with support for fixed-size and fragment-level bins.from_cool(): conversion from existing cooler files.merge(): merging replicate metacool files.- CLI commands:
create pairs,create cool,merge,binnify,zoomify,extract,flatten,dump,network,aggregate,show,info,tree,attrs,metadata,ls,stack,cp,mv,rm. - Four contig ordering strategies for
create pairs—genome-then-length,genome-then-alpha,fasta,alpha— pluscoolforcreate cool. - Four contig table input formats:
contigs,contig_genome,metator, column-index spec. --drop-unassignedflag for excluding unassigned contigs.- YAML/JSON metadata ingestion at creation time and via
metadatasubcommand. - GitHub Actions CI across Python 3.10–3.13.
- Pre-commit hooks: ruff, ty, trailing whitespace, YAML/TOML/JSON validation.