Format specification¶
This page is the authoritative reference for the .metacool and .mmetacool
HDF5 formats. It is intended for developers integrating with metacooler, not
for end users.
.metacool — single-sample file¶
A .metacool file is a valid HDF5 file that is simultaneously a valid cooler
file. Any software that reads cooler files (cooltools, HiGlass, the cooler CLI)
will open a .metacool file without modification.
Top-level structure¶
file.metacool (HDF5)
├── @attrs
│ ├── format = "HDF5::Cooler" ← cooler compat
│ ├── format-url = "https://github.com/open2c/cooler"
│ ├── format-version = 3 ← cooler format version
│ ├── storage-mode = "symmetric-upper"
│ ├── metacooler-format = "HDF5::MetaCooler" ← identity marker
│ ├── metacooler-format-url = "https://github.com/js2264/metacooler"
│ ├── metacooler-format-version = 1
│ ├── contig-ordering = "genome-then-length" ← ordering strategy
│ ├── genome-assembly = <str> ← "unknown" if unset (CLI: --assembly-name)
│ ├── bin-size = <int | null> ← null for variable bins
│ ├── bin-type = "fixed" | "variable"
│ ├── creation-date = <ISO 8601>
│ ├── generated-by = "metacooler-<version>"
│ ├── nchroms = <int> ← = number of contigs
│ ├── ngenomes = <int>
│ ├── nbins = <int>
│ ├── nnz = <int> ← number of non-zero pixels
│ ├── sum = <int> ← total contact count
│ └── metadata = <JSON string> ← user metadata + reserved keys (see below)
│
├── /genomes ← MAG / genome table
├── /contigs ← contig table (canonical chromosome reference)
├── /bins ← genomic bins
├── /pixels ← contacts
├── /indexes ← fast-access offset arrays
└── /chroms → soft link to /contigs ← cooler compat
/genomes group¶
One row per genome / MAG.
| Dataset | dtype | Description |
|---|---|---|
name |
S<n> (fixed bytes) |
Genome name, ASCII only (see Names must be ASCII). __unbinned__ is reserved for unassigned contigs. |
length |
int64 |
Total length in bp (sum of assigned contig lengths). |
n_contigs |
int32 |
Number of contigs assigned to this genome. |
n_bins |
int64 |
Number of bins across all contigs of this genome. |
qc_* |
int64 / float64 |
Optional per-genome contact-quality columns (see Contact-quality block). Present when the file was created with --qc (the default) or annotated with metacooler qc --store. |
| extra columns | any | Optional metadata columns from the --genomes table (e.g. phylum, completeness, contamination), at creation or later with metacooler annotate. Stored as fixed-length bytes or numeric arrays. |
The row set is derived from the contig-to-genome assignments: one row per
distinct genome name in /contigs. A --genomes metadata table only supplies
the extra columns, left-joined on name; it cannot add a genome that owns no
contig, nor suppress one that does.
Row order: __unbinned__ first (index 0), then remaining genomes sorted
alphabetically by name. This order defines the genome_id integers used in
/contigs. For genome-grouped orderings (genome-then-length,
genome-then-alpha) the contigs in /contigs are laid out in ascending
genome_id order, so genome_offset delimits each genome's contig block.
/contigs group¶
One row per contig. This is the canonical chromosome reference table. The soft
link /chroms → /contigs makes it transparent to the cooler ecosystem.
| Dataset | dtype | Description |
|---|---|---|
name |
S<n> (fixed bytes) |
Contig name, unique within the file, ASCII only (see Names must be ASCII). |
length |
int32 |
Contig length in bp. |
genome_id |
int32 |
Zero-based index into /genomes. |
qc_intra_contig_short, qc_intra_contig_long |
int64 / float64 |
Optional per-contig contact-quality columns (see Contact-quality block), present whenever the metadata["qc"] block was stored with them. |
| extra columns | any | Optional metadata columns from the --contigs table (e.g. gc_content, coverage), at creation or later with metacooler annotate. ASCII only: cooler reads every column of this group. Every member of the group must be a dataset; a / in a user column name is stored as _, since HDF5 would read it as a nested group. |
Row order is determined by the --contig-order strategy chosen at creation time
and stored in the contig-ordering attribute.
/bins group¶
One row per genomic bin. Identical to the cooler bins table, except the
chrom column stores an integer contig ID (the row index into /contigs)
rather than a string chromosome name or categorical code.
| Dataset | dtype | Description |
|---|---|---|
chrom |
int32 |
Index into /contigs. |
start |
int32 |
Zero-based start position (inclusive). |
end |
int32 |
Zero-based end position (exclusive). |
/pixels group¶
One row per non-zero contact, stored in upper-triangular form
(bin1_id ≤ bin2_id).
| Dataset | dtype | Description |
|---|---|---|
bin1_id |
int64 |
Row bin index (upper-triangular: always ≤ bin2_id). |
bin2_id |
int64 |
Column bin index. |
count |
int32 or int64 |
Number of contacts. int32 at base resolution; int64 wherever counts are summed (binnify, merge), since aggregation readily exceeds the int32 range. Readers must not assume int32. |
Pixels are sorted by (bin1_id, bin2_id) to allow the bin1_offset index to
work correctly, and strictly so: a (bin1_id, bin2_id) pair appears at most
once. validate() and check --deep refuse a repeated pair, which cooler would
sum on some reads and not on others.
/indexes group¶
| Dataset | dtype | Length | Description |
|---|---|---|---|
chrom_offset |
int64 |
n_contigs + 1 |
chrom_offset[i] is the index of the first bin belonging to contig i. |
bin1_offset |
int64 |
n_bins + 1 |
bin1_offset[i] is the index of the first pixel with bin1_id == i. Written by cooler. |
genome_offset |
int64 |
n_genomes + 1 |
genome_offset[i] is the index of the first contig belonging to genome i. metacooler-specific. |
genome_offset is written for every file, but it only delimits contiguous
per-genome blocks under a genome-grouped contig-ordering (genome-then-length,
genome-then-alpha). Under fasta, alpha, cool or custom ordering a
genome's contigs may be scattered, so readers must fall back to filtering
/contigs/genome_id rather than slicing with genome_offset.
Storage filters¶
Every dataset is chunked and compressed with gzip (level 6) behind the shuffle
filter: cooler's defaults, which it applies to /bins and /pixels, and which
metacooler applies to the datasets it writes itself (/genomes, /contigs,
/bins/chrom, indexes/chrom_offset, indexes/genome_offset). HDF5 filters
are transparent to readers, so this is not part of the format's contract: a
file whose metacooler tables are stored uncompressed (as every file written
before this change is) reads and validates exactly the same.
Identity check¶
A file is a valid .metacool if and only if:
- It is a valid HDF5 file.
f.attrs["metacooler-format"] == "HDF5::MetaCooler".- The
/genomesgroup exists. - It passes
metacooler.schema.validate()without raising.
.mmetacool — multi-sample collection¶
A .mmetacool file is a container for multiple .metacool datasets, stored
as HDF5 groups. It is not itself a valid cooler file.
Top-level structure¶
file.mmetacool (HDF5)
├── @attrs
│ ├── format = "HDF5::MMetaCooler"
│ └── format-version = 1
│
└── /entries
├── <entry_name_1> ← full metacooler (same layout as a .metacool root)
├── <entry_name_2>
└── ...
Each entry under /entries contains a complete metacooler: its own /genomes,
/contigs, /bins, /pixels, /indexes, and top-level attributes. Entries
are independent; they do not share data.
Addressing entries¶
Use the :: URI syntax to address a specific entry:
Multi-resolution files (zoomify output)¶
metacooler zoomify produces a .mmetacool file where each entry is named
after its bin size:
Identity check¶
Contig ordering strategies¶
The order of contigs in /contigs and /bins is fixed at creation time and
stored in the contig-ordering attribute. Merging two files requires identical
ordering. reassign is the one command that can change it after the fact: a
re-grouping rewrites the rows into the named strategy, and --layout keep
downgrades the attribute to custom rather than let the file keep claiming a
grouping its rows no longer have.
| Strategy | Description |
|---|---|
genome-then-length |
Group by genome (alphabetical), longest contig first within each genome. Default. |
genome-then-alpha |
Group by genome (alphabetical), contigs alphabetical within each genome. |
fasta |
Preserve the order of contigs as they appear in the FASTA or pairs header. |
alpha |
All contigs sorted alphabetically, ignoring genome assignment. |
cool |
Preserve the exact contig order of the input cooler. create cool, and collect — where every input is one genome, genomes are laid out in genome_id order, and each keeps its input's contig order, so the layout is genome-grouped even though cool is not declared as such (until a reassign scatters a genome — see reassign). |
custom |
Row order inherited from an earlier layout and no longer described by any strategy. Written by reassign --layout keep when the new assignment breaks genome grouping; not a value create can produce. |
The metadata attribute¶
The root metadata attribute is a JSON object. Every key in it is user
metadata — from --metadata files, metadata add/load — except four that
metacooler writes itself. --metadata and metadata add/load/remove refuse
to set or remove those four; metadata list/get read them like any other key.
| Key | Written by | Content |
|---|---|---|
commands |
every command that writes a map | List of the command lines that produced the file, oldest first. Carried forward by every derived map. |
qc |
create, collect, qc --store, reassign |
The contact-quality block (below). |
reassign |
reassign |
List, one entry per run, oldest first: source (the map reassigned), table (the assignment table; empty when the API was given a mapping), layout (the contig-ordering written) and n_contigs_reassigned. |
annotate |
annotate |
List, one entry per run that changed the map, oldest first: genomes_table and contigs_table (the tables given, or null), replace and missing (the options used), and added, replaced, dropped — the columns affected, each as <table>.<column> (e.g. contigs.gc). A run that changes nothing records nothing. |
Contact-quality block¶
create records where the library's contacts landed, and metacooler qc
recomputes it on demand. It occupies three places in the file, all optional —
a file written with --no-qc has none of them.
metadata["qc"], a JSON object inside the root metadata attribute,
alongside the commands provenance list:
| Key | Meaning |
|---|---|
version |
Block schema version (currently 2; version 1 counted a contact between two different __unbinned__ contigs as intra-genome rather than inter-genome). |
source |
"pairs" — measured on read positions by create pairs, before binning — or "pixels", measured between bin midpoints. |
distance_threshold |
Short/long intra-contig split, in bp (default 10 000). |
n_contacts |
Total contacts classified. |
intra_contig_short, intra_contig_long, inter_contig_intra_genome, inter_genome |
The four mutually exclusive categories; they sum to n_contacts. |
intra_contig, inter_contig, intra_genome |
Convenience sums of the above. |
trans_contig_ratio |
The "3D ratio": inter_contig / n_contacts. |
*_fraction |
Each category's share of n_contacts. null on an empty file. |
binsize |
Bin size at measurement time (pixel source). |
n_pairs_read, n_pairs_discarded |
Pairs seen and dropped while binning (pairs source), so the denominator the ratio does not use stays visible. Summed over every pairs file when create pairs reads several. |
/genomes columns, one row per genome:
qc_intra_contig_short, qc_intra_contig_long,
qc_inter_contig_intra_genome, qc_inter_genome (int64), and
qc_intra_genome_ratio (float64, NaN for a genome with no contact). An
inter-genome contact is credited to both genomes involved, so these columns sum
to more than the file-level totals.
/contigs columns, one row per contig: qc_intra_contig_short and
qc_intra_contig_long (int64; float64 on a map with float counts), the
same split contig by contig. A contig's contacts with itself are credited to
its genome, so each /genomes column of that name is the sum of this one over
the genome's contigs, and the column totals are the block's
intra_contig_short and intra_contig_long. Written with the block by every
command that stores one (create, collect, qc --store, reassign); maps
written by metacooler 0.2.0 and earlier do not have them. These two are the
only qc_* columns metacooler writes to /contigs: any other (qc_pass, from
a --contigs table) is user metadata — every writer keeps it, annotate can
drop it, and merge compares it like any metadata column.
These two categories are the only ones that do not depend on the
contig-to-genome assignment, which is why they are kept per contig:
reassign carries them over, sums them per new genome and
re-measures only the two inter-contig categories from the pixels — exactly, at
any bin size — so a source="pairs" block keeps its split measured on read
positions, and every other key of the block (source, distance_threshold,
binsize, n_pairs_read, n_pairs_discarded) is kept as it was. A source
without these columns (or whose columns no longer add up to its block) has
its block recomputed from the pixels instead, and loses the split measured on
read positions.
All three are statements about one specific pixel table, so binnify and
merge drop them from their output instead of carrying them forward, and
merge ignores qc_* columns when it compares the inputs' /genomes and
/contigs tables for compatibility — two libraries over one assembly
legitimately differ there. The names are reserved: a user table column called
qc_intra_contig_short, say, is stored as user_qc_intra_contig_short.
P(s) is a companion file, not part of the format
create pairs also writes a per-MAG P(s) curve, but as a plain TSV beside
the map (gut_T0.metacool → gut_T0.ps.tsv), never inside the HDF5. It is
measured at base-pair resolution from the read positions, so it holds
detail the binned file cannot represent, and it is a plotting input rather
than something any reader of the map needs. See ps.
Cooler compatibility detail¶
The compatibility mechanism relies on two things written by metacooler at file creation time:
- Soft link
/chroms → /contigs: cooler looks up/chromsfor the chromosome reference table. The soft link redirects it transparently to/contigs. metacooler writes it relative (contigs), so it keeps resolving when a map is copied into an.mmetacoolentry, andvalidate()requires it to resolve to the map's own/contigs. format = "HDF5::Cooler"attribute: cooler'sis_cooler()check passes.
The consequence is that cooler.Cooler("gut_T0.metacool").chroms()[:] returns
the contigs table, and all cooler range queries work at contig resolution.
The genome hierarchy is invisible to cooler.
Names must be ASCII¶
Because /contigs is cooler's /chroms, it inherits cooler's assumption
about the text it holds: cooler decodes those byte columns with an ASCII
.astype("U"). A single non-ASCII byte anywhere in the group — in name or in
any extra metadata column — therefore yields a file that satisfies every
structural rule above and still raises UnicodeDecodeError in
cooler.Cooler(), and so in MetaCooler(), which opens through it. Genome
names are subject to the same rule for the opposite reason: cooler never reads
/genomes, but aggregate --level genome writes those names into a .cool as
chrom names, where the encode fails instead of the decode.
So, in both directions:
| Where | Rule |
|---|---|
/contigs, every string column |
ASCII only |
/genomes/name |
ASCII only |
/genomes, extra metadata columns |
any UTF-8 — cooler never sees them |
| HDF5 attributes, metadata JSON | any UTF-8 |
create refuses non-ASCII names before it bins anything, validate() refuses
them on read, and check reports them; the limitation is upstream, in cooler,
but a file that breaks it is one metacooler can write and then not open, which
is the one thing this format promises never to do.