metacooler vs cooler: benchmarks

Building, storing, converting, zooming, querying and aggregating metagenomic Hi-C maps from the command line, on simulated communities of 10³ to 2 × 10⁶ contigs

Rendered 16 November 2026, 22:46 UTC by GitHub Actions (push, v0.2.0) · metacooler 0.2.0 at b0cf6ae · cooler 0.12.1 · h5py 3.16.0 (HDF5 2.0.0), numpy 2.5.4, pandas 3.1.1, pyarrow 26.0.1 · Python 3.13.18, Quarto 1.10.19
On AMD EPYC 7763 64-Core Processor, 2 cores, 7.7 GB, Linux 7.0.0-1017-azure · simulated datasets from the release benchmark-data-v1

Every command below runs as a fresh process, from bash, as a user would type it, under GNU time and timeout. Each run is one row of timings.tsv; every table and figure is drawn from it. cooler and metacooler get the same inputs and must produce the same contacts — that is checked, not assumed. cooler is version 0.12.1, the one uv.lock pins; bench.sh names it, and the settings cell refuses to time any other.

# Render (runs the benchmarks: hours for the full ladder; minutes for S and M)
uv sync && source .venv/bin/activate
cd benchmarks && quarto render benchmarks.qmd
BENCH_DATASETS="S M" quarto render benchmarks.qmd   # a quick look

bench.sh holds the settings and the one timing function, bench; simulate.py makes the datasets; targets.py picks what the queries ask for. A second render re-uses runs already in timings.tsv (BENCH_REDO=1 times them again). The Benchmarks workflow renders this page for every release, and on demand, and publishes it with the metacooler documentation.

Setup: paths, colours, table and figure helpers
import json
import os
from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from IPython.display import Markdown, display
from matplotlib.lines import Line2D
from matplotlib.ticker import FuncFormatter, LogLocator, NullLocator

WORK = Path(os.environ.get("BENCH_WORK", "_work")).resolve()
RESULTS = Path(os.environ.get("BENCH_RESULTS", "_results")).resolve()
DATA, MAPS, QUERIES = WORK / "data", WORK / "maps", WORK / "queries"
FIGURES = RESULTS / "figures"
FIGURES.mkdir(parents=True, exist_ok=True)
DATASETS = os.environ.get("BENCH_DATASETS", "S M L XL").split()
TIMEOUT = float(os.environ.get("BENCH_TIMEOUT", 1800))
LONG = float(os.environ.get("BENCH_LONG", 600))

# Colour is identity: a tool keeps its hue in every figure (validated for
# colour-blind separation); the datasets, ordered by size, one hue light to dark.
HUE = {"metacooler": "#2a78d6", "cooler": "#eb6834"}
SIZE = dict(zip(["S", "M", "L", "XL"], ["#86b6ef", "#3987e5", "#1c5cab", "#0d366b"]))
INK, INK2, GRID, SURFACE = "#0b0b0b", "#52514e", "#e7e6e2", "#fcfcfb"
plt.rcParams.update({
    "figure.facecolor": SURFACE, "axes.facecolor": SURFACE, "savefig.facecolor": SURFACE,
    "axes.edgecolor": INK2, "axes.labelcolor": INK, "xtick.color": INK2, "ytick.color": INK2,
    "text.color": INK, "axes.spines.top": False, "axes.spines.right": False,
    "axes.grid": True, "grid.color": GRID, "grid.linewidth": 0.6, "axes.axisbelow": True,
    "lines.linewidth": 2, "font.size": 9, "axes.titlesize": 10, "axes.titleweight": "bold",
    "axes.titlelocation": "left", "legend.frameon": False,
})


def save(fig, name):
    """Show a figure; keep it as PNG and SVG beside the tables."""
    fig.savefig(FIGURES / f"{name}.png", dpi=200)
    fig.savefig(FIGURES / f"{name}.svg")
    plt.show()


def show(frame, **formats):
    """A table without row numbers."""
    return frame.style.hide(axis="index").format(formats or None, na_rep="")


def timings():
    """Every run, one row each, with the size of its dataset."""
    t = pd.read_csv(RESULTS / "timings.tsv", sep="\t")
    t = t[t["dataset"].isin(DATASETS)].copy()
    t["contigs"] = t["dataset"].map(CONTIGS)
    t["peak_gb"] = t["peak_mb"] / 1024
    return t


def runs(t, **match):
    """The rows of t whose columns equal the values given."""
    keep = np.ones(len(t), bool)
    for column, value in match.items():
        keep &= (t[column] == value).to_numpy()
    return t[keep]


def finished(section, dataset, tool, task):
    """Whether a command's last run finished: its output is there to check."""
    r = runs(timings(), section=section, dataset=dataset, tool=tool, task=task)
    r = r[r["status"] != "skipped"]
    return len(r) > 0 and r["status"].iloc[-1] == "ok"


def ram(t):
    """The memory of the machine the runs were on (GB)."""
    known = pd.to_numeric(t["memory_gb"], errors="coerce").dropna()
    return float(known.iloc[0]) if len(known) else float("nan")


def digits_for(x):
    """As many decimals as a number's size needs: 0.452, 4.52, 45.2, 452."""
    x = abs(x)
    return 3 if x < 1 else 2 if x < 10 else 1 if x < 100 else 0


def human(s):
    """Seconds as ms, s or min: 650 ms, 61.3 s, 30 min."""
    if s is None or s != s:
        return "–"
    value, unit = (s * 1e3, "ms") if s < 1 else (s, "s") if s < 120 else (s / 60, "min")
    return f"{value:,.0f} ms" if unit == "ms" else f"{value:,.1f}".removesuffix(".0") + f" {unit}"


def cell(rows, value="wall_s", digits=None, spread=True):
    """Mean ± sd (n) over the runs that finished; else what became of the command."""
    ok = rows.loc[rows["status"] == "ok", value]
    if len(ok):
        d = digits_for(ok.mean()) if digits is None else digits
        text = f"{ok.mean():,.{d}f}"
        if spread:
            text += (f" ± {ok.std():,.{d}f}" if len(ok) > 1 else "") + f" (n={len(ok)})"
        return text
    if not len(rows):
        return ""
    last = rows.iloc[-1]
    if last["status"] == "timeout":
        if value == "wall_s":
            return f"> {last['wall_s']:,.0f} (stopped at the time limit)"
        if value == "peak_mb":
            return f"≥ {last[value]:,.0f} (when stopped)" if last[value] == last[value] else "not measured (stopped)"
        return "–"
    if last["status"] == "oom":
        if value == "wall_s":
            return f"out of memory after {last[value]:,.0f} s"
        return f"out of memory at {last[value]:,.0f}" if value == "peak_mb" else "–"
    return {"skipped": "not run", "error": "FAILED"}.get(last["status"], last["status"])


def wide(section, value="wall_s", digits=None, spread=True):
    """One row per command, one column per dataset."""
    t = runs(timings(), section=section)
    keys = ["tool", "task", "dataset"]
    cells = pd.DataFrame(
        [(*k, cell(g, value, digits, spread)) for k, g in t.groupby(keys, sort=False)],
        columns=[*keys, "cell"],
    )
    out = cells.pivot(index=["tool", "task"], columns="dataset", values="cell")
    out = out.reindex(list(dict.fromkeys(zip(cells["tool"], cells["task"]))))
    return out.reindex(columns=[d for d in DATASETS if d in out.columns]).reset_index()


def compact(n):
    """1,000 as 1k, 2,000,000 as 2M: tick labels that fit three panels abreast."""
    for div, unit in ((1e6, "M"), (1e3, "k")):
        if n >= div:
            return f"{n / div:.3g}{unit}"
    return f"{n:,}"


def contig_axis(ax):
    """Datasets along x, at their contig counts."""
    ax.set_xscale("log")
    ax.xaxis.set_minor_locator(NullLocator())
    ax.set_xticks(
        [CONTIGS[d] for d in DATASETS],
        [f"{d}\n{compact(CONTIGS[d])}" for d in DATASETS],
    )
    ax.set_xlim(min(CONTIGS.values()) / 2.5, max(CONTIGS.values()) * 2.5)
    ax.set_xlabel("contigs in the assembly")


def log_y(ax, label):
    """A log axis labelled at 1, 2, 5 of each decade, so that a short range still reads."""
    ax.set_yscale("log")
    ax.yaxis.set_major_locator(LogLocator(subs=(1, 2, 5)))
    ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:g}"))
    ax.yaxis.set_minor_locator(NullLocator())
    ax.set_ylabel(label)


def seconds_axis(ax, unit="s"):
    log_y(ax, f"wall clock ({unit})")


def stopped(ax, x, y, colour, status):
    """A run that did not finish, where it ended. Stopped at the time limit: a
    bar with an arrow up, since it would have taken longer -- a lower bound,
    measured where timeout killed it. Killed for want of memory: a cross."""
    if status == "timeout":
        ax.scatter([x], [y], marker="_", s=160, color=colour, lw=2.5, zorder=5)
        ax.annotate("", xy=(x, y * 2.4), xytext=(x, y), zorder=5,
                    arrowprops=dict(arrowstyle="-|>", color=colour, lw=1.5, mutation_scale=10))
        ax.annotate("stopped at\nthe time limit", (x, y * 1.55), xytext=(-7, 0), textcoords="offset points",
                    ha="right", va="center", color=INK2, fontsize=7.5, zorder=6,
                    bbox=dict(boxstyle="square,pad=0.1", fc=SURFACE, ec="none"))
        ax.update_datalim([(x, y * 2.8)])
    else:
        ax.scatter([x], [y], marker="X", s=60, color=colour, zorder=5)
        ax.annotate("out of memory", (x, y), xytext=(0, 9), textcoords="offset points",
                    ha="center", color=INK2, fontsize=7.5)


def scaling(ax, rows, value="wall_s"):
    """One line per command through the means of its runs; every run a dot,
    and a bar from its fastest to its slowest. A run that did not finish is
    drawn where it ended (see `stopped`), and no line joins it: nothing is
    extrapolated. A command not run is marked so."""
    for (tool, _task), r in rows.groupby(["tool", "task"], sort=False):
        colour = HUE[tool]
        ok = r[r["status"] == "ok"]
        stats = ok.groupby("contigs")[value].agg(["mean", "min", "max"])
        if len(stats):
            ax.errorbar(stats.index, stats["mean"],
                        yerr=[stats["mean"] - stats["min"], stats["max"] - stats["mean"]],
                        color=colour, marker="o", ms=5, capsize=2.5, elinewidth=1, zorder=3)
            ax.scatter(ok["contigs"], ok[value], s=9, color=colour, alpha=0.35, lw=0, zorder=4)
        for _, run in r[r["status"].isin(["timeout", "oom"]) & r[value].notna()].iterrows():
            stopped(ax, run["contigs"], run[value], colour, run["status"])
        for x in r.loc[r["status"] == "skipped", "contigs"]:
            ax.annotate(f"{tool}\nnot run", (x, 0.03), xycoords=("data", "axes fraction"),
                        ha="center", va="bottom", color=colour, fontsize=7.5)
    contig_axis(ax)


def limit_line(ax, rows):
    """The time limit, where a run in the panel reached it."""
    if (rows["status"] == "timeout").any():
        ax.axhline(TIMEOUT, color=INK2, lw=0.8, ls=(0, (1, 2)), zorder=1)
        ax.annotate(f"time limit, {human(TIMEOUT)}", (0.02, TIMEOUT), xycoords=("axes fraction", "data"),
                    xytext=(0, 3), textcoords="offset points", color=INK2, fontsize=7.5)


def ram_line(ax, gb, vertical=False):
    """The memory of the machine the runs were on."""
    if gb != gb:
        return
    if vertical:
        ax.axvline(gb, color=INK2, lw=1, ls=(0, (4, 2)), zorder=1)
        ax.annotate(f"RAM of this machine\n{gb:.1f} GB", (gb, 1.0), xycoords=("data", "axes fraction"),
                    xytext=(-4, -2), textcoords="offset points", ha="right", va="top",
                    color=INK2, fontsize=7.5)
    else:
        ax.axhline(gb, color=INK2, lw=1, ls=(0, (4, 2)), zorder=1)
        ax.annotate(f"RAM of this machine, {gb:.1f} GB", (0.02, gb), xycoords=("axes fraction", "data"),
                    xytext=(0, 3), textcoords="offset points", color=INK2, fontsize=7.5)
        ax.update_datalim([(min(CONTIGS.values()), gb * 1.6)])


COMMANDS = {
    ("create", "cload pairs"): "cooler cload pairs",
    ("create", "create pairs"): "metacooler create pairs",
    ("convert", "create cool"): "metacooler create cool",
    ("convert", "flatten"): "metacooler flatten",
    ("query", "contig", "cooler"): "cooler dump -r (a contig)",
    ("query", "contig", "metacooler"): "metacooler dump -r (a contig)",
    ("query", "genome", "cooler"): "cooler dump --join | awk (a genome)",
    ("query", "genome", "metacooler"): "metacooler dump -r (a genome)",
    ("query", "genome pair", "cooler"): "cooler dump --join | awk (a genome pair)",
    ("query", "genome pair", "metacooler"): "metacooler dump -r -r2 (a genome pair)",
    ("aggregate", "genome network", "cooler"): "cooler dump --join | awk (genome network)",
    ("aggregate", "genome network", "metacooler"): "metacooler network --level genome",
    ("aggregate", "genome-level .cool"): "metacooler aggregate --level genome",
}


def command(section, tool, task):
    """What a user types, in a few words: the name a sentence gives a command."""
    return COMMANDS.get((section, task, tool)) or COMMANDS.get((section, task)) or f"{tool} {task}"


TOOLS_LEGEND = [Line2D([], [], color=HUE[t], marker="o", ms=5, label=t) for t in ("cooler", "metacooler")]

Settings

%%bash
source bench.sh
echo "datasets: $DATASETS · bin size: $BINSIZE bp · zoom levels: $ZOOMS"
echo "time limit: $TIMEOUT s per run"
if [[ -n ${BENCH_REPS:-} ]]; then
    echo "runs per command: $BENCH_REPS"
else
    echo "runs per command: $MIN_REPS to $MAX_REPS (no more than $MIN_REPS once they add up to $BUDGET s;"
    echo "  one, for a command whose run takes over $LONG s)"
fi
metacooler --version
cooler --version
# Every timing is against one cooler: refuse to time another.
installed=$(python -c 'import cooler; print(cooler.__version__)')
if [[ $installed != "$COOLER_VERSION" ]]; then
    echo "cooler $installed is installed; these benchmarks are against cooler $COOLER_VERSION." >&2
    echo "Run uv sync (uv.lock pins it), or bump COOLER_VERSION in bench.sh with the lock." >&2
    exit 1
fi
[[ -n $SKIP_COOLER_CREATE ]] && echo "cooler cload pairs not run on: $SKIP_COOLER_CREATE"
datasets: S M L XL · bin size: 1000 bp · zoom levels: 1000,2000,5000,10000
time limit: 1800 s per run
runs per command: 3 to 5 (no more than 3 once they add up to 120 s;
  one, for a command whose run takes over 600 s)
metacooler, version 0.2.0
cooler, version 0.12.1
cooler cload pairs not run on: XL

Datasets

All four datasets are simulated. No real reads, assembly or binning enters them, so they have no accessions. Each is drawn by simulate.py from a preset and a fixed seed, so every render times the same data; the workflow publishes them as a release (benchmark-data-v<generator version>) and fetches them from there.

  • Genomes. MAGs (log-normal, 2.5 Mb median) and small bins standing for plasmids, phages and other mobile elements (20 kb median) are binned. Low-abundance genomes that no binner would recover make up the rest of the assembly, and their contigs end up in __unbinned__.
  • Contigs. Each genome is cut around its own median contig length, so one MAG comes out in a dozen long contigs and the next in hundreds of short ones. Unbinned contigs are short (~0.9 kb). Nothing is under 500 bp. The assembly lists contigs in random order, as an assembler does, so a genome’s contigs are scattered through any genome-agnostic .cool.
  • Contacts. A pair falls in a genome in proportion to abundance × length; 96 % stay there, at a P(s) ~ 1/s separation (60 %) or uniformly along a circular chromosome (40 %); 4 % join two genomes. Which pairs are intra-contig, inter-contig or inter-genome then follows from the contig lengths, as in a real library.

The presets are sized after real metagenomic Hi-C maps — M about a mock community’s, L about a human gut sample’s, in genomes, contigs, bins and contacts — but no data of those maps is used. What the simulation decides matters most for the genome queries (§5): cooler scans every pixel of the map, metacooler reads those of one genome, so their ratio follows the share of the map’s pixels the queried genome holds. That share is set here by the model’s community — how many genomes, how abundant — not by a sample; the query table gives it for each dataset.

%%bash
source bench.sh
for ds in $DATASETS; do  # fetched, made earlier, or made now
    python simulate.py --preset "$ds" --outdir "$DATA/$ds" --keep | sed "s|$WORK/||"
done
cd "$WORK" && du -sh data/*
S: data/S is up to date (generator v1)
M: data/M is up to date (generator v1)
L: data/L is up to date (generator v1)
XL: data/XL is up to date (generator v1)
249M    data/L
77M data/M
7.6M    data/S
936M    data/XL
Code
SUMMARY = {ds: json.loads((DATA / ds / "summary.json").read_text()) for ds in DATASETS}
CONTIGS = {ds: s["contigs"] for ds, s in SUMMARY.items()}
BINSIZE = int(os.environ.get("BENCH_BINSIZE", 1000))
SIZES = {
    ds: pd.read_csv(DATA / ds / "contigs.sizes", sep="\t", header=None, names=["contig", "length"])
    for ds in DATASETS
}
rows = []
for ds, s in SUMMARY.items():
    rows.append({
        "dataset": ds,
        "source": f"simulated (simulate.py v{s['generator_version']}, seed {s['seed']})",
        "genomes (MAGs + small bins)": f"{s['genomes']:,} ({s['mags']:,} + {s['mge_bins']:,})",
        "contigs": f"{s['contigs']:,}",
        "binned contigs": f"{s['binned_contigs'] / s['contigs']:.0%}",
        "assembly (Mb)": f"{s['assembly_bp'] / 1e6:,.0f}",
        "contig N50 (kb)": f"{s['contig_n50'] / 1e3:.1f}",
        "median contig (bp)": f"{s['contig_median']:,}",
        "contigs / MAG (median)": f"{s['contigs_per_mag_median']:,.0f}",
        "MAG size (median, Mb)": f"{s['mag_bp_median'] / 1e6:.2f}",
        "MAG contig N50 (kb)": f"{s['mag_contig_n50_median'] / 1e3:.1f}",
        "read pairs (M)": f"{s['pairs'] / 1e6:,.0f}",
        "intra-contig": f"{s['pairs_intra_contig']:.1%}",
        "inter-contig, intra-genome": f"{s['pairs_inter_contig_intra_genome']:.1%}",
        "inter-genome": f"{s['pairs_inter_genome']:.1%}",
        "unbinned, inter-contig": f"{s['pairs_unbinned']:.1%}",
        f"bins at {BINSIZE:,} bp": f"{int((-(-SIZES[ds]['length'] // BINSIZE)).sum()):,}",
        "pairs.gz (MB)": f"{s['files_bytes']['contacts.pairs.gz'] / 1e6:,.0f}",
        "pairs md5 (uncompressed)": s["pairs_md5_uncompressed"][:12],
    })
DATASET_TABLE = pd.DataFrame(rows)
DATASET_TABLE.to_csv(RESULTS / "datasets.tsv", sep="\t", index=False)
show(DATASET_TABLE.set_index("dataset").T.reset_index().rename(columns={"index": ""}))
Table 1: The four simulated datasets. Contig N50 is that of the whole assembly; MAG contig N50 is the median over MAGs of each MAG’s own contig N50. Pair categories are the generated truth, measured against the binning.
S M L XL
source simulated (simulate.py v1, seed 1) simulated (simulate.py v1, seed 2) simulated (simulate.py v1, seed 3) simulated (simulate.py v1, seed 4)
genomes (MAGs + small bins) 10 (10 + 0) 100 (50 + 50) 2,000 (200 + 1,800) 20,000 (1,000 + 19,000)
contigs 1,000 20,000 200,000 2,000,000
binned contigs 48% 49% 41% 24%
assembly (Mb) 27 157 764 5,119
contig N50 (kb) 80.1 27.6 11.6 6.5
median contig (bp) 2,491 1,755 1,242 1,032
contigs / MAG (median) 50 122 209 198
MAG size (median, Mb) 2.80 2.72 2.56 2.46
MAG contig N50 (kb) 88.0 27.6 19.1 19.7
read pairs (M) 1 10 30 100
intra-contig 35.4% 28.2% 24.7% 27.0%
inter-contig, intra-genome 62.0% 66.9% 69.1% 63.8%
inter-genome 2.5% 3.7% 3.7% 3.5%
unbinned, inter-contig 0.1% 1.1% 2.5% 5.7%
bins at 1,000 bp 27,498 166,079 852,561 5,978,049
pairs.gz (MB) 8 80 250 874
pairs md5 (uncompressed) 2264f7277b25 1cd62a937421 a0fe807ef85c 59e7cba48488
Code
fig, (left, right) = plt.subplots(1, 2, figsize=(9, 3.2), layout="constrained")
for ds in DATASETS:
    sizes = SIZES[ds]
    binning = pd.read_csv(DATA / ds / "contigs.tsv", sep="\t", usecols=[0, 1])
    per_genome = binning.loc[binning["genome_name"] != "__unbinned__", "genome_name"].value_counts()
    edges = np.geomspace(500, sizes["length"].max() + 1, 30)
    share, _ = np.histogram(sizes["length"], edges)
    left.plot(np.sqrt(edges[1:] * edges[:-1]), share / share.sum(), color=SIZE[ds], label=ds)
    n = np.sort(per_genome.to_numpy())
    right.step(n, 1 - np.arange(len(n)) / len(n), where="post", color=SIZE[ds], label=ds)
for ax, xlabel, ylabel in ((left, "contig length (bp)", "share of contigs"),
                           (right, "contigs per binned genome", "share of genomes with at least as many")):
    ax.set(xscale="log", xlabel=xlabel, ylabel=ylabel)
    ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:.0%}"))
left.legend(title="dataset")
save(fig, "datasets")
Figure 1: What the simulated datasets are made of: contig lengths, as the share of contigs in log-spaced bins (left), and how many contigs a binned genome has, as the share of genomes with at least that many (right).

1 · Creating a map from pairs

cooler cload pairs and metacooler create pairs, on the same pairs, contig lengths and bin size; metacooler also reads the binning and genome tables and builds /genomes, its index, the QC block and the per-MAG P(s) table — its defaults, all on.

On a metagenome the difference is the contig count, not the pixels. cooler.create types /bins/chrom as an HDF5 enumeration with one member per contig. Building it is quadratic, because each insertion rescans the members already there, and past a few thousand contigs the type outgrows HDF5’s object header anyway and is dropped for plain integers. cooler cload pairs sorts externally: it writes the bin table once per chunk of pairs and once per merge, so it pays that cost several times per file. metacooler writes the integers directly.

cooler cload pairs is not run on XL: there it has never finished within the 30-min time limit (every earlier render stopped it at the limit), so a run would only spend another half hour to say so again. BENCH_SKIP_COOLER_CREATE= (empty) runs it on every dataset.

%%bash
source bench.sh
for ds in $DATASETS; do
    d=$DATA/$ds
    warm "$d/contacts.pairs.gz"
    if [[ " $SKIP_COOLER_CREATE " == *" $ds "* ]]; then
        skip create "$ds" cooler "cload pairs" \
            "it has never finished on $ds within the time limit (BENCH_SKIP_COOLER_CREATE)"
    else
        bench create "$ds" cooler "cload pairs" "$MAPS/$ds.cool" -- \
            cooler cload pairs -c1 2 -p1 3 -c2 4 -p2 5 \
                "$d/contigs.sizes:$BINSIZE" "$d/contacts.pairs.gz" "$MAPS/$ds.cool"
    fi
    bench create "$ds" metacooler "create pairs" "$MAPS/$ds.metacool" -- \
        metacooler create pairs --pairs "$d/contacts.pairs.gz" --bins "$BINSIZE" \
            --contigs "$d/contigs.tsv" --genomes "$d/genomes.tsv" -o "$MAPS/$ds.metacool"
done
create/S/cooler/cload pairs                  run 1  ok          1.82 s     431 MB
create/S/cooler/cload pairs                  run 2  ok          1.84 s     430 MB
create/S/cooler/cload pairs                  run 3  ok          1.86 s     430 MB
create/S/cooler/cload pairs                  run 4  ok          1.84 s     430 MB
create/S/cooler/cload pairs                  run 5  ok          1.83 s     430 MB
create/S/metacooler/create pairs             run 1  ok          1.33 s     398 MB
create/S/metacooler/create pairs             run 2  ok          1.18 s     395 MB
create/S/metacooler/create pairs             run 3  ok          1.17 s     396 MB
create/S/metacooler/create pairs             run 4  ok          1.17 s     396 MB
create/S/metacooler/create pairs             run 5  ok          1.16 s     395 MB
create/M/cooler/cload pairs                  run 1  ok         17.90 s    2611 MB
create/M/cooler/cload pairs                  run 2  ok         17.44 s    2614 MB
create/M/cooler/cload pairs                  run 3  ok         17.28 s    2615 MB
create/M/cooler/cload pairs                  run 4  ok         17.50 s    2615 MB
create/M/cooler/cload pairs                  run 5  ok         17.53 s    2615 MB
create/M/metacooler/create pairs             run 1  ok          5.91 s    1476 MB
create/M/metacooler/create pairs             run 2  ok          5.98 s    1476 MB
create/M/metacooler/create pairs             run 3  ok          5.96 s    1476 MB
create/M/metacooler/create pairs             run 4  ok          5.98 s    1460 MB
create/M/metacooler/create pairs             run 5  ok          6.06 s    1476 MB
create/L/cooler/cload pairs                  run 1  ok        410.33 s    4201 MB
create/L/cooler/cload pairs                  run 2  ok        401.60 s    4203 MB
create/L/cooler/cload pairs                  run 3  ok        394.31 s    4203 MB
create/L/metacooler/create pairs             run 1  ok         19.36 s    2370 MB
create/L/metacooler/create pairs             run 2  ok         19.41 s    2365 MB
create/L/metacooler/create pairs             run 3  ok         19.60 s    2437 MB
create/L/metacooler/create pairs             run 4  ok         20.24 s    2304 MB
create/L/metacooler/create pairs             run 5  ok         20.06 s    2379 MB
create/XL/cooler/cload pairs                 skipped: it has never finished on XL within the time limit (BENCH_SKIP_COOLER_CREATE)
create/XL/metacooler/create pairs            run 1  ok        120.30 s    5832 MB
create/XL/metacooler/create pairs            run 2  ok        106.40 s    6165 MB
create/XL/metacooler/create pairs            run 3  ok        106.09 s    6285 MB
Code
show(wide("create"))
Table 2: Creating a map: wall-clock seconds, mean ± sd (n runs).
tool task S M L XL
cooler cload pairs 1.84 ± 0.01 (n=5) 17.5 ± 0.2 (n=5) 402 ± 8 (n=3) not run
metacooler create pairs 1.20 ± 0.07 (n=5) 5.98 ± 0.05 (n=5) 19.7 ± 0.4 (n=5) 111 ± 8 (n=3)
Code
show(wide("create", "peak_mb", digits=0))
Table 3: Creating a map: peak memory (MB), mean ± sd (n runs). A run stopped at the time limit gives the memory it held then, a lower bound.
tool task S M L XL
cooler cload pairs 430 ± 0 (n=5) 2,614 ± 2 (n=5) 4,202 ± 1 (n=3) not run
metacooler create pairs 396 ± 1 (n=5) 1,473 ± 7 (n=5) 2,371 ± 47 (n=5) 6,094 ± 235 (n=3)
Code
t = timings()
rows = runs(t, section="create")
fig, (left, right) = plt.subplots(1, 2, figsize=(9, 3.6), layout="constrained")
scaling(left, rows)
seconds_axis(left)
limit_line(left, rows)
scaling(right, rows, "peak_gb")
log_y(right, "peak memory (GB)")
ram_line(right, ram(t))
left.legend(handles=TOOLS_LEGEND, loc="lower right")
save(fig, "create")
Figure 2: Creating a map from pairs: wall clock (left) and peak memory (right) against the number of contigs. A line joins the means of a command’s runs, each run is a dot, and the bar spans its runs. A run stopped by timeout at the time limit (dotted) is a bar with an arrow up, where it was stopped: a measured lower bound, which no line joins and nothing extrapolates. A command not run is marked so. Right, dashed: the RAM of the machine.

The two tools must hold the same contacts: same number of pixels, same total, read from each file’s own attributes.

Code
MADE_BY = {"cooler": ("cload pairs", "cool"), "metacooler": ("create pairs", "metacool")}
same = []
for ds in DATASETS:
    row, held = {"dataset": ds}, {}
    for tool, (task, ext) in MADE_BY.items():
        if finished("create", ds, tool, task):  # else the map below is not this tool's
            with h5py.File(MAPS / f"{ds}.{ext}", "r") as f:
                held[tool] = int(f.attrs["nnz"]), int(f.attrs["sum"])
            row[f"{tool} nnz"], row[f"{tool} sum"] = held[tool]
    missing = [tool for tool in MADE_BY if tool not in held]
    row["identical"] = (
        f"{' and '.join(missing)} did not finish or was not run" if missing
        else "yes" if held["cooler"] == held["metacooler"] else "NO"
    )
    same.append(row)
same = pd.DataFrame(same)
assert not (same["identical"] == "NO").any(), "cooler and metacooler disagree on the contacts"
show(same, **{c: "{:,.0f}" for c in same.columns if c.endswith(("nnz", "sum"))})
Table 4: The contacts each tool’s map holds: pixels (nnz) and their total, from the file’s own attributes.
dataset cooler nnz cooler sum metacooler nnz metacooler sum identical
S 811,390 1,000,000 811,390 1,000,000 yes
M 7,684,333 10,000,000 7,684,333 10,000,000 yes
L 22,335,454 30,000,000 22,335,454 30,000,000 yes
XL 73,140,434 100,000,000 cooler did not finish or was not run

Where cooler did not finish or was not run, the rest of the page still needs the genome-agnostic .cool. It is built here, untimed, by metacooler in the assembly’s own contig order (--contig-order fasta, which reproduces cooler cload pairs pixel for pixel), then flattened to a plain cooler.

%%bash
source bench.sh
for ds in $DATASETS; do
    [[ -s $MAPS/$ds.cool ]] && continue
    d=$DATA/$ds
    echo "$ds: no $ds.cool from cooler; building it with metacooler"
    metacooler create pairs -q --contig-order fasta --no-qc --no-ps --pairs "$d/contacts.pairs.gz" \
        --bins "$BINSIZE" --contigs "$d/contigs.tsv" -o "$SCRATCH/$ds.fasta.metacool" &&
        metacooler flatten -q "$SCRATCH/$ds.fasta.metacool" -o "$MAPS/$ds.cool"
    rm -f "$SCRATCH/$ds.fasta.metacool" "$SCRATCH/$ds.fasta.ps.tsv"
done
cd "$MAPS" && du -h -- *.cool *.metacool
XL: no XL.cool from cooler; building it with metacooler
49M L.cool
14M M.cool
1.4M    S.cool
196M    XL.cool
38M L.metacool
12M M.metacool
1.4M    S.metacool
154M    XL.metacool

2 · Size on disk

What it takes to keep a metagenome’s map with its genomes, on disk, four ways:

  • .metacool: one file, every contig and contact, with the binning, the genome and contig tables, their index and the QC block (metacooler create pairs, above).
  • .cool + binning tables: the genome-agnostic .cool (cooler cload pairs) and, beside it, the tables that say which contig is in which genome and what each genome is — here simulate.py’s contigs.tsv (contig, genome, GC, coverage) and genomes.tsv (completeness, contamination, quality, GC, abundance), the kind of tables a binner such as MetaTOR writes. The .cool alone holds the same contacts, without knowing a genome from another.
  • One .cool per MAG: each MAG’s contigs and the contacts within it, one file per MAG — the MAGs only, the way they are usually handed on. They hold less than the others: no unbinned contig, no small bin, no contact between two genomes. The table below says how much less. Written untimed by metacooler extract --split, through cooler’s own writer, so they are the files cooler itself writes for those contigs and contacts.

The input pairs (contacts.pairs.gz) are given for scale.

%%bash
source bench.sh
# simulate.py names the MAGs MAG_*: the filter picks them out of /genomes.
for ds in $DATASETS; do
    out=$MAPS/$ds.mags
    [[ -e $out/.complete && $out/.complete -nt $MAPS/$ds.metacool ]] && continue
    rm -rf "$out" && mkdir -p "$out"
    metacooler extract -q "$MAPS/$ds.metacool" --filter 'genomes.name =~ ^MAG_' --split -o "$out/$ds" &&
        touch "$out/.complete"
done
cd "$MAPS" && du -sh -- *.mags
40M L.mags
13M M.mags
1.7M    S.mags
138M    XL.mags
Code
def held_by(path):
    """Contigs, pixels and contacts a cool or metacool holds, from its own attributes."""
    with h5py.File(path, "r") as f:
        return len(f["chroms/name"]), int(f.attrs["nnz"]), int(f.attrs["sum"])


STORAGE, HELD, MAG_SHARE = [], [], {}
for ds in DATASETS:
    d = DATA / ds
    tables = (d / "contigs.tsv").stat().st_size + (d / "genomes.tsv").stat().st_size
    cool = (MAPS / f"{ds}.cool").stat().st_size
    mags = sorted((MAPS / f"{ds}.mags").glob("*.cool"))
    for option, files, size in (
        (".metacool", 1, (MAPS / f"{ds}.metacool").stat().st_size),
        (".cool + binning tables", 3, cool + tables),
        (".cool alone (no genomes)", 1, cool),
        ("one .cool per MAG", len(mags), sum(p.stat().st_size for p in mags)),
        ("contacts.pairs.gz (input)", 1, (d / "contacts.pairs.gz").stat().st_size),
    ):
        STORAGE.append({"dataset": ds, "contigs": CONTIGS[ds], "option": option, "files": files, "mb": size / 1e6})
    contigs, nnz, total = held_by(MAPS / f"{ds}.metacool")
    in_mags = np.array([held_by(p) for p in mags]).reshape(-1, 3).sum(axis=0)
    MAG_SHARE[ds] = {"contigs": in_mags[0] / contigs, "pixels": in_mags[1] / nnz, "contacts": in_mags[2] / total}
    HELD.append({
        "dataset": ds, "MAG files": f"{len(mags):,}",
        "contigs": f"{in_mags[0]:,} of {contigs:,} ({in_mags[0] / contigs:.1%})",
        "pixels": f"{in_mags[1]:,} of {nnz:,} ({in_mags[1] / nnz:.1%})",
        "contacts": f"{in_mags[2]:,} of {total:,} ({in_mags[2] / total:.1%})",
    })
STORAGE = pd.DataFrame(STORAGE)
STORAGE.to_csv(RESULTS / "storage.tsv", sep="\t", index=False)
OPTIONS = list(dict.fromkeys(STORAGE["option"]))
table = STORAGE.assign(
    cell=[f"{mb:,.1f}" + (f" ({files:,} files)" if files > 1 else "") for mb, files in zip(STORAGE["mb"], STORAGE["files"])]
).pivot(index="option", columns="dataset", values="cell").reindex(index=OPTIONS, columns=DATASETS)
show(table.rename_axis("MB on disk").reset_index())
Table 5: Size on disk (MB) of each way to keep the map with its genomes, and how many files it takes. The .cool is cooler’s own where cooler finished, and the untimed one built above where it did not.
MB on disk S M L XL
.metacool 1.4 12.0 38.9 161.3
.cool + binning tables 1.5 (3 files) 15.2 (3 files) 57.5 (3 files) 277.1 (3 files)
.cool alone (no genomes) 1.5 14.5 50.7 204.7
one .cool per MAG 1.7 (10 files) 13.4 (50 files) 41.0 (200 files) 142.5 (1,000 files)
contacts.pairs.gz (input) 7.9 79.7 250.0 874.1
Code
show(pd.DataFrame(HELD))
Table 6: What the per-MAG .cool files hold, out of the whole map (the .metacool and the .cool hold all of it).
dataset MAG files contigs pixels contacts
S 10 483 of 1,000 (48.3%) 785,061 of 811,390 (96.8%) 973,506 of 1,000,000 (97.4%)
M 50 9,380 of 20,000 (46.9%) 7,202,910 of 7,684,333 (93.7%) 9,432,108 of 10,000,000 (94.3%)
L 200 66,285 of 200,000 (33.1%) 19,658,405 of 22,335,454 (88.0%) 24,726,493 of 30,000,000 (82.4%)
XL 1,000 315,307 of 2,000,000 (15.8%) 56,600,973 of 73,140,434 (77.4%) 68,033,411 of 100,000,000 (68.0%)

3 · Converting between the formats

No cooler counterpart: adding the genome layer to an existing .cool (create cool), and dropping it again (flatten).

%%bash
source bench.sh
for ds in $DATASETS; do
    d=$DATA/$ds
    bench convert "$ds" metacooler "create cool" "$SCRATCH/$ds.from-cool.metacool" -- \
        metacooler create cool --cool "$MAPS/$ds.cool" --contigs "$d/contigs.tsv" \
            --genomes "$d/genomes.tsv" -o "$SCRATCH/$ds.from-cool.metacool"
    bench convert "$ds" metacooler flatten "$SCRATCH/$ds.flat.cool" -- \
        metacooler flatten "$MAPS/$ds.metacool" -o "$SCRATCH/$ds.flat.cool"
done
convert/S/metacooler/create cool             run 1  ok          1.00 s     218 MB
convert/S/metacooler/create cool             run 2  ok          0.99 s     218 MB
convert/S/metacooler/create cool             run 3  ok          0.98 s     218 MB
convert/S/metacooler/create cool             run 4  ok          0.99 s     218 MB
convert/S/metacooler/create cool             run 5  ok          0.98 s     217 MB
convert/S/metacooler/flatten                 run 1  ok          0.85 s     217 MB
convert/S/metacooler/flatten                 run 2  ok          0.85 s     217 MB
convert/S/metacooler/flatten                 run 3  ok          0.85 s     217 MB
convert/S/metacooler/flatten                 run 4  ok          0.84 s     217 MB
convert/S/metacooler/flatten                 run 5  ok          0.84 s     217 MB
convert/M/metacooler/create cool             run 1  ok          3.81 s     695 MB
convert/M/metacooler/create cool             run 2  ok          3.78 s     697 MB
convert/M/metacooler/create cool             run 3  ok          3.80 s     696 MB
convert/M/metacooler/create cool             run 4  ok          3.77 s     696 MB
convert/M/metacooler/create cool             run 5  ok          3.78 s     697 MB
convert/M/metacooler/flatten                 run 1  ok          3.04 s     645 MB
convert/M/metacooler/flatten                 run 2  ok          3.04 s     645 MB
convert/M/metacooler/flatten                 run 3  ok          3.03 s     645 MB
convert/M/metacooler/flatten                 run 4  ok          3.01 s     644 MB
convert/M/metacooler/flatten                 run 5  ok          3.04 s     643 MB
convert/L/metacooler/create cool             run 1  ok         12.48 s    1810 MB
convert/L/metacooler/create cool             run 2  ok         12.45 s    1810 MB
convert/L/metacooler/create cool             run 3  ok         12.73 s    1808 MB
convert/L/metacooler/create cool             run 4  ok         12.64 s    1809 MB
convert/L/metacooler/create cool             run 5  ok         12.74 s    1811 MB
convert/L/metacooler/flatten                 run 1  ok          9.07 s    1604 MB
convert/L/metacooler/flatten                 run 2  ok          9.24 s    1605 MB
convert/L/metacooler/flatten                 run 3  ok          9.13 s    1605 MB
convert/L/metacooler/flatten                 run 4  ok          9.07 s    1605 MB
convert/L/metacooler/flatten                 run 5  ok          9.09 s    1604 MB
convert/XL/metacooler/create cool            run 1  ok         72.17 s    4469 MB
convert/XL/metacooler/create cool            run 2  ok         72.39 s    4464 MB
convert/XL/metacooler/create cool            run 3  ok         73.68 s    4494 MB
convert/XL/metacooler/flatten                run 1  ok         42.85 s    4391 MB
convert/XL/metacooler/flatten                run 2  ok         43.04 s    4390 MB
convert/XL/metacooler/flatten                run 3  ok         43.73 s    4391 MB
Code
show(wide("convert"))
Table 7: Converting: wall-clock seconds, mean ± sd (n runs).
tool task S M L XL
metacooler create cool 0.988 ± 0.008 (n=5) 3.79 ± 0.02 (n=5) 12.6 ± 0.1 (n=5) 72.7 ± 0.8 (n=3)
metacooler flatten 0.846 ± 0.005 (n=5) 3.03 ± 0.01 (n=5) 9.12 ± 0.07 (n=5) 43.2 ± 0.5 (n=3)

4 · Multi-resolution maps

cooler zoomify and metacooler zoomify, to the same resolutions. Each coarser level is another cooler.create, so cooler pays the per-contig cost once per level. Where cooler could not build the base map, its pyramid is not attempted.

%%bash
source bench.sh
for ds in $DATASETS; do
    bench zoomify "$ds" metacooler zoomify "$SCRATCH/$ds.mmetacool" -- \
        metacooler zoomify "$MAPS/$ds.metacool" -r "$ZOOMS" -o "$SCRATCH/$ds.mmetacool"
    if unfinished create "$ds" cooler "cload pairs"; then
        skip zoomify "$ds" cooler zoomify "no base map from cooler cload pairs on $ds"
        continue
    fi
    bench zoomify "$ds" cooler zoomify "$SCRATCH/$ds.mcool" -- \
        cooler zoomify -r "$ZOOMS" -o "$SCRATCH/$ds.mcool" "$MAPS/$ds.cool"
done
zoomify/S/metacooler/zoomify                 run 1  ok          1.46 s     245 MB
zoomify/S/metacooler/zoomify                 run 2  ok          1.43 s     245 MB
zoomify/S/metacooler/zoomify                 run 3  ok          1.44 s     245 MB
zoomify/S/metacooler/zoomify                 run 4  ok          1.46 s     245 MB
zoomify/S/metacooler/zoomify                 run 5  ok          1.45 s     245 MB
zoomify/S/cooler/zoomify                     run 1  ok          3.85 s     306 MB
zoomify/S/cooler/zoomify                     run 2  ok          3.89 s     306 MB
zoomify/S/cooler/zoomify                     run 3  ok          3.92 s     306 MB
zoomify/S/cooler/zoomify                     run 4  ok          3.80 s     306 MB
zoomify/S/cooler/zoomify                     run 5  ok          3.78 s     306 MB
zoomify/M/metacooler/zoomify                 run 1  ok          6.32 s     986 MB
zoomify/M/metacooler/zoomify                 run 2  ok          6.31 s     986 MB
zoomify/M/metacooler/zoomify                 run 3  ok          6.29 s     986 MB
zoomify/M/metacooler/zoomify                 run 4  ok          6.28 s     986 MB
zoomify/M/metacooler/zoomify                 run 5  ok          6.30 s     986 MB
zoomify/M/cooler/zoomify                     run 1  ok         61.71 s    1382 MB
zoomify/M/cooler/zoomify                     run 2  ok         60.29 s    1382 MB
zoomify/M/cooler/zoomify                     run 3  ok         61.07 s    1377 MB
zoomify/L/metacooler/zoomify                 run 1  ok         21.33 s    2567 MB
zoomify/L/metacooler/zoomify                 run 2  ok         21.21 s    2568 MB
zoomify/L/metacooler/zoomify                 run 3  ok         21.32 s    2567 MB
zoomify/L/metacooler/zoomify                 run 4  ok         21.03 s    2567 MB
zoomify/L/metacooler/zoomify                 run 5  ok         20.90 s    2567 MB
zoomify/L/cooler/zoomify                     run 1  ok        916.69 s    2574 MB
zoomify/XL/metacooler/zoomify                run 1  ok        105.29 s    6731 MB
zoomify/XL/metacooler/zoomify                run 2  ok        104.90 s    6733 MB
zoomify/XL/metacooler/zoomify                run 3  ok        104.22 s    6715 MB
zoomify/XL/cooler/zoomify                    skipped: no base map from cooler cload pairs on XL
Code
show(wide("zoomify"))
Table 8: Zoomify: wall-clock seconds, mean ± sd (n runs).
tool task S M L XL
metacooler zoomify 1.45 ± 0.01 (n=5) 6.30 ± 0.02 (n=5) 21.2 ± 0.2 (n=5) 105 ± 1 (n=3)
cooler zoomify 3.85 ± 0.06 (n=5) 61.0 ± 0.7 (n=3) 917 (n=1) not run

5 · Querying a contig, a genome, a genome pair

One contig, one MAG and one pair of MAGs per dataset, picked by targets.py with a fixed seed (the pair in proportion to abundance × length, as the library joins genomes, so that it shares contacts). cooler serves a contig through its own index (dump -r). It has no notion of a genome. So a MAG’s pixels are the joined pixels whose two contigs the binning table puts in that MAG, in one pass of awk over the whole map. metacooler resolves the genome by name (dump -r/-r2).

%%bash
source bench.sh
in_genome='NR == FNR { if ($2 == g) keep[$1]; next } ($1 in keep) && ($4 in keep)'
in_pair='NR == FNR { if ($2 == a) A[$1]; if ($2 == b) B[$1]; next }'
in_pair+=' (($1 in A) && ($4 in B)) || (($1 in B) && ($4 in A))'
for ds in $DATASETS; do
    d=$DATA/$ds m=$MAPS/$ds q=$QUERIES/$ds
    read -r contig mag a b <<<"$(python targets.py --contigs "$d/contigs.tsv" \
        --sizes "$d/contigs.sizes" --genomes "$d/genomes.tsv")"
    echo "$ds: contig $contig · genome $mag · genome pair $a × $b"
    bench query "$ds" cooler contig "$q.cooler.contig.tsv" -- \
        cooler dump -t pixels -r "$contig" -o "$q.cooler.contig.tsv" "$m.cool"
    bench query "$ds" metacooler contig "$q.metacooler.contig.tsv" -- \
        metacooler dump "$m.metacool" -t pixels -r "$contig" -o "$q.metacooler.contig.tsv"
    bench query "$ds" cooler genome "$q.cooler.genome.tsv" -- bash -o pipefail -c \
        "$STAGE cooler dump -t pixels --join $m.cool | $STAGE awk -F'\t' -v g=$mag '$in_genome' $d/contigs.tsv - > $q.cooler.genome.tsv"
    bench query "$ds" metacooler genome "$q.metacooler.genome.tsv" -- \
        metacooler dump "$m.metacool" -t pixels -r "$mag" -o "$q.metacooler.genome.tsv"
    bench query "$ds" cooler "genome pair" "$q.cooler.pair.tsv" -- bash -o pipefail -c \
        "$STAGE cooler dump -t pixels --join $m.cool | $STAGE awk -F'\t' -v a=$a -v b=$b '$in_pair' $d/contigs.tsv - > $q.cooler.pair.tsv"
    bench query "$ds" metacooler "genome pair" "$q.metacooler.pair.tsv" -- \
        metacooler dump "$m.metacool" -t pixels -r "$a" -r2 "$b" -o "$q.metacooler.pair.tsv"
done
S: contig k141_648 · genome MAG_09 · genome pair MAG_05 × MAG_01
query/S/cooler/contig                        run 1  ok          0.52 s     150 MB
query/S/cooler/contig                        run 2  ok          0.51 s     150 MB
query/S/cooler/contig                        run 3  ok          0.52 s     150 MB
query/S/cooler/contig                        run 4  ok          0.51 s     150 MB
query/S/cooler/contig                        run 5  ok          0.50 s     151 MB
query/S/metacooler/contig                    run 1  ok          0.55 s     151 MB
query/S/metacooler/contig                    run 2  ok          0.53 s     151 MB
query/S/metacooler/contig                    run 3  ok          0.54 s     151 MB
query/S/metacooler/contig                    run 4  ok          0.53 s     151 MB
query/S/metacooler/contig                    run 5  ok          0.53 s     151 MB
query/S/cooler/genome                        run 1  ok          1.87 s     225 MB
query/S/cooler/genome                        run 2  ok          1.87 s     225 MB
query/S/cooler/genome                        run 3  ok          1.87 s     225 MB
query/S/cooler/genome                        run 4  ok          1.87 s     225 MB
query/S/cooler/genome                        run 5  ok          1.86 s     224 MB
query/S/metacooler/genome                    run 1  ok          0.58 s     157 MB
query/S/metacooler/genome                    run 2  ok          0.59 s     158 MB
query/S/metacooler/genome                    run 3  ok          0.58 s     157 MB
query/S/metacooler/genome                    run 4  ok          0.58 s     157 MB
query/S/metacooler/genome                    run 5  ok          0.58 s     157 MB
query/S/cooler/genome pair                   run 1  ok          1.95 s     225 MB
query/S/cooler/genome pair                   run 2  ok          1.95 s     225 MB
query/S/cooler/genome pair                   run 3  ok          1.95 s     225 MB
query/S/cooler/genome pair                   run 4  ok          1.94 s     225 MB
query/S/cooler/genome pair                   run 5  ok          1.95 s     224 MB
query/S/metacooler/genome pair               run 1  ok          0.57 s     166 MB
query/S/metacooler/genome pair               run 2  ok          0.56 s     166 MB
query/S/metacooler/genome pair               run 3  ok          0.56 s     166 MB
query/S/metacooler/genome pair               run 4  ok          0.56 s     166 MB
query/S/metacooler/genome pair               run 5  ok          0.59 s     166 MB
M: contig k141_13779 · genome MAG_43 · genome pair MAG_14 × MAG_01
query/M/cooler/contig                        run 1  ok          0.53 s     162 MB
query/M/cooler/contig                        run 2  ok          0.54 s     162 MB
query/M/cooler/contig                        run 3  ok          0.52 s     162 MB
query/M/cooler/contig                        run 4  ok          0.53 s     162 MB
query/M/cooler/contig                        run 5  ok          0.53 s     162 MB
query/M/metacooler/contig                    run 1  ok          0.56 s     158 MB
query/M/metacooler/contig                    run 2  ok          0.54 s     158 MB
query/M/metacooler/contig                    run 3  ok          0.55 s     158 MB
query/M/metacooler/contig                    run 4  ok          0.54 s     158 MB
query/M/metacooler/contig                    run 5  ok          0.54 s     158 MB
query/M/cooler/genome                        run 1  ok         14.12 s     295 MB
query/M/cooler/genome                        run 2  ok         14.21 s     289 MB
query/M/cooler/genome                        run 3  ok         14.19 s     308 MB
query/M/cooler/genome                        run 4  ok         14.35 s     300 MB
query/M/cooler/genome                        run 5  ok         14.25 s     297 MB
query/M/metacooler/genome                    run 1  ok          0.57 s     160 MB
query/M/metacooler/genome                    run 2  ok          0.57 s     160 MB
query/M/metacooler/genome                    run 3  ok          0.59 s     160 MB
query/M/metacooler/genome                    run 4  ok          0.57 s     160 MB
query/M/metacooler/genome                    run 5  ok          0.58 s     160 MB
query/M/cooler/genome pair                   run 1  ok         14.92 s     297 MB
query/M/cooler/genome pair                   run 2  ok         14.90 s     289 MB
query/M/cooler/genome pair                   run 3  ok         14.80 s     295 MB
query/M/cooler/genome pair                   run 4  ok         14.99 s     297 MB
query/M/cooler/genome pair                   run 5  ok         14.65 s     287 MB
query/M/metacooler/genome pair               run 1  ok          0.60 s     176 MB
query/M/metacooler/genome pair               run 2  ok          0.61 s     176 MB
query/M/metacooler/genome pair               run 3  ok          0.60 s     176 MB
query/M/metacooler/genome pair               run 4  ok          0.60 s     176 MB
query/M/metacooler/genome pair               run 5  ok          0.61 s     176 MB
L: contig k141_124750 · genome MAG_0171 · genome pair MAG_0059 × MAG_0010
query/L/cooler/contig                        run 1  ok          0.81 s     250 MB
query/L/cooler/contig                        run 2  ok          0.80 s     250 MB
query/L/cooler/contig                        run 3  ok          0.81 s     250 MB
query/L/cooler/contig                        run 4  ok          0.80 s     250 MB
query/L/cooler/contig                        run 5  ok          0.78 s     250 MB
query/L/metacooler/contig                    run 1  ok          0.73 s     207 MB
query/L/metacooler/contig                    run 2  ok          0.72 s     207 MB
query/L/metacooler/contig                    run 3  ok          0.72 s     207 MB
query/L/metacooler/contig                    run 4  ok          0.71 s     207 MB
query/L/metacooler/contig                    run 5  ok          0.71 s     207 MB
query/L/cooler/genome                        run 1  ok         69.77 s     474 MB
query/L/cooler/genome                        run 2  ok         72.30 s     418 MB
query/L/cooler/genome                        run 3  ok         74.86 s     430 MB
query/L/metacooler/genome                    run 1  ok          0.76 s     209 MB
query/L/metacooler/genome                    run 2  ok          0.76 s     209 MB
query/L/metacooler/genome                    run 3  ok          0.75 s     211 MB
query/L/metacooler/genome                    run 4  ok          0.76 s     209 MB
query/L/metacooler/genome                    run 5  ok          0.75 s     209 MB
query/L/cooler/genome pair                   run 1  ok         74.85 s     440 MB
query/L/cooler/genome pair                   run 2  ok         70.16 s     427 MB
query/L/cooler/genome pair                   run 3  ok         68.02 s     434 MB
query/L/metacooler/genome pair               run 1  ok          0.76 s     227 MB
query/L/metacooler/genome pair               run 2  ok          0.76 s     230 MB
query/L/metacooler/genome pair               run 3  ok          0.75 s     228 MB
query/L/metacooler/genome pair               run 4  ok          0.76 s     227 MB
query/L/metacooler/genome pair               run 5  ok          0.75 s     227 MB
XL: contig k141_1232184 · genome MAG_00859 · genome pair MAG_00229 × MAG_00056
query/XL/cooler/contig                       run 1  ok          4.19 s    1003 MB
query/XL/cooler/contig                       run 2  ok          4.19 s    1003 MB
query/XL/cooler/contig                       run 3  ok          4.18 s    1003 MB
query/XL/cooler/contig                       run 4  ok          4.14 s    1003 MB
query/XL/cooler/contig                       run 5  ok          4.16 s    1003 MB
query/XL/metacooler/contig                   run 1  ok          2.84 s     649 MB
query/XL/metacooler/contig                   run 2  ok          2.91 s     649 MB
query/XL/metacooler/contig                   run 3  ok          2.92 s     649 MB
query/XL/metacooler/contig                   run 4  ok          2.88 s     649 MB
query/XL/metacooler/contig                   run 5  ok          2.83 s     649 MB
query/XL/cooler/genome                       run 1  ok       1354.45 s    1207 MB
query/XL/metacooler/genome                   run 1  ok          2.36 s     648 MB
query/XL/metacooler/genome                   run 2  ok          2.36 s     648 MB
query/XL/metacooler/genome                   run 3  ok          2.36 s     648 MB
query/XL/metacooler/genome                   run 4  ok          2.35 s     648 MB
query/XL/metacooler/genome                   run 5  ok          2.34 s     648 MB
query/XL/cooler/genome pair                  run 1  ok       1402.17 s    1194 MB
query/XL/metacooler/genome pair              run 1  ok          2.42 s     648 MB
query/XL/metacooler/genome pair              run 2  ok          2.37 s     648 MB
query/XL/metacooler/genome pair              run 3  ok          2.37 s     648 MB
query/XL/metacooler/genome pair              run 4  ok          2.40 s     648 MB
query/XL/metacooler/genome pair              run 5  ok          2.40 s     648 MB
Code
def pixels(path):
    """Rows and contact total of a dump: the count is the last column."""
    if not path.exists() or path.stat().st_size == 0:
        return 0, 0
    last = pd.read_csv(path, sep="\t", header=None).iloc[:, -1]
    return len(last), int(last.sum())


SHARE = {}  # the share of the map's pixels each query returned: what the scan's cost is set against
agree = []
for ds in DATASETS:
    with h5py.File(MAPS / f"{ds}.metacool", "r") as f:
        nnz = int(f.attrs["nnz"])
    for name, file in (("contig", "contig"), ("genome", "genome"), ("genome pair", "pair")):
        got = {tool: pixels(QUERIES / f"{ds}.{tool}.{file}.tsv")
               for tool in ("metacooler", "cooler") if finished("query", ds, tool, name)}
        missing = [tool for tool in ("cooler", "metacooler") if tool not in got]
        n, total = next(iter(got.values()), (None, None))
        if n is not None:
            SHARE[ds, name] = n / nnz
        agree.append({
            "dataset": ds, "query": name,
            "pixels": "–" if n is None else f"{n:,}",
            "share of the map's pixels": "–" if n is None else f"{n / nnz:.3%}",
            "contacts": "–" if total is None else f"{total:,}",
            "same as cooler": f"{' and '.join(missing)} did not finish" if missing
                              else "yes" if got["cooler"] == got["metacooler"] else "NO",
        })
agree = pd.DataFrame(agree)
assert not (agree["same as cooler"] == "NO").any(), "the two tools returned different pixels"
show(wide("query"))
Table 9: Queries from the command line: wall-clock seconds, mean ± sd (n runs).
tool task S M L XL
cooler contig 0.512 ± 0.008 (n=5) 0.530 ± 0.007 (n=5) 0.800 ± 0.012 (n=5) 4.17 ± 0.02 (n=5)
metacooler contig 0.536 ± 0.009 (n=5) 0.546 ± 0.009 (n=5) 0.718 ± 0.008 (n=5) 2.88 ± 0.04 (n=5)
cooler genome 1.87 ± 0.00 (n=5) 14.2 ± 0.1 (n=5) 72.3 ± 2.5 (n=3) 1,354 (n=1)
metacooler genome 0.582 ± 0.004 (n=5) 0.576 ± 0.009 (n=5) 0.756 ± 0.005 (n=5) 2.35 ± 0.01 (n=5)
cooler genome pair 1.95 ± 0.00 (n=5) 14.9 ± 0.1 (n=5) 71.0 ± 3.5 (n=3) 1,402 (n=1)
metacooler genome pair 0.568 ± 0.013 (n=5) 0.604 ± 0.005 (n=5) 0.756 ± 0.005 (n=5) 2.39 ± 0.02 (n=5)
Code
show(agree)
Table 10: What each query returned (the two tools must agree), and the share of the map’s pixels that is: cooler’s scans read every pixel of the map, metacooler’s reads only these.
dataset query pixels share of the map's pixels contacts same as cooler
S contig 112 0.014% 206 yes
S genome 80,193 9.883% 91,849 yes
S genome pair 1,319 0.163% 1,319 yes
M contig 144 0.002% 198 yes
M genome 26,438 0.344% 28,989 yes
M genome pair 1,006 0.013% 1,006 yes
L contig 1 0.000% 2 yes
L genome 29,017 0.130% 30,248 yes
L genome pair 214 0.001% 214 yes
XL contig 3 0.000% 3 yes
XL genome 3,888 0.005% 3,941 yes
XL genome pair 11 0.000% 11 yes
Code
show(wide("query", "peak_mb", digits=0))
Table 11: Queries: peak memory (MB), mean ± sd (n runs); for cooler’s dump | awk pipelines, both stages added up.
tool task S M L XL
cooler contig 150 ± 0 (n=5) 162 ± 0 (n=5) 250 ± 0 (n=5) 1,003 ± 0 (n=5)
metacooler contig 151 ± 0 (n=5) 158 ± 0 (n=5) 207 ± 0 (n=5) 649 ± 0 (n=5)
cooler genome 225 ± 0 (n=5) 298 ± 7 (n=5) 441 ± 29 (n=3) 1,207 (n=1)
metacooler genome 157 ± 0 (n=5) 160 ± 0 (n=5) 209 ± 1 (n=5) 648 ± 0 (n=5)
cooler genome pair 225 ± 0 (n=5) 293 ± 5 (n=5) 434 ± 7 (n=3) 1,194 (n=1)
metacooler genome pair 166 ± 0 (n=5) 176 ± 0 (n=5) 228 ± 1 (n=5) 648 ± 0 (n=5)

6 · Genome-level aggregation

The genome × genome contact network: metacooler network --level genome, against the same edge list from cooler (dump --join, then an awk sum over the binning table). And the aggregated genome-level .cool (metacooler aggregate --level genome), which cooler has no command for.

%%bash
source bench.sh
by_genome='NR == FNR { g[$1] = $2; next }'
by_genome+=' { a = g[$1]; b = g[$4]; if (a > b) { t = a; a = b; b = t }; n[a "\t" b] += $7 }'
by_genome+=' END { for (k in n) print k "\t" n[k] }'
for ds in $DATASETS; do
    d=$DATA/$ds m=$MAPS/$ds q=$QUERIES/$ds
    bench aggregate "$ds" cooler "genome network" "$q.cooler.network.tsv" -- bash -o pipefail -c \
        "$STAGE cooler dump -t pixels --join $m.cool | $STAGE awk -F'\t' '$by_genome' $d/contigs.tsv - > $q.cooler.network.tsv"
    bench aggregate "$ds" metacooler "genome network" "$q.metacooler.network.tsv" -- \
        metacooler network "$m.metacool" --level genome -o "$q.metacooler.network.tsv"
    bench aggregate "$ds" metacooler "genome-level .cool" "$SCRATCH/$ds.genomes.cool" -- \
        metacooler aggregate "$m.metacool" --level genome -o "$SCRATCH/$ds.genomes.cool"
done
aggregate/S/cooler/genome network            run 1  ok          2.50 s     225 MB
aggregate/S/cooler/genome network            run 2  ok          2.38 s     225 MB
aggregate/S/cooler/genome network            run 3  ok          2.36 s     225 MB
aggregate/S/cooler/genome network            run 4  ok          2.38 s     225 MB
aggregate/S/cooler/genome network            run 5  ok          2.39 s     225 MB
aggregate/S/metacooler/genome network        run 1  ok          0.63 s     205 MB
aggregate/S/metacooler/genome network        run 2  ok          0.63 s     206 MB
aggregate/S/metacooler/genome network        run 3  ok          0.62 s     206 MB
aggregate/S/metacooler/genome network        run 4  ok          0.62 s     205 MB
aggregate/S/metacooler/genome network        run 5  ok          0.62 s     206 MB
aggregate/S/metacooler/genome-level .cool    run 1  ok          0.66 s     211 MB
aggregate/S/metacooler/genome-level .cool    run 2  ok          0.66 s     212 MB
aggregate/S/metacooler/genome-level .cool    run 3  ok          0.68 s     212 MB
aggregate/S/metacooler/genome-level .cool    run 4  ok          0.66 s     212 MB
aggregate/S/metacooler/genome-level .cool    run 5  ok          0.66 s     212 MB
aggregate/M/cooler/genome network            run 1  ok         19.43 s     304 MB
aggregate/M/cooler/genome network            run 2  ok         19.42 s     308 MB
aggregate/M/cooler/genome network            run 3  ok         19.38 s     308 MB
aggregate/M/cooler/genome network            run 4  ok         19.63 s     297 MB
aggregate/M/cooler/genome network            run 5  ok         19.69 s     308 MB
aggregate/M/metacooler/genome network        run 1  ok          1.31 s     631 MB
aggregate/M/metacooler/genome network        run 2  ok          1.30 s     631 MB
aggregate/M/metacooler/genome network        run 3  ok          1.30 s     630 MB
aggregate/M/metacooler/genome network        run 4  ok          1.31 s     631 MB
aggregate/M/metacooler/genome network        run 5  ok          1.30 s     631 MB
aggregate/M/metacooler/genome-level .cool    run 1  ok          1.37 s     686 MB
aggregate/M/metacooler/genome-level .cool    run 2  ok          1.37 s     687 MB
aggregate/M/metacooler/genome-level .cool    run 3  ok          1.38 s     687 MB
aggregate/M/metacooler/genome-level .cool    run 4  ok          1.35 s     686 MB
aggregate/M/metacooler/genome-level .cool    run 5  ok          1.38 s     686 MB
aggregate/L/cooler/genome network            run 1  ok         88.55 s     533 MB
aggregate/L/cooler/genome network            run 2  ok         89.25 s     552 MB
aggregate/L/cooler/genome network            run 3  ok         83.87 s     548 MB
aggregate/L/metacooler/genome network        run 1  ok          2.98 s    1575 MB
aggregate/L/metacooler/genome network        run 2  ok          2.93 s    1575 MB
aggregate/L/metacooler/genome network        run 3  ok          2.94 s    1575 MB
aggregate/L/metacooler/genome network        run 4  ok          2.98 s    1575 MB
aggregate/L/metacooler/genome network        run 5  ok          2.99 s    1575 MB
aggregate/L/metacooler/genome-level .cool    run 1  ok          3.06 s    1739 MB
aggregate/L/metacooler/genome-level .cool    run 2  ok          3.04 s    1740 MB
aggregate/L/metacooler/genome-level .cool    run 3  ok          3.10 s    1740 MB
aggregate/L/metacooler/genome-level .cool    run 4  ok          3.09 s    1739 MB
aggregate/L/metacooler/genome-level .cool    run 5  ok          3.09 s    1739 MB
aggregate/XL/cooler/genome network           run 1  ok       1534.85 s    2269 MB
aggregate/XL/metacooler/genome network       run 1  ok         12.38 s    3899 MB
aggregate/XL/metacooler/genome network       run 2  ok         12.38 s    3898 MB
aggregate/XL/metacooler/genome network       run 3  ok         12.30 s    3899 MB
aggregate/XL/metacooler/genome network       run 4  ok         12.24 s    3898 MB
aggregate/XL/metacooler/genome network       run 5  ok         12.31 s    3899 MB
aggregate/XL/metacooler/genome-level .cool   run 1  ok         12.92 s    4259 MB
aggregate/XL/metacooler/genome-level .cool   run 2  ok         13.05 s    4259 MB
aggregate/XL/metacooler/genome-level .cool   run 3  ok         12.88 s    4259 MB
aggregate/XL/metacooler/genome-level .cool   run 4  ok         13.00 s    4259 MB
aggregate/XL/metacooler/genome-level .cool   run 5  ok         13.06 s    4259 MB
Code
def edges(path):
    e = pd.read_csv(path, sep="\t", header=None, names=["a", "b", "n"], keep_default_na=False)
    e[["a", "b"]] = np.sort(e[["a", "b"]].to_numpy(), axis=1)
    return e.groupby(["a", "b"])["n"].sum()

check = []
for ds in DATASETS:
    nets = {tool: edges(QUERIES / f"{ds}.{tool}.network.tsv").sort_index()
            for tool in ("metacooler", "cooler") if finished("aggregate", ds, tool, "genome network")}
    missing = [tool for tool in ("cooler", "metacooler") if tool not in nets]
    net = next(iter(nets.values()), None)
    check.append({
        "dataset": ds,
        "edges": "–" if net is None else f"{len(net):,}",
        "contacts": "–" if net is None else f"{net.sum():,}",
        "same as cooler": f"{' and '.join(missing)} did not finish" if missing
                          else "yes" if nets["cooler"].equals(nets["metacooler"]) else "NO",
    })
check = pd.DataFrame(check)
assert not (check["same as cooler"] == "NO").any(), "the two networks differ"
display(show(wide("aggregate")))
display(show(wide("aggregate", "peak_mb", digits=0).rename(columns={"task": "task (peak MB)"})))
show(check)
Table 12: Genome-level aggregation: wall-clock seconds, mean ± sd (n runs), peak memory, and the edges each network has (they must agree).
tool task S M L XL
cooler genome network 2.40 ± 0.06 (n=5) 19.5 ± 0.1 (n=5) 87.2 ± 2.9 (n=3) 1,535 (n=1)
metacooler genome network 0.624 ± 0.005 (n=5) 1.30 ± 0.01 (n=5) 2.96 ± 0.03 (n=5) 12.3 ± 0.1 (n=5)
metacooler genome-level .cool 0.664 ± 0.009 (n=5) 1.37 ± 0.01 (n=5) 3.08 ± 0.03 (n=5) 13.0 ± 0.1 (n=5)
tool task (peak MB) S M L XL
cooler genome network 225 ± 0 (n=5) 305 ± 5 (n=5) 544 ± 10 (n=3) 2,269 (n=1)
metacooler genome network 206 ± 1 (n=5) 631 ± 0 (n=5) 1,575 ± 0 (n=5) 3,899 ± 1 (n=5)
metacooler genome-level .cool 212 ± 0 (n=5) 686 ± 1 (n=5) 1,739 ± 1 (n=5) 4,259 ± 0 (n=5)
dataset edges contacts same as cooler
S 64 1,000,000 yes
M 2,612 10,000,000 yes
L 96,536 30,000,000 yes
XL 1,237,358 100,000,000 yes

Summary

Code
t = timings()
RAM = ram(t)
PANELS = [
    ("create", ["cload pairs", "create pairs"], "create a map"),
    ("zoomify", ["zoomify"], "zoomify"),
    ("query", ["contig"], "dump a contig"),
    ("query", ["genome"], "dump a genome"),
    ("query", ["genome pair"], "dump a genome pair"),
    ("aggregate", ["genome network"], "genome network"),
]
STORE_STYLE = {
    ".metacool": dict(color=HUE["metacooler"], marker="o"),
    ".cool + binning tables": dict(color=HUE["cooler"], marker="o"),
    ".cool alone (no genomes)": dict(color=HUE["cooler"], marker="o", ls=(0, (1, 1.5)), mfc=SURFACE),
    "one .cool per MAG": dict(color=HUE["cooler"], marker="D", ls=(0, (6, 2)), ms=4),
}


def memory_at(ax, ds):
    """Peak memory of every command on one dataset, cooler and metacooler on one row."""
    rows = runs(t, dataset=ds)
    top = 0.0
    for i, (section, tasks, title) in enumerate(PANELS):
        y = len(PANELS) - 1 - i
        at = {}
        for tool in ("cooler", "metacooler"):
            r = rows[(rows["section"] == section) & rows["task"].isin(tasks) & (rows["tool"] == tool)]
            if not len(r):
                continue
            ok, last = r[r["status"] == "ok"], r["status"].iloc[-1]
            if len(ok):
                at[tool] = ok["peak_gb"].max()
                ax.scatter([at[tool]], [y], s=40, color=HUE[tool], zorder=3)
            elif last in ("timeout", "oom") and r["peak_gb"].notna().any():
                at[tool] = r["peak_gb"].max()
                ax.scatter([at[tool]], [y], marker="|" if last == "timeout" else "X", s=90,
                           color=HUE[tool], lw=2.5, zorder=3)
                if last == "timeout":
                    ax.annotate("", xy=(at[tool] * 1.25 + 0.3, y), xytext=(at[tool], y), zorder=3,
                                arrowprops=dict(arrowstyle="-|>", color=HUE[tool], lw=1.5, mutation_scale=9))
            elif last == "skipped":
                ax.annotate(f"{tool} not run", (0.02, y), xycoords=("axes fraction", "data"),
                            xytext=(0, -9), textcoords="offset points", color=INK2, fontsize=7.5)
        if len(at) == 2:
            ax.plot(list(at.values()), [y, y], color=GRID, lw=4, zorder=1, solid_capstyle="round")
        top = max([top, *at.values()])
    ax.set_yticks(range(len(PANELS)), [title for _, _, title in reversed(PANELS)])
    ax.set_ylim(-0.7, len(PANELS) - 0.3)
    ax.grid(axis="y", visible=False)
    ram_line(ax, RAM, vertical=True)
    ax.set_xlim(0, max(top, RAM if RAM == RAM else 0) * 1.12)
    ax.set_xlabel(f"peak memory on {ds} (GB)")


fig, axes = plt.subplots(3, 3, figsize=(10.5, 10.4), layout="constrained")
axes = axes.ravel()
for ax, letter, (section, tasks, title) in zip(axes, "ABCDEF", PANELS):
    rows = t[(t["section"] == section) & t["task"].isin(tasks)]
    scaling(ax, rows)
    seconds_axis(ax)
    limit_line(ax, rows)
    ax.set_title(f"{letter}   {title}")
    if ax not in (axes[0], axes[3]):
        ax.set_ylabel("")
    if ax in axes[:3]:
        ax.set_xlabel("")
axes[0].legend(handles=TOOLS_LEGEND, loc="upper left")

g = axes[6]
create = runs(t, section="create")
scaling(g, create, "peak_gb")
log_y(g, "peak memory (GB)")
ram_line(g, RAM)
g.set_title("G   memory: create a map")

memory_at(axes[7], DATASETS[-1])
axes[7].set_title(f"H   memory: every command, {DATASETS[-1]}")

i = axes[8]
for option, r in STORAGE[STORAGE["option"].isin(list(STORE_STYLE))].groupby("option", sort=False):
    i.plot(r["contigs"], r["mb"], label=option, **{"ms": 5, **STORE_STYLE[option]})
contig_axis(i)
log_y(i, "size on disk (MB)")
i.set_title("I   size on disk")
i.legend(loc="upper left", fontsize=7.5)
save(fig, "commands")
Figure 3: Every command on the four simulated datasets (simulate.py; no real data), cooler in orange, metacooler in blue. A–F, wall clock against the number of contigs: a line joins the means of a command’s runs, each run is a dot and the bar spans them. A run stopped by timeout at the time limit (dotted line) is a bar with an arrow up, drawn where it was killed: a measured lower bound, which no line joins and nothing extrapolates; a command not run is marked so. Both are named below the figure. G, peak memory of map creation against the number of contigs; H, peak memory of every command on the largest dataset (an arrow marks the memory a stopped run held when stopped). Dashed in G and H, the RAM of the machine the runs were on. I, size on disk of the map with its genomes: one .metacool, the .cool with its binning tables or alone, or one .cool per MAG (section 2, which also gives what each holds).

Every run finished within the 30 min time limit.

Not run. cooler cload pairs on XL: it has never finished on XL within the time limit (BENCH_SKIP_COOLER_CREATE); cooler zoomify on XL: no base map from cooler cload pairs on XL.

How many times faster metacooler runs than cooler (wall clock, means over runs); > where cooler was stopped at the time limit, which makes the ratio a lower bound. cooler’s pyramid is not run where it could not build the base map.

Code
FOLDS = [
    ("create", "cload pairs", "create pairs", "create a map from pairs"),
    ("zoomify", "zoomify", "zoomify", "zoomify"),
    ("query", "contig", "contig", "dump a contig"),
    ("query", "genome", "genome", "dump a genome"),
    ("query", "genome pair", "genome pair", "dump a genome pair"),
    ("aggregate", "genome network", "genome network", "genome network"),
]


def fold(section, cooler_task, metacooler_task, ds):
    """cooler's seconds over metacooler's, and how cooler's run ended: "ok",
    "timeout" (the ratio is then a lower bound), "oom" or "skipped" (no ratio)."""
    pick = runs(t, section=section, dataset=ds)
    c = pick[(pick["tool"] == "cooler") & (pick["task"] == cooler_task)]
    m = pick[(pick["tool"] == "metacooler") & (pick["task"] == metacooler_task) & (pick["status"] == "ok")]
    if c.empty or m.empty:
        return None
    if (c["status"] == "ok").any():
        return c.loc[c["status"] == "ok", "wall_s"].mean() / m["wall_s"].mean(), "ok"
    if (c["status"] == "timeout").any():
        return c["wall_s"].max() / m["wall_s"].mean(), "timeout"
    for status in ("oom", "skipped"):
        if (c["status"] == status).any():
            return None, status
    return None


def fold_text(f):
    match f:
        case None:
            return ""
        case value, "ok":
            return f"{value:,.1f}×"
        case value, "timeout":
            return f"> {value:,.1f}×"
        case _, "skipped":
            return "cooler not run"
        case _:
            return "cooler out of memory"


rows = []
for section, ct, mt, label in FOLDS:
    row = {"": label}
    for ds in DATASETS:
        row[ds] = fold_text(fold(section, ct, mt, ds))
    rows.append(row)
FOLD_TABLE = pd.DataFrame(rows)
FOLD_TABLE.to_csv(RESULTS / "speedups.tsv", sep="\t", index=False)
show(FOLD_TABLE)
Table 13: Speed-up: cooler’s wall clock over metacooler’s.
S M L XL
create a map from pairs 1.5× 2.9× 20.4× cooler not run
zoomify 2.7× 9.7× 43.3× cooler not run
dump a contig 1.0× 1.0× 1.1× 1.5×
dump a genome 3.2× 24.7× 95.6× 575.4×
dump a genome pair 3.4× 24.6× 93.9× 586.2×
genome network 3.8× 15.0× 29.4× 124.6×

Replicates

How much the runs of a command vary: the number of runs n of each command on each dataset, and their coefficient of variation (sd / mean). A command runs at least 3 times unless a single run takes longer than the threshold below, or it is stopped at the time limit.

Code
ok = t[t["status"] == "ok"]
REPS = (
    ok.groupby(["section", "tool", "task", "dataset"], sort=False)["wall_s"]
    .agg(n="count", mean="mean", sd="std", fastest="min", slowest="max")
    .reset_index()
)
REPS["cv"] = REPS["sd"] / REPS["mean"]
REPS.to_csv(RESULTS / "replicates.tsv", sep="\t", index=False)
REPS["cell"] = [f"n={n} · {cv:.1%}" if n > 1 else f"n=1 · {human(mean)}"
                for n, cv, mean in zip(REPS["n"], REPS["cv"], REPS["mean"])]
grid = REPS.pivot(index=["section", "tool", "task"], columns="dataset", values="cell")
grid = grid.reindex(list(dict.fromkeys(zip(REPS["section"], REPS["tool"], REPS["task"]))))
show(grid.reindex(columns=[d for d in DATASETS if d in grid.columns]).reset_index())
Table 14: Runs per command and dataset: n, and the coefficient of variation of their wall clock (sd / mean). One run: its time instead.
section tool task S M L XL
create cooler cload pairs n=5 · 0.8% n=5 · 1.3% n=3 · 2.0%
create metacooler create pairs n=5 · 6.0% n=5 · 0.9% n=5 · 2.0% n=3 · 7.3%
convert metacooler create cool n=5 · 0.8% n=5 · 0.4% n=5 · 1.1% n=3 · 1.1%
convert metacooler flatten n=5 · 0.6% n=5 · 0.4% n=5 · 0.8% n=3 · 1.1%
zoomify metacooler zoomify n=5 · 0.9% n=5 · 0.3% n=5 · 0.9% n=3 · 0.5%
zoomify cooler zoomify n=5 · 1.5% n=3 · 1.2% n=1 · 15.3 min
query cooler contig n=5 · 1.6% n=5 · 1.3% n=5 · 1.5% n=5 · 0.5%
query metacooler contig n=5 · 1.7% n=5 · 1.6% n=5 · 1.2% n=5 · 1.4%
query cooler genome n=5 · 0.2% n=5 · 0.6% n=3 · 3.5% n=1 · 22.6 min
query metacooler genome n=5 · 0.8% n=5 · 1.6% n=5 · 0.7% n=5 · 0.4%
query cooler genome pair n=5 · 0.2% n=5 · 0.9% n=3 · 4.9% n=1 · 23.4 min
query metacooler genome pair n=5 · 2.3% n=5 · 0.9% n=5 · 0.7% n=5 · 0.9%
aggregate cooler genome network n=5 · 2.3% n=5 · 0.7% n=3 · 3.4% n=1 · 25.6 min
aggregate metacooler genome network n=5 · 0.9% n=5 · 0.4% n=5 · 0.9% n=5 · 0.5%
aggregate metacooler genome-level .cool n=5 · 1.3% n=5 · 0.9% n=5 · 0.8% n=5 · 0.6%

54 of 58 command × dataset combinations ran at least 3 times; across them the coefficient of variation has a median of 0.9% and a maximum of 7.3% (metacooler create pairs on XL). 4 ran once, each for over 10 min: cooler zoomify on L, 15.3 min; cooler dump --join | awk (a genome) on XL, 22.6 min; cooler dump --join | awk (a genome pair) on XL, 23.4 min; cooler dump --join | awk (genome network) on XL, 25.6 min.

Numbers for the text

Filled in from this render; the sentences follow the manuscript draft.

  • All benchmarks are on simulated data: four communities drawn by simulate.py (generator v1) from a fixed seed each, of 1,000 to 2,000,000 contigs. No real reads, assembly or binning is used, and there are no accessions.
  • metacooler 0.2.0 creates metacool files 20-fold faster than cooler v0.12.1 creates the equivalent .cool file for an assembly of 200,000 contigs (19.7 s against 6.7 min), while also ingesting and indexing the /genomes table.
  • At 2,000,000 contigs (20,000 genomes), cooler cload pairs was not run in this render: it has never finished on XL within the time limit (BENCH_SKIP_COOLER_CREATE); metacooler took 110.9 s (peak 6.1 GB).
  • Every command ran on one machine with 7.7 GB of RAM (2 cores). On 2,000,000 contigs, the largest peak of each tool, among the runs that finished: cooler dump --join | awk (genome network), 2.2 GB and metacooler zoomify, 6.6 GB.
  • On disk, at 2,000,000 contigs: 161 MB for the .metacool, 277 MB for the .cool and its binning tables (205 MB for the .cool alone), and 142 MB for 1,000 per-MAG .cool files, which hold only 68% of the contacts and 16% of the contigs.
  • With cooler’s command line, extracting a single MAG from a genome-agnostic .cool file is a linear scan of every pixel (cooler dump --join, filtered against the binning table): 1.9 s at 1,000, 14.2 s at 20,000, 72.3 s at 200,000 and 22.6 min at 2,000,000 contigs. metacooler dump -r <MAG> takes 582 ms at 1,000, 576 ms at 20,000, 756 ms at 200,000 and 2.4 s at 2,000,000 contigs. On the simulated 2,000,000-contig map that is 575.4×. cooler’s scan reads every pixel of the map, metacooler only the MAG’s (0.01% of them here), so the fold depends on how a map’s pixels split between its genomes: here the simulated community sets that, not a real sample.
  • A full genome-by-genome aggregated matrix is computed in 13 s for a simulated community of 20,000 genomes (1,000 MAGs and 19,000 small bins; 2,000,000 contigs, 100 M read pairs; metacooler aggregate --level genome). Its edge list takes 12.3 s (metacooler network --level genome), against 25.6 min for cooler’s scan.
  • Timings are means over 3–5 runs (one, for a run over 10 min); their coefficient of variation has a median of 0.9%.

Artifacts

Beside this page, in _results/: published with it on the documentation site, and attached to every release and workflow run.

Code
t = timings()
problems = t[t["status"] == "error"]
if len(problems):
    display(Markdown(f"**{len(problems)} run(s) failed** — see their logs in `_work/logs/`:"))
    display(show(problems[["section", "dataset", "tool", "task", "command"]]))


def link(path):
    """A file's name, linked when the page can reach it: written beside the page (the
    notebook's directory), as _results/ is unless BENCH_RESULTS moves it."""
    name, href = str(path.relative_to(RESULTS)), os.path.relpath(path)
    return name if href.startswith("..") else f'<a href="{href}">{name}</a>'


files = sorted(p for p in RESULTS.rglob("*") if p.is_file())
show(pd.DataFrame({
    "file": [link(p) for p in files],
    "kB": [p.stat().st_size / 1e3 for p in files],
}), **{"kB": "{:,.1f}"})
file what
timings.tsv every timed run: command, status, wall/CPU seconds, peak memory, output size, versions, machine
datasets.tsv the datasets table above
storage.tsv the size on disk of each way to keep the map, per dataset
speedups.tsv the fold table above
replicates.tsv per command and dataset: runs, mean, sd, fastest, slowest, coefficient of variation
figures/ every figure, PNG and SVG