Rendered 16 November 2026, 22:46 UTC by GitHub Actions (push, v0.2.0) · metacooler 0.2.0 at b0cf6ae · cooler 0.12.1 · h5py 3.16.0 (HDF5 2.0.0), numpy 2.5.4, pandas 3.1.1, pyarrow 26.0.1 · Python 3.13.18, Quarto 1.10.19
On AMD EPYC 7763 64-Core Processor, 2 cores, 7.7 GB, Linux 7.0.0-1017-azure · simulated datasets from the release benchmark-data-v1
metacooler vs cooler: benchmarks
Building, storing, converting, zooming, querying and aggregating metagenomic Hi-C maps from the command line, on simulated communities of 10³ to 2 × 10⁶ contigs
Every command below runs as a fresh process, from bash, as a user would type it, under GNU time and timeout. Each run is one row of timings.tsv; every table and figure is drawn from it. cooler and metacooler get the same inputs and must produce the same contacts — that is checked, not assumed. cooler is version 0.12.1, the one uv.lock pins; bench.sh names it, and the settings cell refuses to time any other.
- Data. All four datasets are simulated, by
simulate.pyfrom a fixed seed each. No real reads, assembly or binning is used, so there are no accessions. See Datasets for the model, and what it decides. - Replicates. Every command runs at least 3 times, and up to 5 while its runs add up to under 2 min. A command whose single run takes over 10 min runs once (they are listed in Replicates). Tables give the mean ± sd and the number of runs n; figures draw every run, and the range of its runs as a bar.
- Measures. Wall clock; CPU (user + system, children included); peak memory, the largest resident set among the command’s processes — for cooler’s
dump | awkpipelines, both stages’ added up, since they run side by side. - Time limit. A run still going after the time limit (30 min) is stopped by
timeout. Its time and memory are where it was stopped: measured lower bounds, never extrapolations. A run the kernel kills for want of memory is reported as out of memory.cooler cload pairsis not run on XL, where it has never finished within the limit (see §1).
# Render (runs the benchmarks: hours for the full ladder; minutes for S and M)
uv sync && source .venv/bin/activate
cd benchmarks && quarto render benchmarks.qmd
BENCH_DATASETS="S M" quarto render benchmarks.qmd # a quick lookbench.sh holds the settings and the one timing function, bench; simulate.py makes the datasets; targets.py picks what the queries ask for. A second render re-uses runs already in timings.tsv (BENCH_REDO=1 times them again). The Benchmarks workflow renders this page for every release, and on demand, and publishes it with the metacooler documentation.
Setup: paths, colours, table and figure helpers
import json
import os
from pathlib import Path
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from IPython.display import Markdown, display
from matplotlib.lines import Line2D
from matplotlib.ticker import FuncFormatter, LogLocator, NullLocator
WORK = Path(os.environ.get("BENCH_WORK", "_work")).resolve()
RESULTS = Path(os.environ.get("BENCH_RESULTS", "_results")).resolve()
DATA, MAPS, QUERIES = WORK / "data", WORK / "maps", WORK / "queries"
FIGURES = RESULTS / "figures"
FIGURES.mkdir(parents=True, exist_ok=True)
DATASETS = os.environ.get("BENCH_DATASETS", "S M L XL").split()
TIMEOUT = float(os.environ.get("BENCH_TIMEOUT", 1800))
LONG = float(os.environ.get("BENCH_LONG", 600))
# Colour is identity: a tool keeps its hue in every figure (validated for
# colour-blind separation); the datasets, ordered by size, one hue light to dark.
HUE = {"metacooler": "#2a78d6", "cooler": "#eb6834"}
SIZE = dict(zip(["S", "M", "L", "XL"], ["#86b6ef", "#3987e5", "#1c5cab", "#0d366b"]))
INK, INK2, GRID, SURFACE = "#0b0b0b", "#52514e", "#e7e6e2", "#fcfcfb"
plt.rcParams.update({
"figure.facecolor": SURFACE, "axes.facecolor": SURFACE, "savefig.facecolor": SURFACE,
"axes.edgecolor": INK2, "axes.labelcolor": INK, "xtick.color": INK2, "ytick.color": INK2,
"text.color": INK, "axes.spines.top": False, "axes.spines.right": False,
"axes.grid": True, "grid.color": GRID, "grid.linewidth": 0.6, "axes.axisbelow": True,
"lines.linewidth": 2, "font.size": 9, "axes.titlesize": 10, "axes.titleweight": "bold",
"axes.titlelocation": "left", "legend.frameon": False,
})
def save(fig, name):
"""Show a figure; keep it as PNG and SVG beside the tables."""
fig.savefig(FIGURES / f"{name}.png", dpi=200)
fig.savefig(FIGURES / f"{name}.svg")
plt.show()
def show(frame, **formats):
"""A table without row numbers."""
return frame.style.hide(axis="index").format(formats or None, na_rep="")
def timings():
"""Every run, one row each, with the size of its dataset."""
t = pd.read_csv(RESULTS / "timings.tsv", sep="\t")
t = t[t["dataset"].isin(DATASETS)].copy()
t["contigs"] = t["dataset"].map(CONTIGS)
t["peak_gb"] = t["peak_mb"] / 1024
return t
def runs(t, **match):
"""The rows of t whose columns equal the values given."""
keep = np.ones(len(t), bool)
for column, value in match.items():
keep &= (t[column] == value).to_numpy()
return t[keep]
def finished(section, dataset, tool, task):
"""Whether a command's last run finished: its output is there to check."""
r = runs(timings(), section=section, dataset=dataset, tool=tool, task=task)
r = r[r["status"] != "skipped"]
return len(r) > 0 and r["status"].iloc[-1] == "ok"
def ram(t):
"""The memory of the machine the runs were on (GB)."""
known = pd.to_numeric(t["memory_gb"], errors="coerce").dropna()
return float(known.iloc[0]) if len(known) else float("nan")
def digits_for(x):
"""As many decimals as a number's size needs: 0.452, 4.52, 45.2, 452."""
x = abs(x)
return 3 if x < 1 else 2 if x < 10 else 1 if x < 100 else 0
def human(s):
"""Seconds as ms, s or min: 650 ms, 61.3 s, 30 min."""
if s is None or s != s:
return "–"
value, unit = (s * 1e3, "ms") if s < 1 else (s, "s") if s < 120 else (s / 60, "min")
return f"{value:,.0f} ms" if unit == "ms" else f"{value:,.1f}".removesuffix(".0") + f" {unit}"
def cell(rows, value="wall_s", digits=None, spread=True):
"""Mean ± sd (n) over the runs that finished; else what became of the command."""
ok = rows.loc[rows["status"] == "ok", value]
if len(ok):
d = digits_for(ok.mean()) if digits is None else digits
text = f"{ok.mean():,.{d}f}"
if spread:
text += (f" ± {ok.std():,.{d}f}" if len(ok) > 1 else "") + f" (n={len(ok)})"
return text
if not len(rows):
return ""
last = rows.iloc[-1]
if last["status"] == "timeout":
if value == "wall_s":
return f"> {last['wall_s']:,.0f} (stopped at the time limit)"
if value == "peak_mb":
return f"≥ {last[value]:,.0f} (when stopped)" if last[value] == last[value] else "not measured (stopped)"
return "–"
if last["status"] == "oom":
if value == "wall_s":
return f"out of memory after {last[value]:,.0f} s"
return f"out of memory at {last[value]:,.0f}" if value == "peak_mb" else "–"
return {"skipped": "not run", "error": "FAILED"}.get(last["status"], last["status"])
def wide(section, value="wall_s", digits=None, spread=True):
"""One row per command, one column per dataset."""
t = runs(timings(), section=section)
keys = ["tool", "task", "dataset"]
cells = pd.DataFrame(
[(*k, cell(g, value, digits, spread)) for k, g in t.groupby(keys, sort=False)],
columns=[*keys, "cell"],
)
out = cells.pivot(index=["tool", "task"], columns="dataset", values="cell")
out = out.reindex(list(dict.fromkeys(zip(cells["tool"], cells["task"]))))
return out.reindex(columns=[d for d in DATASETS if d in out.columns]).reset_index()
def compact(n):
"""1,000 as 1k, 2,000,000 as 2M: tick labels that fit three panels abreast."""
for div, unit in ((1e6, "M"), (1e3, "k")):
if n >= div:
return f"{n / div:.3g}{unit}"
return f"{n:,}"
def contig_axis(ax):
"""Datasets along x, at their contig counts."""
ax.set_xscale("log")
ax.xaxis.set_minor_locator(NullLocator())
ax.set_xticks(
[CONTIGS[d] for d in DATASETS],
[f"{d}\n{compact(CONTIGS[d])}" for d in DATASETS],
)
ax.set_xlim(min(CONTIGS.values()) / 2.5, max(CONTIGS.values()) * 2.5)
ax.set_xlabel("contigs in the assembly")
def log_y(ax, label):
"""A log axis labelled at 1, 2, 5 of each decade, so that a short range still reads."""
ax.set_yscale("log")
ax.yaxis.set_major_locator(LogLocator(subs=(1, 2, 5)))
ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:g}"))
ax.yaxis.set_minor_locator(NullLocator())
ax.set_ylabel(label)
def seconds_axis(ax, unit="s"):
log_y(ax, f"wall clock ({unit})")
def stopped(ax, x, y, colour, status):
"""A run that did not finish, where it ended. Stopped at the time limit: a
bar with an arrow up, since it would have taken longer -- a lower bound,
measured where timeout killed it. Killed for want of memory: a cross."""
if status == "timeout":
ax.scatter([x], [y], marker="_", s=160, color=colour, lw=2.5, zorder=5)
ax.annotate("", xy=(x, y * 2.4), xytext=(x, y), zorder=5,
arrowprops=dict(arrowstyle="-|>", color=colour, lw=1.5, mutation_scale=10))
ax.annotate("stopped at\nthe time limit", (x, y * 1.55), xytext=(-7, 0), textcoords="offset points",
ha="right", va="center", color=INK2, fontsize=7.5, zorder=6,
bbox=dict(boxstyle="square,pad=0.1", fc=SURFACE, ec="none"))
ax.update_datalim([(x, y * 2.8)])
else:
ax.scatter([x], [y], marker="X", s=60, color=colour, zorder=5)
ax.annotate("out of memory", (x, y), xytext=(0, 9), textcoords="offset points",
ha="center", color=INK2, fontsize=7.5)
def scaling(ax, rows, value="wall_s"):
"""One line per command through the means of its runs; every run a dot,
and a bar from its fastest to its slowest. A run that did not finish is
drawn where it ended (see `stopped`), and no line joins it: nothing is
extrapolated. A command not run is marked so."""
for (tool, _task), r in rows.groupby(["tool", "task"], sort=False):
colour = HUE[tool]
ok = r[r["status"] == "ok"]
stats = ok.groupby("contigs")[value].agg(["mean", "min", "max"])
if len(stats):
ax.errorbar(stats.index, stats["mean"],
yerr=[stats["mean"] - stats["min"], stats["max"] - stats["mean"]],
color=colour, marker="o", ms=5, capsize=2.5, elinewidth=1, zorder=3)
ax.scatter(ok["contigs"], ok[value], s=9, color=colour, alpha=0.35, lw=0, zorder=4)
for _, run in r[r["status"].isin(["timeout", "oom"]) & r[value].notna()].iterrows():
stopped(ax, run["contigs"], run[value], colour, run["status"])
for x in r.loc[r["status"] == "skipped", "contigs"]:
ax.annotate(f"{tool}\nnot run", (x, 0.03), xycoords=("data", "axes fraction"),
ha="center", va="bottom", color=colour, fontsize=7.5)
contig_axis(ax)
def limit_line(ax, rows):
"""The time limit, where a run in the panel reached it."""
if (rows["status"] == "timeout").any():
ax.axhline(TIMEOUT, color=INK2, lw=0.8, ls=(0, (1, 2)), zorder=1)
ax.annotate(f"time limit, {human(TIMEOUT)}", (0.02, TIMEOUT), xycoords=("axes fraction", "data"),
xytext=(0, 3), textcoords="offset points", color=INK2, fontsize=7.5)
def ram_line(ax, gb, vertical=False):
"""The memory of the machine the runs were on."""
if gb != gb:
return
if vertical:
ax.axvline(gb, color=INK2, lw=1, ls=(0, (4, 2)), zorder=1)
ax.annotate(f"RAM of this machine\n{gb:.1f} GB", (gb, 1.0), xycoords=("data", "axes fraction"),
xytext=(-4, -2), textcoords="offset points", ha="right", va="top",
color=INK2, fontsize=7.5)
else:
ax.axhline(gb, color=INK2, lw=1, ls=(0, (4, 2)), zorder=1)
ax.annotate(f"RAM of this machine, {gb:.1f} GB", (0.02, gb), xycoords=("axes fraction", "data"),
xytext=(0, 3), textcoords="offset points", color=INK2, fontsize=7.5)
ax.update_datalim([(min(CONTIGS.values()), gb * 1.6)])
COMMANDS = {
("create", "cload pairs"): "cooler cload pairs",
("create", "create pairs"): "metacooler create pairs",
("convert", "create cool"): "metacooler create cool",
("convert", "flatten"): "metacooler flatten",
("query", "contig", "cooler"): "cooler dump -r (a contig)",
("query", "contig", "metacooler"): "metacooler dump -r (a contig)",
("query", "genome", "cooler"): "cooler dump --join | awk (a genome)",
("query", "genome", "metacooler"): "metacooler dump -r (a genome)",
("query", "genome pair", "cooler"): "cooler dump --join | awk (a genome pair)",
("query", "genome pair", "metacooler"): "metacooler dump -r -r2 (a genome pair)",
("aggregate", "genome network", "cooler"): "cooler dump --join | awk (genome network)",
("aggregate", "genome network", "metacooler"): "metacooler network --level genome",
("aggregate", "genome-level .cool"): "metacooler aggregate --level genome",
}
def command(section, tool, task):
"""What a user types, in a few words: the name a sentence gives a command."""
return COMMANDS.get((section, task, tool)) or COMMANDS.get((section, task)) or f"{tool} {task}"
TOOLS_LEGEND = [Line2D([], [], color=HUE[t], marker="o", ms=5, label=t) for t in ("cooler", "metacooler")]Settings
%%bash
source bench.sh
echo "datasets: $DATASETS · bin size: $BINSIZE bp · zoom levels: $ZOOMS"
echo "time limit: $TIMEOUT s per run"
if [[ -n ${BENCH_REPS:-} ]]; then
echo "runs per command: $BENCH_REPS"
else
echo "runs per command: $MIN_REPS to $MAX_REPS (no more than $MIN_REPS once they add up to $BUDGET s;"
echo " one, for a command whose run takes over $LONG s)"
fi
metacooler --version
cooler --version
# Every timing is against one cooler: refuse to time another.
installed=$(python -c 'import cooler; print(cooler.__version__)')
if [[ $installed != "$COOLER_VERSION" ]]; then
echo "cooler $installed is installed; these benchmarks are against cooler $COOLER_VERSION." >&2
echo "Run uv sync (uv.lock pins it), or bump COOLER_VERSION in bench.sh with the lock." >&2
exit 1
fi
[[ -n $SKIP_COOLER_CREATE ]] && echo "cooler cload pairs not run on: $SKIP_COOLER_CREATE"datasets: S M L XL · bin size: 1000 bp · zoom levels: 1000,2000,5000,10000
time limit: 1800 s per run
runs per command: 3 to 5 (no more than 3 once they add up to 120 s;
one, for a command whose run takes over 600 s)
metacooler, version 0.2.0
cooler, version 0.12.1
cooler cload pairs not run on: XL
Datasets
All four datasets are simulated. No real reads, assembly or binning enters them, so they have no accessions. Each is drawn by simulate.py from a preset and a fixed seed, so every render times the same data; the workflow publishes them as a release (benchmark-data-v<generator version>) and fetches them from there.
- Genomes. MAGs (log-normal, 2.5 Mb median) and small bins standing for plasmids, phages and other mobile elements (20 kb median) are binned. Low-abundance genomes that no binner would recover make up the rest of the assembly, and their contigs end up in
__unbinned__. - Contigs. Each genome is cut around its own median contig length, so one MAG comes out in a dozen long contigs and the next in hundreds of short ones. Unbinned contigs are short (~0.9 kb). Nothing is under 500 bp. The assembly lists contigs in random order, as an assembler does, so a genome’s contigs are scattered through any genome-agnostic
.cool. - Contacts. A pair falls in a genome in proportion to abundance × length; 96 % stay there, at a P(s) ~ 1/s separation (60 %) or uniformly along a circular chromosome (40 %); 4 % join two genomes. Which pairs are intra-contig, inter-contig or inter-genome then follows from the contig lengths, as in a real library.
The presets are sized after real metagenomic Hi-C maps — M about a mock community’s, L about a human gut sample’s, in genomes, contigs, bins and contacts — but no data of those maps is used. What the simulation decides matters most for the genome queries (§5): cooler scans every pixel of the map, metacooler reads those of one genome, so their ratio follows the share of the map’s pixels the queried genome holds. That share is set here by the model’s community — how many genomes, how abundant — not by a sample; the query table gives it for each dataset.
%%bash
source bench.sh
for ds in $DATASETS; do # fetched, made earlier, or made now
python simulate.py --preset "$ds" --outdir "$DATA/$ds" --keep | sed "s|$WORK/||"
done
cd "$WORK" && du -sh data/*S: data/S is up to date (generator v1)
M: data/M is up to date (generator v1)
L: data/L is up to date (generator v1)
XL: data/XL is up to date (generator v1)
249M data/L
77M data/M
7.6M data/S
936M data/XL
Code
SUMMARY = {ds: json.loads((DATA / ds / "summary.json").read_text()) for ds in DATASETS}
CONTIGS = {ds: s["contigs"] for ds, s in SUMMARY.items()}
BINSIZE = int(os.environ.get("BENCH_BINSIZE", 1000))
SIZES = {
ds: pd.read_csv(DATA / ds / "contigs.sizes", sep="\t", header=None, names=["contig", "length"])
for ds in DATASETS
}
rows = []
for ds, s in SUMMARY.items():
rows.append({
"dataset": ds,
"source": f"simulated (simulate.py v{s['generator_version']}, seed {s['seed']})",
"genomes (MAGs + small bins)": f"{s['genomes']:,} ({s['mags']:,} + {s['mge_bins']:,})",
"contigs": f"{s['contigs']:,}",
"binned contigs": f"{s['binned_contigs'] / s['contigs']:.0%}",
"assembly (Mb)": f"{s['assembly_bp'] / 1e6:,.0f}",
"contig N50 (kb)": f"{s['contig_n50'] / 1e3:.1f}",
"median contig (bp)": f"{s['contig_median']:,}",
"contigs / MAG (median)": f"{s['contigs_per_mag_median']:,.0f}",
"MAG size (median, Mb)": f"{s['mag_bp_median'] / 1e6:.2f}",
"MAG contig N50 (kb)": f"{s['mag_contig_n50_median'] / 1e3:.1f}",
"read pairs (M)": f"{s['pairs'] / 1e6:,.0f}",
"intra-contig": f"{s['pairs_intra_contig']:.1%}",
"inter-contig, intra-genome": f"{s['pairs_inter_contig_intra_genome']:.1%}",
"inter-genome": f"{s['pairs_inter_genome']:.1%}",
"unbinned, inter-contig": f"{s['pairs_unbinned']:.1%}",
f"bins at {BINSIZE:,} bp": f"{int((-(-SIZES[ds]['length'] // BINSIZE)).sum()):,}",
"pairs.gz (MB)": f"{s['files_bytes']['contacts.pairs.gz'] / 1e6:,.0f}",
"pairs md5 (uncompressed)": s["pairs_md5_uncompressed"][:12],
})
DATASET_TABLE = pd.DataFrame(rows)
DATASET_TABLE.to_csv(RESULTS / "datasets.tsv", sep="\t", index=False)
show(DATASET_TABLE.set_index("dataset").T.reset_index().rename(columns={"index": ""}))| S | M | L | XL | |
|---|---|---|---|---|
| source | simulated (simulate.py v1, seed 1) | simulated (simulate.py v1, seed 2) | simulated (simulate.py v1, seed 3) | simulated (simulate.py v1, seed 4) |
| genomes (MAGs + small bins) | 10 (10 + 0) | 100 (50 + 50) | 2,000 (200 + 1,800) | 20,000 (1,000 + 19,000) |
| contigs | 1,000 | 20,000 | 200,000 | 2,000,000 |
| binned contigs | 48% | 49% | 41% | 24% |
| assembly (Mb) | 27 | 157 | 764 | 5,119 |
| contig N50 (kb) | 80.1 | 27.6 | 11.6 | 6.5 |
| median contig (bp) | 2,491 | 1,755 | 1,242 | 1,032 |
| contigs / MAG (median) | 50 | 122 | 209 | 198 |
| MAG size (median, Mb) | 2.80 | 2.72 | 2.56 | 2.46 |
| MAG contig N50 (kb) | 88.0 | 27.6 | 19.1 | 19.7 |
| read pairs (M) | 1 | 10 | 30 | 100 |
| intra-contig | 35.4% | 28.2% | 24.7% | 27.0% |
| inter-contig, intra-genome | 62.0% | 66.9% | 69.1% | 63.8% |
| inter-genome | 2.5% | 3.7% | 3.7% | 3.5% |
| unbinned, inter-contig | 0.1% | 1.1% | 2.5% | 5.7% |
| bins at 1,000 bp | 27,498 | 166,079 | 852,561 | 5,978,049 |
| pairs.gz (MB) | 8 | 80 | 250 | 874 |
| pairs md5 (uncompressed) | 2264f7277b25 | 1cd62a937421 | a0fe807ef85c | 59e7cba48488 |
Code
fig, (left, right) = plt.subplots(1, 2, figsize=(9, 3.2), layout="constrained")
for ds in DATASETS:
sizes = SIZES[ds]
binning = pd.read_csv(DATA / ds / "contigs.tsv", sep="\t", usecols=[0, 1])
per_genome = binning.loc[binning["genome_name"] != "__unbinned__", "genome_name"].value_counts()
edges = np.geomspace(500, sizes["length"].max() + 1, 30)
share, _ = np.histogram(sizes["length"], edges)
left.plot(np.sqrt(edges[1:] * edges[:-1]), share / share.sum(), color=SIZE[ds], label=ds)
n = np.sort(per_genome.to_numpy())
right.step(n, 1 - np.arange(len(n)) / len(n), where="post", color=SIZE[ds], label=ds)
for ax, xlabel, ylabel in ((left, "contig length (bp)", "share of contigs"),
(right, "contigs per binned genome", "share of genomes with at least as many")):
ax.set(xscale="log", xlabel=xlabel, ylabel=ylabel)
ax.yaxis.set_major_formatter(FuncFormatter(lambda v, _: f"{v:.0%}"))
left.legend(title="dataset")
save(fig, "datasets")1 · Creating a map from pairs
cooler cload pairs and metacooler create pairs, on the same pairs, contig lengths and bin size; metacooler also reads the binning and genome tables and builds /genomes, its index, the QC block and the per-MAG P(s) table — its defaults, all on.
On a metagenome the difference is the contig count, not the pixels. cooler.create types /bins/chrom as an HDF5 enumeration with one member per contig. Building it is quadratic, because each insertion rescans the members already there, and past a few thousand contigs the type outgrows HDF5’s object header anyway and is dropped for plain integers. cooler cload pairs sorts externally: it writes the bin table once per chunk of pairs and once per merge, so it pays that cost several times per file. metacooler writes the integers directly.
cooler cload pairs is not run on XL: there it has never finished within the 30-min time limit (every earlier render stopped it at the limit), so a run would only spend another half hour to say so again. BENCH_SKIP_COOLER_CREATE= (empty) runs it on every dataset.
%%bash
source bench.sh
for ds in $DATASETS; do
d=$DATA/$ds
warm "$d/contacts.pairs.gz"
if [[ " $SKIP_COOLER_CREATE " == *" $ds "* ]]; then
skip create "$ds" cooler "cload pairs" \
"it has never finished on $ds within the time limit (BENCH_SKIP_COOLER_CREATE)"
else
bench create "$ds" cooler "cload pairs" "$MAPS/$ds.cool" -- \
cooler cload pairs -c1 2 -p1 3 -c2 4 -p2 5 \
"$d/contigs.sizes:$BINSIZE" "$d/contacts.pairs.gz" "$MAPS/$ds.cool"
fi
bench create "$ds" metacooler "create pairs" "$MAPS/$ds.metacool" -- \
metacooler create pairs --pairs "$d/contacts.pairs.gz" --bins "$BINSIZE" \
--contigs "$d/contigs.tsv" --genomes "$d/genomes.tsv" -o "$MAPS/$ds.metacool"
donecreate/S/cooler/cload pairs run 1 ok 1.82 s 431 MB
create/S/cooler/cload pairs run 2 ok 1.84 s 430 MB
create/S/cooler/cload pairs run 3 ok 1.86 s 430 MB
create/S/cooler/cload pairs run 4 ok 1.84 s 430 MB
create/S/cooler/cload pairs run 5 ok 1.83 s 430 MB
create/S/metacooler/create pairs run 1 ok 1.33 s 398 MB
create/S/metacooler/create pairs run 2 ok 1.18 s 395 MB
create/S/metacooler/create pairs run 3 ok 1.17 s 396 MB
create/S/metacooler/create pairs run 4 ok 1.17 s 396 MB
create/S/metacooler/create pairs run 5 ok 1.16 s 395 MB
create/M/cooler/cload pairs run 1 ok 17.90 s 2611 MB
create/M/cooler/cload pairs run 2 ok 17.44 s 2614 MB
create/M/cooler/cload pairs run 3 ok 17.28 s 2615 MB
create/M/cooler/cload pairs run 4 ok 17.50 s 2615 MB
create/M/cooler/cload pairs run 5 ok 17.53 s 2615 MB
create/M/metacooler/create pairs run 1 ok 5.91 s 1476 MB
create/M/metacooler/create pairs run 2 ok 5.98 s 1476 MB
create/M/metacooler/create pairs run 3 ok 5.96 s 1476 MB
create/M/metacooler/create pairs run 4 ok 5.98 s 1460 MB
create/M/metacooler/create pairs run 5 ok 6.06 s 1476 MB
create/L/cooler/cload pairs run 1 ok 410.33 s 4201 MB
create/L/cooler/cload pairs run 2 ok 401.60 s 4203 MB
create/L/cooler/cload pairs run 3 ok 394.31 s 4203 MB
create/L/metacooler/create pairs run 1 ok 19.36 s 2370 MB
create/L/metacooler/create pairs run 2 ok 19.41 s 2365 MB
create/L/metacooler/create pairs run 3 ok 19.60 s 2437 MB
create/L/metacooler/create pairs run 4 ok 20.24 s 2304 MB
create/L/metacooler/create pairs run 5 ok 20.06 s 2379 MB
create/XL/cooler/cload pairs skipped: it has never finished on XL within the time limit (BENCH_SKIP_COOLER_CREATE)
create/XL/metacooler/create pairs run 1 ok 120.30 s 5832 MB
create/XL/metacooler/create pairs run 2 ok 106.40 s 6165 MB
create/XL/metacooler/create pairs run 3 ok 106.09 s 6285 MB
Code
show(wide("create"))| tool | task | S | M | L | XL |
|---|---|---|---|---|---|
| cooler | cload pairs | 1.84 ± 0.01 (n=5) | 17.5 ± 0.2 (n=5) | 402 ± 8 (n=3) | not run |
| metacooler | create pairs | 1.20 ± 0.07 (n=5) | 5.98 ± 0.05 (n=5) | 19.7 ± 0.4 (n=5) | 111 ± 8 (n=3) |
Code
show(wide("create", "peak_mb", digits=0))| tool | task | S | M | L | XL |
|---|---|---|---|---|---|
| cooler | cload pairs | 430 ± 0 (n=5) | 2,614 ± 2 (n=5) | 4,202 ± 1 (n=3) | not run |
| metacooler | create pairs | 396 ± 1 (n=5) | 1,473 ± 7 (n=5) | 2,371 ± 47 (n=5) | 6,094 ± 235 (n=3) |
Code
t = timings()
rows = runs(t, section="create")
fig, (left, right) = plt.subplots(1, 2, figsize=(9, 3.6), layout="constrained")
scaling(left, rows)
seconds_axis(left)
limit_line(left, rows)
scaling(right, rows, "peak_gb")
log_y(right, "peak memory (GB)")
ram_line(right, ram(t))
left.legend(handles=TOOLS_LEGEND, loc="lower right")
save(fig, "create")The two tools must hold the same contacts: same number of pixels, same total, read from each file’s own attributes.
Code
MADE_BY = {"cooler": ("cload pairs", "cool"), "metacooler": ("create pairs", "metacool")}
same = []
for ds in DATASETS:
row, held = {"dataset": ds}, {}
for tool, (task, ext) in MADE_BY.items():
if finished("create", ds, tool, task): # else the map below is not this tool's
with h5py.File(MAPS / f"{ds}.{ext}", "r") as f:
held[tool] = int(f.attrs["nnz"]), int(f.attrs["sum"])
row[f"{tool} nnz"], row[f"{tool} sum"] = held[tool]
missing = [tool for tool in MADE_BY if tool not in held]
row["identical"] = (
f"{' and '.join(missing)} did not finish or was not run" if missing
else "yes" if held["cooler"] == held["metacooler"] else "NO"
)
same.append(row)
same = pd.DataFrame(same)
assert not (same["identical"] == "NO").any(), "cooler and metacooler disagree on the contacts"
show(same, **{c: "{:,.0f}" for c in same.columns if c.endswith(("nnz", "sum"))})| dataset | cooler nnz | cooler sum | metacooler nnz | metacooler sum | identical |
|---|---|---|---|---|---|
| S | 811,390 | 1,000,000 | 811,390 | 1,000,000 | yes |
| M | 7,684,333 | 10,000,000 | 7,684,333 | 10,000,000 | yes |
| L | 22,335,454 | 30,000,000 | 22,335,454 | 30,000,000 | yes |
| XL | 73,140,434 | 100,000,000 | cooler did not finish or was not run |
Where cooler did not finish or was not run, the rest of the page still needs the genome-agnostic .cool. It is built here, untimed, by metacooler in the assembly’s own contig order (--contig-order fasta, which reproduces cooler cload pairs pixel for pixel), then flattened to a plain cooler.
%%bash
source bench.sh
for ds in $DATASETS; do
[[ -s $MAPS/$ds.cool ]] && continue
d=$DATA/$ds
echo "$ds: no $ds.cool from cooler; building it with metacooler"
metacooler create pairs -q --contig-order fasta --no-qc --no-ps --pairs "$d/contacts.pairs.gz" \
--bins "$BINSIZE" --contigs "$d/contigs.tsv" -o "$SCRATCH/$ds.fasta.metacool" &&
metacooler flatten -q "$SCRATCH/$ds.fasta.metacool" -o "$MAPS/$ds.cool"
rm -f "$SCRATCH/$ds.fasta.metacool" "$SCRATCH/$ds.fasta.ps.tsv"
done
cd "$MAPS" && du -h -- *.cool *.metacoolXL: no XL.cool from cooler; building it with metacooler
49M L.cool
14M M.cool
1.4M S.cool
196M XL.cool
38M L.metacool
12M M.metacool
1.4M S.metacool
154M XL.metacool
2 · Size on disk
What it takes to keep a metagenome’s map with its genomes, on disk, four ways:
.metacool: one file, every contig and contact, with the binning, the genome and contig tables, their index and the QC block (metacooler create pairs, above)..cool+ binning tables: the genome-agnostic.cool(cooler cload pairs) and, beside it, the tables that say which contig is in which genome and what each genome is — heresimulate.py’scontigs.tsv(contig, genome, GC, coverage) andgenomes.tsv(completeness, contamination, quality, GC, abundance), the kind of tables a binner such as MetaTOR writes. The.coolalone holds the same contacts, without knowing a genome from another.- One
.coolper MAG: each MAG’s contigs and the contacts within it, one file per MAG — the MAGs only, the way they are usually handed on. They hold less than the others: no unbinned contig, no small bin, no contact between two genomes. The table below says how much less. Written untimed bymetacooler extract --split, through cooler’s own writer, so they are the files cooler itself writes for those contigs and contacts.
The input pairs (contacts.pairs.gz) are given for scale.
%%bash
source bench.sh
# simulate.py names the MAGs MAG_*: the filter picks them out of /genomes.
for ds in $DATASETS; do
out=$MAPS/$ds.mags
[[ -e $out/.complete && $out/.complete -nt $MAPS/$ds.metacool ]] && continue
rm -rf "$out" && mkdir -p "$out"
metacooler extract -q "$MAPS/$ds.metacool" --filter 'genomes.name =~ ^MAG_' --split -o "$out/$ds" &&
touch "$out/.complete"
done
cd "$MAPS" && du -sh -- *.mags40M L.mags
13M M.mags
1.7M S.mags
138M XL.mags
Code
def held_by(path):
"""Contigs, pixels and contacts a cool or metacool holds, from its own attributes."""
with h5py.File(path, "r") as f:
return len(f["chroms/name"]), int(f.attrs["nnz"]), int(f.attrs["sum"])
STORAGE, HELD, MAG_SHARE = [], [], {}
for ds in DATASETS:
d = DATA / ds
tables = (d / "contigs.tsv").stat().st_size + (d / "genomes.tsv").stat().st_size
cool = (MAPS / f"{ds}.cool").stat().st_size
mags = sorted((MAPS / f"{ds}.mags").glob("*.cool"))
for option, files, size in (
(".metacool", 1, (MAPS / f"{ds}.metacool").stat().st_size),
(".cool + binning tables", 3, cool + tables),
(".cool alone (no genomes)", 1, cool),
("one .cool per MAG", len(mags), sum(p.stat().st_size for p in mags)),
("contacts.pairs.gz (input)", 1, (d / "contacts.pairs.gz").stat().st_size),
):
STORAGE.append({"dataset": ds, "contigs": CONTIGS[ds], "option": option, "files": files, "mb": size / 1e6})
contigs, nnz, total = held_by(MAPS / f"{ds}.metacool")
in_mags = np.array([held_by(p) for p in mags]).reshape(-1, 3).sum(axis=0)
MAG_SHARE[ds] = {"contigs": in_mags[0] / contigs, "pixels": in_mags[1] / nnz, "contacts": in_mags[2] / total}
HELD.append({
"dataset": ds, "MAG files": f"{len(mags):,}",
"contigs": f"{in_mags[0]:,} of {contigs:,} ({in_mags[0] / contigs:.1%})",
"pixels": f"{in_mags[1]:,} of {nnz:,} ({in_mags[1] / nnz:.1%})",
"contacts": f"{in_mags[2]:,} of {total:,} ({in_mags[2] / total:.1%})",
})
STORAGE = pd.DataFrame(STORAGE)
STORAGE.to_csv(RESULTS / "storage.tsv", sep="\t", index=False)
OPTIONS = list(dict.fromkeys(STORAGE["option"]))
table = STORAGE.assign(
cell=[f"{mb:,.1f}" + (f" ({files:,} files)" if files > 1 else "") for mb, files in zip(STORAGE["mb"], STORAGE["files"])]
).pivot(index="option", columns="dataset", values="cell").reindex(index=OPTIONS, columns=DATASETS)
show(table.rename_axis("MB on disk").reset_index()).cool is cooler’s own where cooler finished, and the untimed one built above where it did not.
| MB on disk | S | M | L | XL |
|---|---|---|---|---|
| .metacool | 1.4 | 12.0 | 38.9 | 161.3 |
| .cool + binning tables | 1.5 (3 files) | 15.2 (3 files) | 57.5 (3 files) | 277.1 (3 files) |
| .cool alone (no genomes) | 1.5 | 14.5 | 50.7 | 204.7 |
| one .cool per MAG | 1.7 (10 files) | 13.4 (50 files) | 41.0 (200 files) | 142.5 (1,000 files) |
| contacts.pairs.gz (input) | 7.9 | 79.7 | 250.0 | 874.1 |
Code
show(pd.DataFrame(HELD)).cool files hold, out of the whole map (the .metacool and the .cool hold all of it).
| dataset | MAG files | contigs | pixels | contacts |
|---|---|---|---|---|
| S | 10 | 483 of 1,000 (48.3%) | 785,061 of 811,390 (96.8%) | 973,506 of 1,000,000 (97.4%) |
| M | 50 | 9,380 of 20,000 (46.9%) | 7,202,910 of 7,684,333 (93.7%) | 9,432,108 of 10,000,000 (94.3%) |
| L | 200 | 66,285 of 200,000 (33.1%) | 19,658,405 of 22,335,454 (88.0%) | 24,726,493 of 30,000,000 (82.4%) |
| XL | 1,000 | 315,307 of 2,000,000 (15.8%) | 56,600,973 of 73,140,434 (77.4%) | 68,033,411 of 100,000,000 (68.0%) |
3 · Converting between the formats
No cooler counterpart: adding the genome layer to an existing .cool (create cool), and dropping it again (flatten).
%%bash
source bench.sh
for ds in $DATASETS; do
d=$DATA/$ds
bench convert "$ds" metacooler "create cool" "$SCRATCH/$ds.from-cool.metacool" -- \
metacooler create cool --cool "$MAPS/$ds.cool" --contigs "$d/contigs.tsv" \
--genomes "$d/genomes.tsv" -o "$SCRATCH/$ds.from-cool.metacool"
bench convert "$ds" metacooler flatten "$SCRATCH/$ds.flat.cool" -- \
metacooler flatten "$MAPS/$ds.metacool" -o "$SCRATCH/$ds.flat.cool"
doneconvert/S/metacooler/create cool run 1 ok 1.00 s 218 MB
convert/S/metacooler/create cool run 2 ok 0.99 s 218 MB
convert/S/metacooler/create cool run 3 ok 0.98 s 218 MB
convert/S/metacooler/create cool run 4 ok 0.99 s 218 MB
convert/S/metacooler/create cool run 5 ok 0.98 s 217 MB
convert/S/metacooler/flatten run 1 ok 0.85 s 217 MB
convert/S/metacooler/flatten run 2 ok 0.85 s 217 MB
convert/S/metacooler/flatten run 3 ok 0.85 s 217 MB
convert/S/metacooler/flatten run 4 ok 0.84 s 217 MB
convert/S/metacooler/flatten run 5 ok 0.84 s 217 MB
convert/M/metacooler/create cool run 1 ok 3.81 s 695 MB
convert/M/metacooler/create cool run 2 ok 3.78 s 697 MB
convert/M/metacooler/create cool run 3 ok 3.80 s 696 MB
convert/M/metacooler/create cool run 4 ok 3.77 s 696 MB
convert/M/metacooler/create cool run 5 ok 3.78 s 697 MB
convert/M/metacooler/flatten run 1 ok 3.04 s 645 MB
convert/M/metacooler/flatten run 2 ok 3.04 s 645 MB
convert/M/metacooler/flatten run 3 ok 3.03 s 645 MB
convert/M/metacooler/flatten run 4 ok 3.01 s 644 MB
convert/M/metacooler/flatten run 5 ok 3.04 s 643 MB
convert/L/metacooler/create cool run 1 ok 12.48 s 1810 MB
convert/L/metacooler/create cool run 2 ok 12.45 s 1810 MB
convert/L/metacooler/create cool run 3 ok 12.73 s 1808 MB
convert/L/metacooler/create cool run 4 ok 12.64 s 1809 MB
convert/L/metacooler/create cool run 5 ok 12.74 s 1811 MB
convert/L/metacooler/flatten run 1 ok 9.07 s 1604 MB
convert/L/metacooler/flatten run 2 ok 9.24 s 1605 MB
convert/L/metacooler/flatten run 3 ok 9.13 s 1605 MB
convert/L/metacooler/flatten run 4 ok 9.07 s 1605 MB
convert/L/metacooler/flatten run 5 ok 9.09 s 1604 MB
convert/XL/metacooler/create cool run 1 ok 72.17 s 4469 MB
convert/XL/metacooler/create cool run 2 ok 72.39 s 4464 MB
convert/XL/metacooler/create cool run 3 ok 73.68 s 4494 MB
convert/XL/metacooler/flatten run 1 ok 42.85 s 4391 MB
convert/XL/metacooler/flatten run 2 ok 43.04 s 4390 MB
convert/XL/metacooler/flatten run 3 ok 43.73 s 4391 MB
Code
show(wide("convert"))| tool | task | S | M | L | XL |
|---|---|---|---|---|---|
| metacooler | create cool | 0.988 ± 0.008 (n=5) | 3.79 ± 0.02 (n=5) | 12.6 ± 0.1 (n=5) | 72.7 ± 0.8 (n=3) |
| metacooler | flatten | 0.846 ± 0.005 (n=5) | 3.03 ± 0.01 (n=5) | 9.12 ± 0.07 (n=5) | 43.2 ± 0.5 (n=3) |
4 · Multi-resolution maps
cooler zoomify and metacooler zoomify, to the same resolutions. Each coarser level is another cooler.create, so cooler pays the per-contig cost once per level. Where cooler could not build the base map, its pyramid is not attempted.
%%bash
source bench.sh
for ds in $DATASETS; do
bench zoomify "$ds" metacooler zoomify "$SCRATCH/$ds.mmetacool" -- \
metacooler zoomify "$MAPS/$ds.metacool" -r "$ZOOMS" -o "$SCRATCH/$ds.mmetacool"
if unfinished create "$ds" cooler "cload pairs"; then
skip zoomify "$ds" cooler zoomify "no base map from cooler cload pairs on $ds"
continue
fi
bench zoomify "$ds" cooler zoomify "$SCRATCH/$ds.mcool" -- \
cooler zoomify -r "$ZOOMS" -o "$SCRATCH/$ds.mcool" "$MAPS/$ds.cool"
donezoomify/S/metacooler/zoomify run 1 ok 1.46 s 245 MB
zoomify/S/metacooler/zoomify run 2 ok 1.43 s 245 MB
zoomify/S/metacooler/zoomify run 3 ok 1.44 s 245 MB
zoomify/S/metacooler/zoomify run 4 ok 1.46 s 245 MB
zoomify/S/metacooler/zoomify run 5 ok 1.45 s 245 MB
zoomify/S/cooler/zoomify run 1 ok 3.85 s 306 MB
zoomify/S/cooler/zoomify run 2 ok 3.89 s 306 MB
zoomify/S/cooler/zoomify run 3 ok 3.92 s 306 MB
zoomify/S/cooler/zoomify run 4 ok 3.80 s 306 MB
zoomify/S/cooler/zoomify run 5 ok 3.78 s 306 MB
zoomify/M/metacooler/zoomify run 1 ok 6.32 s 986 MB
zoomify/M/metacooler/zoomify run 2 ok 6.31 s 986 MB
zoomify/M/metacooler/zoomify run 3 ok 6.29 s 986 MB
zoomify/M/metacooler/zoomify run 4 ok 6.28 s 986 MB
zoomify/M/metacooler/zoomify run 5 ok 6.30 s 986 MB
zoomify/M/cooler/zoomify run 1 ok 61.71 s 1382 MB
zoomify/M/cooler/zoomify run 2 ok 60.29 s 1382 MB
zoomify/M/cooler/zoomify run 3 ok 61.07 s 1377 MB
zoomify/L/metacooler/zoomify run 1 ok 21.33 s 2567 MB
zoomify/L/metacooler/zoomify run 2 ok 21.21 s 2568 MB
zoomify/L/metacooler/zoomify run 3 ok 21.32 s 2567 MB
zoomify/L/metacooler/zoomify run 4 ok 21.03 s 2567 MB
zoomify/L/metacooler/zoomify run 5 ok 20.90 s 2567 MB
zoomify/L/cooler/zoomify run 1 ok 916.69 s 2574 MB
zoomify/XL/metacooler/zoomify run 1 ok 105.29 s 6731 MB
zoomify/XL/metacooler/zoomify run 2 ok 104.90 s 6733 MB
zoomify/XL/metacooler/zoomify run 3 ok 104.22 s 6715 MB
zoomify/XL/cooler/zoomify skipped: no base map from cooler cload pairs on XL
Code
show(wide("zoomify"))| tool | task | S | M | L | XL |
|---|---|---|---|---|---|
| metacooler | zoomify | 1.45 ± 0.01 (n=5) | 6.30 ± 0.02 (n=5) | 21.2 ± 0.2 (n=5) | 105 ± 1 (n=3) |
| cooler | zoomify | 3.85 ± 0.06 (n=5) | 61.0 ± 0.7 (n=3) | 917 (n=1) | not run |
5 · Querying a contig, a genome, a genome pair
One contig, one MAG and one pair of MAGs per dataset, picked by targets.py with a fixed seed (the pair in proportion to abundance × length, as the library joins genomes, so that it shares contacts). cooler serves a contig through its own index (dump -r). It has no notion of a genome. So a MAG’s pixels are the joined pixels whose two contigs the binning table puts in that MAG, in one pass of awk over the whole map. metacooler resolves the genome by name (dump -r/-r2).
%%bash
source bench.sh
in_genome='NR == FNR { if ($2 == g) keep[$1]; next } ($1 in keep) && ($4 in keep)'
in_pair='NR == FNR { if ($2 == a) A[$1]; if ($2 == b) B[$1]; next }'
in_pair+=' (($1 in A) && ($4 in B)) || (($1 in B) && ($4 in A))'
for ds in $DATASETS; do
d=$DATA/$ds m=$MAPS/$ds q=$QUERIES/$ds
read -r contig mag a b <<<"$(python targets.py --contigs "$d/contigs.tsv" \
--sizes "$d/contigs.sizes" --genomes "$d/genomes.tsv")"
echo "$ds: contig $contig · genome $mag · genome pair $a × $b"
bench query "$ds" cooler contig "$q.cooler.contig.tsv" -- \
cooler dump -t pixels -r "$contig" -o "$q.cooler.contig.tsv" "$m.cool"
bench query "$ds" metacooler contig "$q.metacooler.contig.tsv" -- \
metacooler dump "$m.metacool" -t pixels -r "$contig" -o "$q.metacooler.contig.tsv"
bench query "$ds" cooler genome "$q.cooler.genome.tsv" -- bash -o pipefail -c \
"$STAGE cooler dump -t pixels --join $m.cool | $STAGE awk -F'\t' -v g=$mag '$in_genome' $d/contigs.tsv - > $q.cooler.genome.tsv"
bench query "$ds" metacooler genome "$q.metacooler.genome.tsv" -- \
metacooler dump "$m.metacool" -t pixels -r "$mag" -o "$q.metacooler.genome.tsv"
bench query "$ds" cooler "genome pair" "$q.cooler.pair.tsv" -- bash -o pipefail -c \
"$STAGE cooler dump -t pixels --join $m.cool | $STAGE awk -F'\t' -v a=$a -v b=$b '$in_pair' $d/contigs.tsv - > $q.cooler.pair.tsv"
bench query "$ds" metacooler "genome pair" "$q.metacooler.pair.tsv" -- \
metacooler dump "$m.metacool" -t pixels -r "$a" -r2 "$b" -o "$q.metacooler.pair.tsv"
doneS: contig k141_648 · genome MAG_09 · genome pair MAG_05 × MAG_01
query/S/cooler/contig run 1 ok 0.52 s 150 MB
query/S/cooler/contig run 2 ok 0.51 s 150 MB
query/S/cooler/contig run 3 ok 0.52 s 150 MB
query/S/cooler/contig run 4 ok 0.51 s 150 MB
query/S/cooler/contig run 5 ok 0.50 s 151 MB
query/S/metacooler/contig run 1 ok 0.55 s 151 MB
query/S/metacooler/contig run 2 ok 0.53 s 151 MB
query/S/metacooler/contig run 3 ok 0.54 s 151 MB
query/S/metacooler/contig run 4 ok 0.53 s 151 MB
query/S/metacooler/contig run 5 ok 0.53 s 151 MB
query/S/cooler/genome run 1 ok 1.87 s 225 MB
query/S/cooler/genome run 2 ok 1.87 s 225 MB
query/S/cooler/genome run 3 ok 1.87 s 225 MB
query/S/cooler/genome run 4 ok 1.87 s 225 MB
query/S/cooler/genome run 5 ok 1.86 s 224 MB
query/S/metacooler/genome run 1 ok 0.58 s 157 MB
query/S/metacooler/genome run 2 ok 0.59 s 158 MB
query/S/metacooler/genome run 3 ok 0.58 s 157 MB
query/S/metacooler/genome run 4 ok 0.58 s 157 MB
query/S/metacooler/genome run 5 ok 0.58 s 157 MB
query/S/cooler/genome pair run 1 ok 1.95 s 225 MB
query/S/cooler/genome pair run 2 ok 1.95 s 225 MB
query/S/cooler/genome pair run 3 ok 1.95 s 225 MB
query/S/cooler/genome pair run 4 ok 1.94 s 225 MB
query/S/cooler/genome pair run 5 ok 1.95 s 224 MB
query/S/metacooler/genome pair run 1 ok 0.57 s 166 MB
query/S/metacooler/genome pair run 2 ok 0.56 s 166 MB
query/S/metacooler/genome pair run 3 ok 0.56 s 166 MB
query/S/metacooler/genome pair run 4 ok 0.56 s 166 MB
query/S/metacooler/genome pair run 5 ok 0.59 s 166 MB
M: contig k141_13779 · genome MAG_43 · genome pair MAG_14 × MAG_01
query/M/cooler/contig run 1 ok 0.53 s 162 MB
query/M/cooler/contig run 2 ok 0.54 s 162 MB
query/M/cooler/contig run 3 ok 0.52 s 162 MB
query/M/cooler/contig run 4 ok 0.53 s 162 MB
query/M/cooler/contig run 5 ok 0.53 s 162 MB
query/M/metacooler/contig run 1 ok 0.56 s 158 MB
query/M/metacooler/contig run 2 ok 0.54 s 158 MB
query/M/metacooler/contig run 3 ok 0.55 s 158 MB
query/M/metacooler/contig run 4 ok 0.54 s 158 MB
query/M/metacooler/contig run 5 ok 0.54 s 158 MB
query/M/cooler/genome run 1 ok 14.12 s 295 MB
query/M/cooler/genome run 2 ok 14.21 s 289 MB
query/M/cooler/genome run 3 ok 14.19 s 308 MB
query/M/cooler/genome run 4 ok 14.35 s 300 MB
query/M/cooler/genome run 5 ok 14.25 s 297 MB
query/M/metacooler/genome run 1 ok 0.57 s 160 MB
query/M/metacooler/genome run 2 ok 0.57 s 160 MB
query/M/metacooler/genome run 3 ok 0.59 s 160 MB
query/M/metacooler/genome run 4 ok 0.57 s 160 MB
query/M/metacooler/genome run 5 ok 0.58 s 160 MB
query/M/cooler/genome pair run 1 ok 14.92 s 297 MB
query/M/cooler/genome pair run 2 ok 14.90 s 289 MB
query/M/cooler/genome pair run 3 ok 14.80 s 295 MB
query/M/cooler/genome pair run 4 ok 14.99 s 297 MB
query/M/cooler/genome pair run 5 ok 14.65 s 287 MB
query/M/metacooler/genome pair run 1 ok 0.60 s 176 MB
query/M/metacooler/genome pair run 2 ok 0.61 s 176 MB
query/M/metacooler/genome pair run 3 ok 0.60 s 176 MB
query/M/metacooler/genome pair run 4 ok 0.60 s 176 MB
query/M/metacooler/genome pair run 5 ok 0.61 s 176 MB
L: contig k141_124750 · genome MAG_0171 · genome pair MAG_0059 × MAG_0010
query/L/cooler/contig run 1 ok 0.81 s 250 MB
query/L/cooler/contig run 2 ok 0.80 s 250 MB
query/L/cooler/contig run 3 ok 0.81 s 250 MB
query/L/cooler/contig run 4 ok 0.80 s 250 MB
query/L/cooler/contig run 5 ok 0.78 s 250 MB
query/L/metacooler/contig run 1 ok 0.73 s 207 MB
query/L/metacooler/contig run 2 ok 0.72 s 207 MB
query/L/metacooler/contig run 3 ok 0.72 s 207 MB
query/L/metacooler/contig run 4 ok 0.71 s 207 MB
query/L/metacooler/contig run 5 ok 0.71 s 207 MB
query/L/cooler/genome run 1 ok 69.77 s 474 MB
query/L/cooler/genome run 2 ok 72.30 s 418 MB
query/L/cooler/genome run 3 ok 74.86 s 430 MB
query/L/metacooler/genome run 1 ok 0.76 s 209 MB
query/L/metacooler/genome run 2 ok 0.76 s 209 MB
query/L/metacooler/genome run 3 ok 0.75 s 211 MB
query/L/metacooler/genome run 4 ok 0.76 s 209 MB
query/L/metacooler/genome run 5 ok 0.75 s 209 MB
query/L/cooler/genome pair run 1 ok 74.85 s 440 MB
query/L/cooler/genome pair run 2 ok 70.16 s 427 MB
query/L/cooler/genome pair run 3 ok 68.02 s 434 MB
query/L/metacooler/genome pair run 1 ok 0.76 s 227 MB
query/L/metacooler/genome pair run 2 ok 0.76 s 230 MB
query/L/metacooler/genome pair run 3 ok 0.75 s 228 MB
query/L/metacooler/genome pair run 4 ok 0.76 s 227 MB
query/L/metacooler/genome pair run 5 ok 0.75 s 227 MB
XL: contig k141_1232184 · genome MAG_00859 · genome pair MAG_00229 × MAG_00056
query/XL/cooler/contig run 1 ok 4.19 s 1003 MB
query/XL/cooler/contig run 2 ok 4.19 s 1003 MB
query/XL/cooler/contig run 3 ok 4.18 s 1003 MB
query/XL/cooler/contig run 4 ok 4.14 s 1003 MB
query/XL/cooler/contig run 5 ok 4.16 s 1003 MB
query/XL/metacooler/contig run 1 ok 2.84 s 649 MB
query/XL/metacooler/contig run 2 ok 2.91 s 649 MB
query/XL/metacooler/contig run 3 ok 2.92 s 649 MB
query/XL/metacooler/contig run 4 ok 2.88 s 649 MB
query/XL/metacooler/contig run 5 ok 2.83 s 649 MB
query/XL/cooler/genome run 1 ok 1354.45 s 1207 MB
query/XL/metacooler/genome run 1 ok 2.36 s 648 MB
query/XL/metacooler/genome run 2 ok 2.36 s 648 MB
query/XL/metacooler/genome run 3 ok 2.36 s 648 MB
query/XL/metacooler/genome run 4 ok 2.35 s 648 MB
query/XL/metacooler/genome run 5 ok 2.34 s 648 MB
query/XL/cooler/genome pair run 1 ok 1402.17 s 1194 MB
query/XL/metacooler/genome pair run 1 ok 2.42 s 648 MB
query/XL/metacooler/genome pair run 2 ok 2.37 s 648 MB
query/XL/metacooler/genome pair run 3 ok 2.37 s 648 MB
query/XL/metacooler/genome pair run 4 ok 2.40 s 648 MB
query/XL/metacooler/genome pair run 5 ok 2.40 s 648 MB
Code
def pixels(path):
"""Rows and contact total of a dump: the count is the last column."""
if not path.exists() or path.stat().st_size == 0:
return 0, 0
last = pd.read_csv(path, sep="\t", header=None).iloc[:, -1]
return len(last), int(last.sum())
SHARE = {} # the share of the map's pixels each query returned: what the scan's cost is set against
agree = []
for ds in DATASETS:
with h5py.File(MAPS / f"{ds}.metacool", "r") as f:
nnz = int(f.attrs["nnz"])
for name, file in (("contig", "contig"), ("genome", "genome"), ("genome pair", "pair")):
got = {tool: pixels(QUERIES / f"{ds}.{tool}.{file}.tsv")
for tool in ("metacooler", "cooler") if finished("query", ds, tool, name)}
missing = [tool for tool in ("cooler", "metacooler") if tool not in got]
n, total = next(iter(got.values()), (None, None))
if n is not None:
SHARE[ds, name] = n / nnz
agree.append({
"dataset": ds, "query": name,
"pixels": "–" if n is None else f"{n:,}",
"share of the map's pixels": "–" if n is None else f"{n / nnz:.3%}",
"contacts": "–" if total is None else f"{total:,}",
"same as cooler": f"{' and '.join(missing)} did not finish" if missing
else "yes" if got["cooler"] == got["metacooler"] else "NO",
})
agree = pd.DataFrame(agree)
assert not (agree["same as cooler"] == "NO").any(), "the two tools returned different pixels"
show(wide("query"))| tool | task | S | M | L | XL |
|---|---|---|---|---|---|
| cooler | contig | 0.512 ± 0.008 (n=5) | 0.530 ± 0.007 (n=5) | 0.800 ± 0.012 (n=5) | 4.17 ± 0.02 (n=5) |
| metacooler | contig | 0.536 ± 0.009 (n=5) | 0.546 ± 0.009 (n=5) | 0.718 ± 0.008 (n=5) | 2.88 ± 0.04 (n=5) |
| cooler | genome | 1.87 ± 0.00 (n=5) | 14.2 ± 0.1 (n=5) | 72.3 ± 2.5 (n=3) | 1,354 (n=1) |
| metacooler | genome | 0.582 ± 0.004 (n=5) | 0.576 ± 0.009 (n=5) | 0.756 ± 0.005 (n=5) | 2.35 ± 0.01 (n=5) |
| cooler | genome pair | 1.95 ± 0.00 (n=5) | 14.9 ± 0.1 (n=5) | 71.0 ± 3.5 (n=3) | 1,402 (n=1) |
| metacooler | genome pair | 0.568 ± 0.013 (n=5) | 0.604 ± 0.005 (n=5) | 0.756 ± 0.005 (n=5) | 2.39 ± 0.02 (n=5) |
Code
show(agree)| dataset | query | pixels | share of the map's pixels | contacts | same as cooler |
|---|---|---|---|---|---|
| S | contig | 112 | 0.014% | 206 | yes |
| S | genome | 80,193 | 9.883% | 91,849 | yes |
| S | genome pair | 1,319 | 0.163% | 1,319 | yes |
| M | contig | 144 | 0.002% | 198 | yes |
| M | genome | 26,438 | 0.344% | 28,989 | yes |
| M | genome pair | 1,006 | 0.013% | 1,006 | yes |
| L | contig | 1 | 0.000% | 2 | yes |
| L | genome | 29,017 | 0.130% | 30,248 | yes |
| L | genome pair | 214 | 0.001% | 214 | yes |
| XL | contig | 3 | 0.000% | 3 | yes |
| XL | genome | 3,888 | 0.005% | 3,941 | yes |
| XL | genome pair | 11 | 0.000% | 11 | yes |
Code
show(wide("query", "peak_mb", digits=0))| tool | task | S | M | L | XL |
|---|---|---|---|---|---|
| cooler | contig | 150 ± 0 (n=5) | 162 ± 0 (n=5) | 250 ± 0 (n=5) | 1,003 ± 0 (n=5) |
| metacooler | contig | 151 ± 0 (n=5) | 158 ± 0 (n=5) | 207 ± 0 (n=5) | 649 ± 0 (n=5) |
| cooler | genome | 225 ± 0 (n=5) | 298 ± 7 (n=5) | 441 ± 29 (n=3) | 1,207 (n=1) |
| metacooler | genome | 157 ± 0 (n=5) | 160 ± 0 (n=5) | 209 ± 1 (n=5) | 648 ± 0 (n=5) |
| cooler | genome pair | 225 ± 0 (n=5) | 293 ± 5 (n=5) | 434 ± 7 (n=3) | 1,194 (n=1) |
| metacooler | genome pair | 166 ± 0 (n=5) | 176 ± 0 (n=5) | 228 ± 1 (n=5) | 648 ± 0 (n=5) |
6 · Genome-level aggregation
The genome × genome contact network: metacooler network --level genome, against the same edge list from cooler (dump --join, then an awk sum over the binning table). And the aggregated genome-level .cool (metacooler aggregate --level genome), which cooler has no command for.
%%bash
source bench.sh
by_genome='NR == FNR { g[$1] = $2; next }'
by_genome+=' { a = g[$1]; b = g[$4]; if (a > b) { t = a; a = b; b = t }; n[a "\t" b] += $7 }'
by_genome+=' END { for (k in n) print k "\t" n[k] }'
for ds in $DATASETS; do
d=$DATA/$ds m=$MAPS/$ds q=$QUERIES/$ds
bench aggregate "$ds" cooler "genome network" "$q.cooler.network.tsv" -- bash -o pipefail -c \
"$STAGE cooler dump -t pixels --join $m.cool | $STAGE awk -F'\t' '$by_genome' $d/contigs.tsv - > $q.cooler.network.tsv"
bench aggregate "$ds" metacooler "genome network" "$q.metacooler.network.tsv" -- \
metacooler network "$m.metacool" --level genome -o "$q.metacooler.network.tsv"
bench aggregate "$ds" metacooler "genome-level .cool" "$SCRATCH/$ds.genomes.cool" -- \
metacooler aggregate "$m.metacool" --level genome -o "$SCRATCH/$ds.genomes.cool"
doneaggregate/S/cooler/genome network run 1 ok 2.50 s 225 MB
aggregate/S/cooler/genome network run 2 ok 2.38 s 225 MB
aggregate/S/cooler/genome network run 3 ok 2.36 s 225 MB
aggregate/S/cooler/genome network run 4 ok 2.38 s 225 MB
aggregate/S/cooler/genome network run 5 ok 2.39 s 225 MB
aggregate/S/metacooler/genome network run 1 ok 0.63 s 205 MB
aggregate/S/metacooler/genome network run 2 ok 0.63 s 206 MB
aggregate/S/metacooler/genome network run 3 ok 0.62 s 206 MB
aggregate/S/metacooler/genome network run 4 ok 0.62 s 205 MB
aggregate/S/metacooler/genome network run 5 ok 0.62 s 206 MB
aggregate/S/metacooler/genome-level .cool run 1 ok 0.66 s 211 MB
aggregate/S/metacooler/genome-level .cool run 2 ok 0.66 s 212 MB
aggregate/S/metacooler/genome-level .cool run 3 ok 0.68 s 212 MB
aggregate/S/metacooler/genome-level .cool run 4 ok 0.66 s 212 MB
aggregate/S/metacooler/genome-level .cool run 5 ok 0.66 s 212 MB
aggregate/M/cooler/genome network run 1 ok 19.43 s 304 MB
aggregate/M/cooler/genome network run 2 ok 19.42 s 308 MB
aggregate/M/cooler/genome network run 3 ok 19.38 s 308 MB
aggregate/M/cooler/genome network run 4 ok 19.63 s 297 MB
aggregate/M/cooler/genome network run 5 ok 19.69 s 308 MB
aggregate/M/metacooler/genome network run 1 ok 1.31 s 631 MB
aggregate/M/metacooler/genome network run 2 ok 1.30 s 631 MB
aggregate/M/metacooler/genome network run 3 ok 1.30 s 630 MB
aggregate/M/metacooler/genome network run 4 ok 1.31 s 631 MB
aggregate/M/metacooler/genome network run 5 ok 1.30 s 631 MB
aggregate/M/metacooler/genome-level .cool run 1 ok 1.37 s 686 MB
aggregate/M/metacooler/genome-level .cool run 2 ok 1.37 s 687 MB
aggregate/M/metacooler/genome-level .cool run 3 ok 1.38 s 687 MB
aggregate/M/metacooler/genome-level .cool run 4 ok 1.35 s 686 MB
aggregate/M/metacooler/genome-level .cool run 5 ok 1.38 s 686 MB
aggregate/L/cooler/genome network run 1 ok 88.55 s 533 MB
aggregate/L/cooler/genome network run 2 ok 89.25 s 552 MB
aggregate/L/cooler/genome network run 3 ok 83.87 s 548 MB
aggregate/L/metacooler/genome network run 1 ok 2.98 s 1575 MB
aggregate/L/metacooler/genome network run 2 ok 2.93 s 1575 MB
aggregate/L/metacooler/genome network run 3 ok 2.94 s 1575 MB
aggregate/L/metacooler/genome network run 4 ok 2.98 s 1575 MB
aggregate/L/metacooler/genome network run 5 ok 2.99 s 1575 MB
aggregate/L/metacooler/genome-level .cool run 1 ok 3.06 s 1739 MB
aggregate/L/metacooler/genome-level .cool run 2 ok 3.04 s 1740 MB
aggregate/L/metacooler/genome-level .cool run 3 ok 3.10 s 1740 MB
aggregate/L/metacooler/genome-level .cool run 4 ok 3.09 s 1739 MB
aggregate/L/metacooler/genome-level .cool run 5 ok 3.09 s 1739 MB
aggregate/XL/cooler/genome network run 1 ok 1534.85 s 2269 MB
aggregate/XL/metacooler/genome network run 1 ok 12.38 s 3899 MB
aggregate/XL/metacooler/genome network run 2 ok 12.38 s 3898 MB
aggregate/XL/metacooler/genome network run 3 ok 12.30 s 3899 MB
aggregate/XL/metacooler/genome network run 4 ok 12.24 s 3898 MB
aggregate/XL/metacooler/genome network run 5 ok 12.31 s 3899 MB
aggregate/XL/metacooler/genome-level .cool run 1 ok 12.92 s 4259 MB
aggregate/XL/metacooler/genome-level .cool run 2 ok 13.05 s 4259 MB
aggregate/XL/metacooler/genome-level .cool run 3 ok 12.88 s 4259 MB
aggregate/XL/metacooler/genome-level .cool run 4 ok 13.00 s 4259 MB
aggregate/XL/metacooler/genome-level .cool run 5 ok 13.06 s 4259 MB
Code
def edges(path):
e = pd.read_csv(path, sep="\t", header=None, names=["a", "b", "n"], keep_default_na=False)
e[["a", "b"]] = np.sort(e[["a", "b"]].to_numpy(), axis=1)
return e.groupby(["a", "b"])["n"].sum()
check = []
for ds in DATASETS:
nets = {tool: edges(QUERIES / f"{ds}.{tool}.network.tsv").sort_index()
for tool in ("metacooler", "cooler") if finished("aggregate", ds, tool, "genome network")}
missing = [tool for tool in ("cooler", "metacooler") if tool not in nets]
net = next(iter(nets.values()), None)
check.append({
"dataset": ds,
"edges": "–" if net is None else f"{len(net):,}",
"contacts": "–" if net is None else f"{net.sum():,}",
"same as cooler": f"{' and '.join(missing)} did not finish" if missing
else "yes" if nets["cooler"].equals(nets["metacooler"]) else "NO",
})
check = pd.DataFrame(check)
assert not (check["same as cooler"] == "NO").any(), "the two networks differ"
display(show(wide("aggregate")))
display(show(wide("aggregate", "peak_mb", digits=0).rename(columns={"task": "task (peak MB)"})))
show(check)| tool | task | S | M | L | XL |
|---|---|---|---|---|---|
| cooler | genome network | 2.40 ± 0.06 (n=5) | 19.5 ± 0.1 (n=5) | 87.2 ± 2.9 (n=3) | 1,535 (n=1) |
| metacooler | genome network | 0.624 ± 0.005 (n=5) | 1.30 ± 0.01 (n=5) | 2.96 ± 0.03 (n=5) | 12.3 ± 0.1 (n=5) |
| metacooler | genome-level .cool | 0.664 ± 0.009 (n=5) | 1.37 ± 0.01 (n=5) | 3.08 ± 0.03 (n=5) | 13.0 ± 0.1 (n=5) |
| tool | task (peak MB) | S | M | L | XL |
|---|---|---|---|---|---|
| cooler | genome network | 225 ± 0 (n=5) | 305 ± 5 (n=5) | 544 ± 10 (n=3) | 2,269 (n=1) |
| metacooler | genome network | 206 ± 1 (n=5) | 631 ± 0 (n=5) | 1,575 ± 0 (n=5) | 3,899 ± 1 (n=5) |
| metacooler | genome-level .cool | 212 ± 0 (n=5) | 686 ± 1 (n=5) | 1,739 ± 1 (n=5) | 4,259 ± 0 (n=5) |
| dataset | edges | contacts | same as cooler |
|---|---|---|---|
| S | 64 | 1,000,000 | yes |
| M | 2,612 | 10,000,000 | yes |
| L | 96,536 | 30,000,000 | yes |
| XL | 1,237,358 | 100,000,000 | yes |
Summary
Code
t = timings()
RAM = ram(t)
PANELS = [
("create", ["cload pairs", "create pairs"], "create a map"),
("zoomify", ["zoomify"], "zoomify"),
("query", ["contig"], "dump a contig"),
("query", ["genome"], "dump a genome"),
("query", ["genome pair"], "dump a genome pair"),
("aggregate", ["genome network"], "genome network"),
]
STORE_STYLE = {
".metacool": dict(color=HUE["metacooler"], marker="o"),
".cool + binning tables": dict(color=HUE["cooler"], marker="o"),
".cool alone (no genomes)": dict(color=HUE["cooler"], marker="o", ls=(0, (1, 1.5)), mfc=SURFACE),
"one .cool per MAG": dict(color=HUE["cooler"], marker="D", ls=(0, (6, 2)), ms=4),
}
def memory_at(ax, ds):
"""Peak memory of every command on one dataset, cooler and metacooler on one row."""
rows = runs(t, dataset=ds)
top = 0.0
for i, (section, tasks, title) in enumerate(PANELS):
y = len(PANELS) - 1 - i
at = {}
for tool in ("cooler", "metacooler"):
r = rows[(rows["section"] == section) & rows["task"].isin(tasks) & (rows["tool"] == tool)]
if not len(r):
continue
ok, last = r[r["status"] == "ok"], r["status"].iloc[-1]
if len(ok):
at[tool] = ok["peak_gb"].max()
ax.scatter([at[tool]], [y], s=40, color=HUE[tool], zorder=3)
elif last in ("timeout", "oom") and r["peak_gb"].notna().any():
at[tool] = r["peak_gb"].max()
ax.scatter([at[tool]], [y], marker="|" if last == "timeout" else "X", s=90,
color=HUE[tool], lw=2.5, zorder=3)
if last == "timeout":
ax.annotate("", xy=(at[tool] * 1.25 + 0.3, y), xytext=(at[tool], y), zorder=3,
arrowprops=dict(arrowstyle="-|>", color=HUE[tool], lw=1.5, mutation_scale=9))
elif last == "skipped":
ax.annotate(f"{tool} not run", (0.02, y), xycoords=("axes fraction", "data"),
xytext=(0, -9), textcoords="offset points", color=INK2, fontsize=7.5)
if len(at) == 2:
ax.plot(list(at.values()), [y, y], color=GRID, lw=4, zorder=1, solid_capstyle="round")
top = max([top, *at.values()])
ax.set_yticks(range(len(PANELS)), [title for _, _, title in reversed(PANELS)])
ax.set_ylim(-0.7, len(PANELS) - 0.3)
ax.grid(axis="y", visible=False)
ram_line(ax, RAM, vertical=True)
ax.set_xlim(0, max(top, RAM if RAM == RAM else 0) * 1.12)
ax.set_xlabel(f"peak memory on {ds} (GB)")
fig, axes = plt.subplots(3, 3, figsize=(10.5, 10.4), layout="constrained")
axes = axes.ravel()
for ax, letter, (section, tasks, title) in zip(axes, "ABCDEF", PANELS):
rows = t[(t["section"] == section) & t["task"].isin(tasks)]
scaling(ax, rows)
seconds_axis(ax)
limit_line(ax, rows)
ax.set_title(f"{letter} {title}")
if ax not in (axes[0], axes[3]):
ax.set_ylabel("")
if ax in axes[:3]:
ax.set_xlabel("")
axes[0].legend(handles=TOOLS_LEGEND, loc="upper left")
g = axes[6]
create = runs(t, section="create")
scaling(g, create, "peak_gb")
log_y(g, "peak memory (GB)")
ram_line(g, RAM)
g.set_title("G memory: create a map")
memory_at(axes[7], DATASETS[-1])
axes[7].set_title(f"H memory: every command, {DATASETS[-1]}")
i = axes[8]
for option, r in STORAGE[STORAGE["option"].isin(list(STORE_STYLE))].groupby("option", sort=False):
i.plot(r["contigs"], r["mb"], label=option, **{"ms": 5, **STORE_STYLE[option]})
contig_axis(i)
log_y(i, "size on disk (MB)")
i.set_title("I size on disk")
i.legend(loc="upper left", fontsize=7.5)
save(fig, "commands")Every run finished within the 30 min time limit.
Not run. cooler cload pairs on XL: it has never finished on XL within the time limit (BENCH_SKIP_COOLER_CREATE); cooler zoomify on XL: no base map from cooler cload pairs on XL.
How many times faster metacooler runs than cooler (wall clock, means over runs); > where cooler was stopped at the time limit, which makes the ratio a lower bound. cooler’s pyramid is not run where it could not build the base map.
Code
FOLDS = [
("create", "cload pairs", "create pairs", "create a map from pairs"),
("zoomify", "zoomify", "zoomify", "zoomify"),
("query", "contig", "contig", "dump a contig"),
("query", "genome", "genome", "dump a genome"),
("query", "genome pair", "genome pair", "dump a genome pair"),
("aggregate", "genome network", "genome network", "genome network"),
]
def fold(section, cooler_task, metacooler_task, ds):
"""cooler's seconds over metacooler's, and how cooler's run ended: "ok",
"timeout" (the ratio is then a lower bound), "oom" or "skipped" (no ratio)."""
pick = runs(t, section=section, dataset=ds)
c = pick[(pick["tool"] == "cooler") & (pick["task"] == cooler_task)]
m = pick[(pick["tool"] == "metacooler") & (pick["task"] == metacooler_task) & (pick["status"] == "ok")]
if c.empty or m.empty:
return None
if (c["status"] == "ok").any():
return c.loc[c["status"] == "ok", "wall_s"].mean() / m["wall_s"].mean(), "ok"
if (c["status"] == "timeout").any():
return c["wall_s"].max() / m["wall_s"].mean(), "timeout"
for status in ("oom", "skipped"):
if (c["status"] == status).any():
return None, status
return None
def fold_text(f):
match f:
case None:
return ""
case value, "ok":
return f"{value:,.1f}×"
case value, "timeout":
return f"> {value:,.1f}×"
case _, "skipped":
return "cooler not run"
case _:
return "cooler out of memory"
rows = []
for section, ct, mt, label in FOLDS:
row = {"": label}
for ds in DATASETS:
row[ds] = fold_text(fold(section, ct, mt, ds))
rows.append(row)
FOLD_TABLE = pd.DataFrame(rows)
FOLD_TABLE.to_csv(RESULTS / "speedups.tsv", sep="\t", index=False)
show(FOLD_TABLE)| S | M | L | XL | |
|---|---|---|---|---|
| create a map from pairs | 1.5× | 2.9× | 20.4× | cooler not run |
| zoomify | 2.7× | 9.7× | 43.3× | cooler not run |
| dump a contig | 1.0× | 1.0× | 1.1× | 1.5× |
| dump a genome | 3.2× | 24.7× | 95.6× | 575.4× |
| dump a genome pair | 3.4× | 24.6× | 93.9× | 586.2× |
| genome network | 3.8× | 15.0× | 29.4× | 124.6× |
Replicates
How much the runs of a command vary: the number of runs n of each command on each dataset, and their coefficient of variation (sd / mean). A command runs at least 3 times unless a single run takes longer than the threshold below, or it is stopped at the time limit.
Code
ok = t[t["status"] == "ok"]
REPS = (
ok.groupby(["section", "tool", "task", "dataset"], sort=False)["wall_s"]
.agg(n="count", mean="mean", sd="std", fastest="min", slowest="max")
.reset_index()
)
REPS["cv"] = REPS["sd"] / REPS["mean"]
REPS.to_csv(RESULTS / "replicates.tsv", sep="\t", index=False)
REPS["cell"] = [f"n={n} · {cv:.1%}" if n > 1 else f"n=1 · {human(mean)}"
for n, cv, mean in zip(REPS["n"], REPS["cv"], REPS["mean"])]
grid = REPS.pivot(index=["section", "tool", "task"], columns="dataset", values="cell")
grid = grid.reindex(list(dict.fromkeys(zip(REPS["section"], REPS["tool"], REPS["task"]))))
show(grid.reindex(columns=[d for d in DATASETS if d in grid.columns]).reset_index())| section | tool | task | S | M | L | XL |
|---|---|---|---|---|---|---|
| create | cooler | cload pairs | n=5 · 0.8% | n=5 · 1.3% | n=3 · 2.0% | |
| create | metacooler | create pairs | n=5 · 6.0% | n=5 · 0.9% | n=5 · 2.0% | n=3 · 7.3% |
| convert | metacooler | create cool | n=5 · 0.8% | n=5 · 0.4% | n=5 · 1.1% | n=3 · 1.1% |
| convert | metacooler | flatten | n=5 · 0.6% | n=5 · 0.4% | n=5 · 0.8% | n=3 · 1.1% |
| zoomify | metacooler | zoomify | n=5 · 0.9% | n=5 · 0.3% | n=5 · 0.9% | n=3 · 0.5% |
| zoomify | cooler | zoomify | n=5 · 1.5% | n=3 · 1.2% | n=1 · 15.3 min | |
| query | cooler | contig | n=5 · 1.6% | n=5 · 1.3% | n=5 · 1.5% | n=5 · 0.5% |
| query | metacooler | contig | n=5 · 1.7% | n=5 · 1.6% | n=5 · 1.2% | n=5 · 1.4% |
| query | cooler | genome | n=5 · 0.2% | n=5 · 0.6% | n=3 · 3.5% | n=1 · 22.6 min |
| query | metacooler | genome | n=5 · 0.8% | n=5 · 1.6% | n=5 · 0.7% | n=5 · 0.4% |
| query | cooler | genome pair | n=5 · 0.2% | n=5 · 0.9% | n=3 · 4.9% | n=1 · 23.4 min |
| query | metacooler | genome pair | n=5 · 2.3% | n=5 · 0.9% | n=5 · 0.7% | n=5 · 0.9% |
| aggregate | cooler | genome network | n=5 · 2.3% | n=5 · 0.7% | n=3 · 3.4% | n=1 · 25.6 min |
| aggregate | metacooler | genome network | n=5 · 0.9% | n=5 · 0.4% | n=5 · 0.9% | n=5 · 0.5% |
| aggregate | metacooler | genome-level .cool | n=5 · 1.3% | n=5 · 0.9% | n=5 · 0.8% | n=5 · 0.6% |
54 of 58 command × dataset combinations ran at least 3 times; across them the coefficient of variation has a median of 0.9% and a maximum of 7.3% (metacooler create pairs on XL). 4 ran once, each for over 10 min: cooler zoomify on L, 15.3 min; cooler dump --join | awk (a genome) on XL, 22.6 min; cooler dump --join | awk (a genome pair) on XL, 23.4 min; cooler dump --join | awk (genome network) on XL, 25.6 min.
Numbers for the text
Filled in from this render; the sentences follow the manuscript draft.
- All benchmarks are on simulated data: four communities drawn by
simulate.py(generator v1) from a fixed seed each, of 1,000 to 2,000,000 contigs. No real reads, assembly or binning is used, and there are no accessions. - metacooler 0.2.0 creates metacool files 20-fold faster than cooler v0.12.1 creates the equivalent .cool file for an assembly of 200,000 contigs (19.7 s against 6.7 min), while also ingesting and indexing the /genomes table.
- At 2,000,000 contigs (20,000 genomes),
cooler cload pairswas not run in this render: it has never finished on XL within the time limit (BENCH_SKIP_COOLER_CREATE); metacooler took 110.9 s (peak 6.1 GB). - Every command ran on one machine with 7.7 GB of RAM (2 cores). On 2,000,000 contigs, the largest peak of each tool, among the runs that finished:
cooler dump --join | awk (genome network), 2.2 GB andmetacooler zoomify, 6.6 GB. - On disk, at 2,000,000 contigs: 161 MB for the .metacool, 277 MB for the .cool and its binning tables (205 MB for the .cool alone), and 142 MB for 1,000 per-MAG .cool files, which hold only 68% of the contacts and 16% of the contigs.
- With cooler’s command line, extracting a single MAG from a genome-agnostic .cool file is a linear scan of every pixel (
cooler dump --join, filtered against the binning table): 1.9 s at 1,000, 14.2 s at 20,000, 72.3 s at 200,000 and 22.6 min at 2,000,000 contigs.metacooler dump -r <MAG>takes 582 ms at 1,000, 576 ms at 20,000, 756 ms at 200,000 and 2.4 s at 2,000,000 contigs. On the simulated 2,000,000-contig map that is 575.4×. cooler’s scan reads every pixel of the map, metacooler only the MAG’s (0.01% of them here), so the fold depends on how a map’s pixels split between its genomes: here the simulated community sets that, not a real sample. - A full genome-by-genome aggregated matrix is computed in 13 s for a simulated community of 20,000 genomes (1,000 MAGs and 19,000 small bins; 2,000,000 contigs, 100 M read pairs;
metacooler aggregate --level genome). Its edge list takes 12.3 s (metacooler network --level genome), against 25.6 min for cooler’s scan. - Timings are means over 3–5 runs (one, for a run over 10 min); their coefficient of variation has a median of 0.9%.
Artifacts
Beside this page, in _results/: published with it on the documentation site, and attached to every release and workflow run.
Code
t = timings()
problems = t[t["status"] == "error"]
if len(problems):
display(Markdown(f"**{len(problems)} run(s) failed** — see their logs in `_work/logs/`:"))
display(show(problems[["section", "dataset", "tool", "task", "command"]]))
def link(path):
"""A file's name, linked when the page can reach it: written beside the page (the
notebook's directory), as _results/ is unless BENCH_RESULTS moves it."""
name, href = str(path.relative_to(RESULTS)), os.path.relpath(path)
return name if href.startswith("..") else f'<a href="{href}">{name}</a>'
files = sorted(p for p in RESULTS.rglob("*") if p.is_file())
show(pd.DataFrame({
"file": [link(p) for p in files],
"kB": [p.stat().st_size / 1e3 for p in files],
}), **{"kB": "{:,.1f}"})| file | kB |
|---|---|
| datasets.tsv | 0.9 |
| figures/commands.png | 432.8 |
| figures/commands.svg | 245.2 |
| figures/create.png | 113.3 |
| figures/create.svg | 64.8 |
| figures/datasets.png | 134.0 |
| figures/datasets.svg | 82.7 |
| replicates.tsv | 5.4 |
| speedups.tsv | 0.3 |
| storage.tsv | 0.9 |
| timings.tsv | 102.3 |
| file | what |
|---|---|
timings.tsv |
every timed run: command, status, wall/CPU seconds, peak memory, output size, versions, machine |
datasets.tsv |
the datasets table above |
storage.tsv |
the size on disk of each way to keep the map, per dataset |
speedups.tsv |
the fold table above |
replicates.tsv |
per command and dataset: runs, mean, sd, fastest, slowest, coefficient of variation |
figures/ |
every figure, PNG and SVG |