Data Backend for Expedited Acquisition and Neighbourhood Search

Installation

make install          # data-beans + data-beans-sim, HDF5 if detected
make install HDF5=off # without .h5/.h5ad support
make help             # what was detected on this host, and the overrides

or from crates.io (CPU, no HDF5):

cargo install data-beans
cargo install data-beans --features sim   # data-beans-sim

Here, we implement a wrapper for sparse matrix backends for swift data access by rows and columns without populating everything in memory. Being inspired by anndata-rs, we use hdf5 and zarr as the back-end storage format. The hdf5 file or .zarr.zip archive (or plain zarr directory with --no-zip) is organized as follows:

(root)
    ├── nrow
    ├── ncol
    ├── nnz
    ├── by_column
    │   ├── data
    │   ├── indices (row indices)
    │   └── indptr (column pointers)
    └── by_row
        ├── data
        ├── indices (column indices)
        └── indptr (row pointers)

Views

One rule decides, for every command with a view:

• qc, stat and histogram open their view on a terminal, and run straight through otherwise;
• row-names, column-names, list-h5 and list-zarr open a browser on a terminal in place of printing (a file -o is still written);
• merge-backend, merge-mtx, align-data, subsample, split and every from-* show a plan of what they write (or wrote) first, and write only on Enter;
• columns, rows, subset and subset-* open the explorer to pick from only when the command line selects nothing;
• --batch-process, or DATA_BEANS_BATCH=1 in the environment, asks for no view: put export DATA_BEANS_BATCH=1 at the top of a script started from a terminal, whose commands would otherwise take it for yours;
• a terminal is stdin and stdout both a terminal, and not TERM=dumb. Without one (output redirected, say) a view is skipped with one line on stderr; a picker with nothing to pick from stops and names the flags that select instead.

Results go to stdout and nothing else does: logs, the QC histograms, the skipped-view line and errors go to stderr. The MCP server's commands always run in batch.

data-beans qc shows each cell QC rule with its edge, how many cells it fails and how many only it removes; the selected rule's histogram in three layers (every cell, the cells passing the other rules, the cells only it removes); total counts against detected genes; and the removed cells by the rules they fail. ←/→ move a rule along its MAD band, m switches it to a value of its own (e types one), o turns it off or on, p shows the --qc-* flags that decide the same cells (c copies them to the clipboard), s saves the figure as PDF and PNG, Enter writes the kept cells to -o (or asks where, offering a free name; it never replaces a file or rewrites the input) and q cancels. On leaving it prints the same run as a --batch-process command line.

qc also does what squeeze did: its nnz floors are --qc-min-cell-nnz (cells) and --qc-feature-min-cells (genes), and --qc-auto-cutoff puts both at the trough of their histograms. Several files (qc a.zarr.zip,b.zarr.zip) are pooled first, as merge-backend merges them (--common-rows keeps the genes all share), and QC'd as one; the output comes with their batch labels.

stat explores per-row or per-column statistics; s saves its figure (the four histograms and mean against sd, marked entries in the accent). histogram opens the same explorer on the --stat and --dim asked.

Names and trees. row-names and column-names list a matrix's names, list-h5 and list-zarr a file's hierarchy (Enter or the arrows open and close a group). / filters (a regular expression when the text is one), Space marks, a marks every entry shown, w writes the marked entries (or, with none marked, those shown; a tree's as paths) and y copies them.

Plans. Before a command writes, its plan says what it will make: merge-backend and merge-mtx each input's rows and columns, how many rows every input has and how many only some, and the output's size (c keeps the common rows, s squeezes after); align-data each file's data type and column group, and the rows and columns each keeps (r sets the data types); subsample the size it draws (n cells, g genes, s seed); split each fold's train and test cells (t test fraction, k folds, s seed, g grid under --coord). After a from-* build, the import preview shows its rows' and columns' non-zeros on log bins with where the squeeze cutoffs fall (s squeeze, r and c the cutoffs); q there deletes the build. A plan that cannot be written says why, and Enter waits; elsewhere q writes nothing.

In kitty, iTerm2, WezTerm and Ghostty the plots are the figure itself, drawn as images; other terminals get text plots. DATA_BEANS_TUI_IMAGES=0 keeps text, =1 asks any terminal. --figure writes the same figure without the view, and the MCP qc and statistics tools return it to an agent.

Each view runs as a detachable session: Ctrl+Alt+D leaves it running (as does a dropped connection), and data-beans attach returns to it; data-beans sessions lists them and data-beans kill stops one as its quit key does. --no-detach runs a view in the terminal only.

Agents (MCP)

data-beans mcp serves data-beans to an AI agent as Model Context Protocol tools over stdio, for any MCP client: Claude Code, Claude Desktop, Cursor, VS Code, Gemini CLI, Codex CLI and others. The server talks only to the client that started it, over stdin and stdout; data-beans sends nothing anywhere itself. What leaves your machine is up to the agent's own client and model.

Connect a client

The server is in the default build (the mcp feature; cargo install data-beans has it). Importing .h5ad and 10x .h5 files needs --features hdf5 too.

The server keeps its records in a project folder: its working directory, or LEGUME_MCP_PROJECT. Set it for clients that start servers somewhere else (Claude Desktop, say).

Most clients (Claude Desktop, Cursor, Gemini CLI) take an mcpServers entry:

{
  "mcpServers": {
    "data-beans": {
      "command": "data-beans",
      "args": ["mcp"],
      "env": { "LEGUME_MCP_PROJECT": "/path/to/project" }
    }
  }
}

Claude Code, from the project folder:

claude mcp add data-beans -e LEGUME_MCP_PROJECT="$PWD" -- data-beans mcp

VS Code (.vscode/mcp.json):

{
  "servers": {
    "data-beans": {
      "type": "stdio",
      "command": "data-beans",
      "args": ["mcp"],
      "env": { "LEGUME_MCP_PROJECT": "${workspaceFolder}" }
    }
  }
}

Codex CLI (~/.codex/config.toml):

[mcp_servers.data-beans]
command = "data-beans"
args = ["mcp"]
env = { LEGUME_MCP_PROJECT = "/path/to/project" }

What the agent is told

The server hands these instructions to the client when it connects. Clients that do not pass MCP instructions on to the model can paste them into an AGENTS.md or the system prompt:

data-beans stores sparse single-cell matrices, features (genes, peaks) as rows and cells as columns, in .zarr.zip, .zarr or .h5 backends.

Look before you derive:
1. info gives a backend's size and metadata; list_contents shows what an outside .zarr or .h5 holds before you import it.
2. names pages through row or column names, with a regex filter, to find the exact names to pass on.
3. statistics summarizes each row or column (nnz, tot, mu, sig) with quantiles and the top entries by a chosen statistic; values reads the counts of a few named rows or columns.

Quality control: qc runs the cell QC senna and pinto apply (a near-empty floor, then robust MAD bands on detected genes, total counts and the mito fraction). Call it without `out` first: it reports each metric's median, the band and how many cells fall outside, and writes nothing. Adjust n_mads or the caps until the drops are what the data warrant, then call it with `out` to write the kept cells as a new backend. Pass `figure` to see the same settings as a figure, returned as an image: each rule's histogram with its edge, the cells it alone removes, counts against genes, and the removed cells by the rules they fail; look at it before and after tuning. statistics takes `figure` too: the histograms of nnz, sum, mean and sd, and mean against sd.

Derive new backends with import (mtx, zarr triplets, fragments, h5ad, 10x; say in `source` where the input came from), subset, subsample, split, convert and merge. Each writes to `out`, which must not exist yet: no tool replaces a file. Leave `out` out and the output goes in today's results folder, named after its input and the tool. Outputs are .zarr.zip archives unless `out` ends in .zarr (a directory) or .h5. Paths are resolved against the server's working directory; results give the paths written. On a big store, pass `background` to these (and to qc with `out`): the call returns a job at once, with the paths it will write.

A person may be watching (`data-beans mcp watch`): a "Note from the user" in a result is theirs to you; take it into account. A call they stopped fails with "stopped by the user": do not run it again unasked.

After these, legume-mcp adds its own notes: where the records go (below), and how to follow a background job.

Tools

Tool	What it does	Writes
info	a backend's size (rows, columns, non-zeros) and metadata
list_contents	the groups and arrays in a .zarr, .zarr.zip or .h5, before importing it
names	row or column names, paged, filtered by a case-insensitive regex
statistics	per-row or per-column nnz, total, mean and sd: quantiles and the top entries; with figure, their histograms and mean against sd as an image	a .parquet when out is given; the figure
values	the counts of up to 100 named or indexed rows or columns	a table when out is given
qc	senna's and pinto's cell QC: the near-empty floor, MAD bands on genes, counts and mito fraction, caps, the gene floor; several backends pooled first; with figure, the QC figure as an image	the kept cells when out is given (and a pool's batch labels); a dry run otherwise; the figure
import	a backend from .mtx, zarr triplets, fragments, .h5ad or 10x .h5, its source noted	a backend
subset	some rows or columns: by names, a pattern (regex) or indices (ranges such as 0-19612)	a backend
subsample	cells and/or genes drawn at random	a backend
split	cells split train/test, or into K folds; by region or group with options	the halves
convert	zarr ↔ h5, .zarr ↔ .zarr.zip	a backend
merge	backends side by side, batch-labelled; common_rows keeps the rows all share	a backend and its .batch.gz
notebook	the agent's plan, observations and conclusions (from legume-mcp)	the day's notebook.md
job_status, job_wait, job_cancel	follow or stop a step started with background (from legume-mcp)

Every tool that writes a backend (qc with out, import, subset, subsample, split, convert, merge) takes background: it returns at once with the job and the paths it will write (written, and merge's batches, qc's report). Wait for job_wait to say done before using them: a .zarr or .h5 is there, half written, while the job runs. What the command printed (qc's medians and bands, say) is the job's output. Until the job ends, its outputs are no other call's to write.

No tool replaces a file: an output that is an input, or anything named after the output beside it, is refused.

Records

The server keeps the project's records the way legume-mcp lays them out, after Noble's guide to organizing computational biology projects [1]:

• outputs the agent does not name go in results/<YYYY-MM-DD>/, named after their input and the tool (pbmc.imported.qc.subset.zarr.zip);
• every step that writes goes into the day's results/<YYYY-MM-DD>/runall.sh, with paths relative to the project and steps skipped when their outputs exist, so a copy of the project replays the session;
• results/INDEX.tsv lists each output with a one-line summary;
• import notes where its input came from (the agent's source: a URL, an accession, a version) in data/README.md;
• the agent keeps results/<YYYY-MM-DD>/notebook.md through the notebook tool.

LEGUME_MCP_RECORDS=off keeps no records.

[1] W. S. Noble, "A quick guide to organizing computational biology projects", PLoS Comput Biol 5(7): e1000424 (2009). doi:10.1371/journal.pcbi.1000424

Watching

data-beans mcp watch

follows a running data-beans server in a detachable session: the agent's calls as they come, each one's log and result, the notebook. s stops the selected call, n leaves the agent a note (it arrives with its next result, and goes into the notebook), o opens a qc call in the QC view, on the data the agent looked at with its settings (try others, c copies their flags for a note back; Enter writes only where you say), and a backend another call wrote in the statistics explorer, F follows again after you move. --watch-only offers no stop or note. With several servers running it asks which; data-beans mcp watch <folder> names one. LEGUME_MCP_WATCH=off keeps a server from being watched.

The same on the command line

data-beans qc data.zarr.zip                            # the view, on a terminal
data-beans qc data.zarr.zip --batch-process            # summary only
data-beans qc data.zarr.zip --batch-process -o clean   # and the kept cells
data-beans qc data.zarr.zip --batch-process --figure qc        # qc.pdf and qc.png
data-beans qc a.zarr.zip,b.zarr.zip --batch-process -o both    # pooled, with both.batch.gz
data-beans stat data.zarr.zip -s row --figure genes.png        # the statistics figure

What squeeze -r R -c C -o out did, without the MAD bands or the mito metric:

data-beans qc data.zarr.zip --batch-process -o out \
  --qc-feature-min-cells R --qc-min-cell-nnz C \
  --qc-mad-on-genes=false --qc-mad-on-counts=false --qc-mito-pattern=

qc starts from --qc-mito-pattern '(?i)^MT-'; an empty --qc-mito-pattern= turns the mito metric off.