[Claptar]

Closes #93. Part of #87. For 0.17.5: merge after 0.17.4 is tagged.

What changed, by finding

• F011 (high): CSC inputs are read once each, in blocks of entries sized by --chunk, and the output's columns put together through column-range buckets in an HDF5 scratch file beside the output, instead of one read per column per input. Every output column's size is known up front, so each bucket is a fixed region of three contiguous datasets (_ColumnBuckets): three datasets open, and the scratch exactly 16 bytes plus the value per entry. A column holding more than a block is in a bucket of its own and streamed; if an input stored it out of order, the bucket is sorted whole. The output stays canonical (sorted rows). The free-space check now counts the scratch (about 16 bytes plus the value per stored entry of the largest CSC matrix), for a Zarr output too, since the scratch is HDF5; that check moved from commands/concat.py into concat_on_disk, which knows the encodings. Tested: against anndata.concat on three CSC inputs (reordered and missing genes, empty columns, several buckets) on h5ad, zarr2 and zarr3, inner and outer; index dtypes and compression kept; reads of X don't grow with n_var at fixed nnz on h5ad and Zarr; open HDF5 datasets don't grow with the bucket count; the scratch on disk equals the estimate at --chunk 1 and 1024.
• F075 (medium): dense X, layers and obsm arrays are created with the first input's chunking and compression (dataset_create_kwargs + fit_chunks, as varm already was). Tested: gzip and (3, 2) chunks survive on all three backends, inner and outer; values equal anndata's.
• F076 (medium): inputs are opened through a pool of at most MAX_OPEN_INPUTS: 64, or a quarter of RLIMIT_NOFILE if that is fewer (as split's MAX_OPEN_OUTPUTS), since each open HDF5 input costs about a megabyte. Nothing holds an object across inputs any more: every element is looked up again on each pass. When the pool is full it closes the most recently used input, since every pass scans first to last (LRU would reopen every input on every pass: 344 opens against 56 with one input over the limit). Identical var lists are held once; merge comparisons stream, holding at most two values. An EMFILE names ulimit -n. Tested: a rich concat with the pool at 1 equals the one at 64 and anndata's (h5py invalidates closed handles, so any held object would fail); 100 inputs under a 64-descriptor limit in a subprocess; peak open HDF5 files independent of input count; reopen count with one input over the limit.
• F113 (low, D2): an outer join's var is the inputs' order when all have the same var in the same order, else the sorted union, as pandas/anndata give it; empty inputs have no say. Tested: against anndata for reordered, new-gene and subset cases, CSR/CSC/dense, all backends.
• F114 (low, D2): merged categories follow anndata: the first input's order when its categories already cover every input's and need no cast (integers meeting floats are recast), else natsorted. util/names.natural_key now compares text in NFD form, as natsort does ("Müller" after "Muscle"). Ordered categoricals stay ordered only when every input has the same categories in the same order; otherwise they are written unordered with a warning (anndata gives plain values). Tested: against anndata on text, integer and float categories, outer joins where the first input lacks the column, and five ordered cases.
• F116 (low): --fill-value widens a dense matrix only when some input actually leaves cells unfilled, and only if the dtype cannot hold the value (exactly, for integers and booleans; within range, for floats); then float64. Tested: inner join keeps int32 as anndata does (fill -1 and NaN); outer join over nine dtype/fill pairs, values equal anndata's.
• F118 (low): varp follows --merge as varm does. Dense entries are compared by digest on the target genes (dtype included, as anndata); sparse ones by an order-free digest of their nonzero triples, so CSR equals CSC and float32 equals float64, as in anndata. A kept sparse entry already on the target genes is copied; one in another gene order is rebuilt through buckets. Without --merge, varp is dropped with a note; obsp is always dropped. Tested: against anndata for all four strategies × dense/CSR/CSC with a permuted gene order, a rebuild from shuffled genes on h5ad and Zarr, the value comparisons, an outer join, and the notes.
• F132 (low): CSR concat's row pointers are one preallocated int64 array, and recast's sparsify keeps per-major counts in a numpy array. Tested: allocation guards: 12.7 B/row for concat (main: 56.6) and 14.3 B/row for sparsify (main: 46.8).

Also: a matrix (X, a layer, an obsm array) whose rows differ from its input's obs is now refused before the output exists, naming the input; dense obsm used to be stacked out of line with the cells.

None of the findings failed to reproduce: each new test failed on main before the fix.

Outside core/concat.py

• src/adata/core/recast.py: the F132 sparsify fix (counts in an int64 array; an unused List import dropped).
• src/adata/util/names.py: natural_key NFD-normalises (two lines). This moves accented labels to anndata's order wherever natural_sorted is used (conversion, table and pickle imports too).
• src/adata/commands/concat.py: passes space_check through; the check itself moved into concat_on_disk.
• src/adata/cli.py: concat's help text (varp, --fill-value).
• docs/reference/commands.md: the concat section.

Tests run locally

• COLUMNS=80 uv run pytest -m "not integration" --cov=adata --cov-fail-under=90: 2818 passed, 1 skipped; total coverage 95.8 % (run before merging main; after the merge, which brought only the 0.17.4 release files, the docs and concat test files pass again: 647).
• ADATA_REQUIRE_VERSION_FIXTURES=1 uv run pytest tests/test_concat_versions.py -m integration: 24 passed (a new read-back of a CSC concat with varp and an ordered categorical, by anndata 0.8 to 0.13, h5ad and Zarr).
• ADATA_REQUIRE_VERSION_FIXTURES=1 uv run pytest tests/test_anndata_versions.py -m integration -k concat: 12 passed.

Changelog entry

Changed

• An outer join's var and merged categories are in anndata's order (#93).
  concat --join outer keeps the inputs' var order when they all share it
  and sorts the union otherwise, as anndata.concat does; it used to be the
  order of first appearance. Merged categories keep the first input's order
  when it already holds every category, and are natsorted otherwise
  (accented letters as natsort orders them, "Müller" after "Muscle"). An
  ordered categorical whose inputs disagree on the order is written
  unordered, with a warning, where it used to keep the first input's order.

Fixed

• concat reads CSC inputs once, not once per column (#93).
  Each column of each input was a separate read that decompressed a whole
  chunk: 37 times the inputs' size was read on h5ad and 572 times on Zarr,
  so atlas-sized CSC inputs never finished. Inputs are now read in blocks
  and the columns put together in a scratch file beside the output, which
  the free-space check counts.
• concat takes any number of inputs (#93).
  Every input was held open, so concat failed with "Too many open files" at
  about 253 h5ad inputs on macOS (1,020 on Linux), and memory grew with the
  number of inputs. Inputs are now opened as they are needed, at most 64 at
  once.
• Dense matrices keep their compression in concat (#93).
  Dense X, layers and obsm arrays were written uncompressed and unchunked,
  whatever the inputs were: 34 times larger for a matrix of zeros.
• concat --merge keeps varp (#93).
  varp was always dropped, with a reason that holds only for obsp. It now
  follows --merge as varm does, sparse entries compared by value as
  anndata compares them.
• --fill-value no longer makes integer matrices float64 for nothing (#93).
  An integer X became float64 even on an inner join, where nothing is
  filled. It now widens only when an outer join fills cells with a value
  the dtype cannot hold.
• Less memory per row in concat and recast --layout csr (#93).
  Row pointers and per-row counts were Python lists, about 36 bytes a row
  against 8: 400 MB more at 10M cells.

Independent review

A fresh subagent with only the diff, the audit entries and AGENTS.md reviewed the PR. It ran a 300-case CSC fuzz (h5ad, zarr2 and zarr3 mixed; zero-row and zero-var inputs; int64 pointers; unsorted columns; --chunk 1 to 1024; pool sizes 1 to 64), a 120-case varp/varm merge fuzz against anndata, and sharded Zarr inputs; all correct. Its findings and what was done:

1. The scratch was larger than the free-space estimate (1.6× at the default --chunk, 16× at --chunk 1; each bucket's growable datasets took a whole chunk). Fixed: fixed bucket regions in contiguous datasets; a test checks the size on disk against the estimate.
2. Memory grew with n_obs / --chunk: about 57 KB per open bucket. Fixed by the same change; a guard checks that open HDF5 datasets don't grow with the bucket count (old code: 16 to 124).
3. The pool could hold 1,024 inputs at about 1 MB each. Fixed: capped at 64.
4. Accented category labels were out of natsort's order. Fixed (NFD). Superscript digits, which natsort reads as numbers, are still text here; left as is.
5. A sparse varp entry against a dense one: anndata's answer depends on input order; this PR never finds them equal. Kept, and documented.
6. Integer categories meeting float ones kept the first input's order, where anndata recasts and reorders. Fixed; the test now stores numeric categories as numbers.
7. Dense obsm out of step with obs was stacked misaligned (pre-existing). Fixed: refused before writing.

Also: a float32 matrix with a fill beyond its range now becomes float64 rather than holding infinity. Outer-join var columns/varm/varp still count an input lacking genes as lacking the entry (B3a's rule; documented). _merge_values is kept only as the value-returning wrapper test_commands_coverage.py tests.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-30T17:10:46.263501Z We couldn’t complete this review. Comment @codex review to try again.	c95fd31	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[github-actions]

Test Results (py3.13)

3 285 tests  +155   3 281 ✅ +155   12m 43s ⏱️ + 1m 49s
    1 suites ±  0       4 💤 ±  0 
    1 files   ±  0       0 ❌ ±  0 

Results for commit 0074d79. ± Comparison against base commit be38de4.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (py3.12)

3 285 tests  +155   3 281 ✅ +155   14m 33s ⏱️ + 1m 22s
    1 suites ±  0       4 💤 ±  0 
    1 files   ±  0       0 ❌ ±  0 

Results for commit 0074d79. ± Comparison against base commit be38de4.

♻️ This comment has been updated with latest results.