vsparse

vsparse provides VCSCArray/VCSRArray: Value-Compressed Sparse Column/Row (VCSC/VCSR) array types, implemented in NumPy and accelerated with Numba. The array types are standalone -- they can be built from and converted to SciPy sparse arrays and used entirely without AnnData. vsparse also ships an optional AnnData integration (from_anndata, to_layer, and the VCSCAnnData subclass) for projects that want compressed arrays backed directly into an AnnData object.

VCSC/VCSR are compressed-sparse layouts inspired by IVSparse's VCSC. In addition to the usual compressed-sparse pointer/index arrays, nonzero values within each major-axis slice (columns for VCSC, rows for VCSR) are deduplicated: each unique value is stored once, alongside the list of minor-axis positions that share it. This is a strict memory win whenever values repeat heavily within a slice — as is typical for integer count matrices, e.g. single-cell RNA-seq counts.

Install

Install from PyPI:

pip install vsparse
# or with uv
uv add vsparse

For the CUDA support described below (needs an NVIDIA GPU; pulls in CuPy):

pip install "vsparse[cuda]"
# or with uv
uv add "vsparse[cuda]"

From source:

pip install git+https://github.com/meyer-lab/vsparse.git
# or with uv
uv add git+https://github.com/meyer-lab/vsparse.git

For local development:

git clone https://github.com/meyer-lab/vsparse.git
cd vsparse
uv sync --all-extras --dev

Quickstart

Working with VCSC / VCSR arrays

VCSCArray/VCSRArray are standalone array types -- they only need a SciPy sparse array (or an AnnData object, if you have one) to be built, and none of the operations below require AnnData at all:

import vsparse
import scipy.sparse as sp

# Build directly from a SciPy sparse array -- no AnnData involved
csc = sp.random(1000, 500, density=0.1, format="csc")
v = vsparse.VCSCArray.from_scipy(csc)

# Or build from an AnnData object or SciPy sparse array
adata = ...  # an AnnData object
v = vsparse.from_anndata(adata)  # VCSCArray (column-compressed) from adata.X
vr = vsparse.from_anndata(adata, format="csr")  # VCSRArray (row-compressed)

# Transposition is zero-copy (swaps major/minor axes and shares buffers)
vr = v.T  # VCSRArray

# Scalar arithmetic & math
v2 = v * 2.0  # Scalar multiplication
v_div = v / 2.0  # Scalar division
v_neg = -v  # Negation
v_log = v.log1p()  # Elementwise log1p

# Matrix & vector products (Numba-parallelized)
y = v @ x  # Matrix-vector: (n_rows, n_cols) @ (n_cols,) -> (n_rows,)
y_left = x @ v  # Vector-matrix: (n_rows,) @ (n_rows, n_cols) -> (n_cols,)
Y = v @ B  # Matrix-matrix: (n_rows, n_cols) @ (n_cols, k) -> (n_rows, k)
Y_left = B @ v  # Matrix-matrix: (k, n_rows) @ (n_rows, n_cols) -> (k, n_cols)

# Slicing
col_slice = v[:, [1, 3, 5]]  # Fast major-axis slicing (returns VCSCArray)
sub = v[0:10, 0:10]  # 2D slicing (falls back to scipy)

# Conversion & layers
sp_csc = v.to_scipy()  # -> scipy.sparse.csc_array (or to_csr())
dense = v.toarray()  # -> numpy.ndarray
vsparse.to_layer(adata, v, key="counts_vcsc")  # Attach to AnnData layer

VCSCAnnData: AnnData with direct VCSC/VCSR backing

For projects that want the compressed array wired directly into AnnData, vsparse.VCSCAnnData is an AnnData subclass whose X (and optionally raw_X) is backed directly by a VCSCArray or VCSRArray:

import vsparse

va = vsparse.VCSCAnnData.from_anndata(adata)  # Compresses X and raw.X
va.X  # VCSCArray
va.raw_X  # VCSCArray (separate from anndata's .raw)

# Persist to HDF5 (.h5ad) or Zarr with default zstd compression
# (stored as IVCSC/IVCSR, see below)
va.write_h5ad("compressed.h5ad")  # Read back with VCSCAnnData.read_h5ad
va2 = vsparse.VCSCAnnData.read_h5ad("compressed.h5ad")

va.write_zarr("compressed.zarr")  # Read back with VCSCAnnData.read_zarr
va3 = vsparse.VCSCAnnData.read_zarr("compressed.zarr")

# Escape hatch back to standard AnnData
plain = va.to_anndata()

Byte-Packed On-Disk Format (IVCSC / IVCSR)

write_h5ad / write_zarr always store X/raw_X as IVCSC (from VCSC) or IVCSR (from VCSR). This byte-packs the minor-axis indices with delta + varint encoding (inspired by IVSparse's IVCSC).

It is purely an archival storage format: read_h5ad / read_zarr decompress the indices on load and return a standard VCSCArray/VCSRArray.

# Writes delta+varint byte-packed indices
va.write_h5ad("archived.h5ad")

# Reads back directly as an ordinary VCSCAnnData
va_loaded = vsparse.VCSCAnnData.read_h5ad("archived.h5ad")

Loading, subsetting, and normalizing

Loading, subsetting, and normalization are separate steps, so they can be mixed and matched. vsparse.load_packed decodes an IVCSR/IVCSC-stored .h5ad into a VCSCAnnData; normalization is then a view over the stored array (.normalized("parafac2") reproduces parafac2.normalize.prepare_dataset's depth normalization and log10 transform) that is never materialized.

va = vsparse.load_packed("archived.h5ad")
norm = va.X.normalized("parafac2")  # lazy view; norm @ B works without densifying

load_packed can also subset as it decodes: obs_idx/var_idx take a boolean mask, an array of integer positions, or a slice, and give the same result as indexing a full load, without ever building the full matrix. With IVCSR, only the selected cells are decoded and unselected genes are dropped in the same pass (IVCSC is the same with the axes swapped). Every packed byte is still read, so reading the stream sets the floor: on a 1.26M-cell file, a 1% cell subset decodes X in 0.21 s against 0.50 s for all of it.

import anndata, h5py

with h5py.File("archived.h5ad") as f:  # read only obs to build the mask
    obs = anndata.io.read_elem(f["obs"])
va_blood = vsparse.load_packed("archived.h5ad", obs_idx=obs["tissue"] == "blood")

Statistics sidecar

Every VCSCArray/VCSRArray carries per-row and per-column summary statistics as arr.stats (a vsparse.VCSStats): row_sums/col_sums (total counts), row_nnz/col_nnz (n-genes/n-cells by counts), and col_depth_sums (per-gene sums of depth-normalized counts). They are computed on first access and written next to X/raw_X by write_h5ad/write_zarr whether or not they were ever computed or a normalized view was ever made, so a file read back has them without touching the data. Every recipe's per-cell and per-gene scales (a/b) are derived from them, so only centering/scaling recipes still need a pass over the data. Files without a usable sidecar (older versions, other tools) load as before, and the statistics are recomputed when first needed.

va = vsparse.VCSCAnnData.read_h5ad("archived.h5ad")
keep = va.X.stats.row_sums >= 500  # QC filter from the sidecar alone

CUDA (to_gpu)

A normalized view moves onto an NVIDIA GPU with .to_gpu(), keeping its value-compressed layout there rather than expanding to one float per nonzero — so a matrix that only fits in device memory compressed still fits. Everything floating-point on the device is float32.

gpu = vsparse.VCSRArray.from_scipy(counts).normalized("parafac2").to_gpu()
out = gpu @ B  # float32 CuPy array; B @ gpu likewise

cuSPARSE reads only CSR/CSC, so walking the VCS layout on the device needs custom kernels; vsparse._cuda supplies four, one per (format, direction). Against the alternative — expanding to a sorted CuPy CSR and calling cuSPARSE — they measured (RTX 5080, 60k x 2k, 6M nonzeros, width 16):

direction	kernel / cuSPARSE	device bytes vs CSR
VCSR @ B	0.71x–0.86x	0.62x
VCSC @ B	0.83x	0.51x
B @ VCSC	0.32x	0.51x
B @ VCSR	0.23x	0.62x

CudaNormalizedView.to_cupy_sparse() builds that CSR baseline if you want it.

See docs/ for full usage guides and API documentation.

Development

uv sync --all-extras --dev     # add --no-extra cuda to skip the ~1 GB CuPy wheel
uv run pytest
uv run ruff check .
uv run ty check
uv run sphinx-build -b html docs docs/_build/html

The CUDA tests and benchmarks need an NVIDIA GPU; without one tests/test_cuda.py skips itself. On a GPU machine:

uv run pytest -m cuda
uv run python -m benchmarks.run --set cuda