llmopt

A mathematics and physics ML lab where nothing counts until an oracle agrees. Answers are checked by symbolic equivalence, not string match. Decoding is checked to be token-identical to eager greedy. Generated assembly is assembled and run. Weights are scored by what they compute, never by their distance to other weights. Every experiment is pre-registered with a threshold it can fail, and the failures are published beside the wins.

That is every gate neuron in a 19M-parameter model born on this lab's math corpus — 12,288 rows of weight matrices, drawn three ways: global PCA axes, directions alone on the unit sphere, and phase against magnitude. Color is each neuron's magnitude. Nothing in that image was designed. It is the model, drawn — checkpoint hash and repo commit are stamped in the footer, so the pixels trace to exact artifacts.

---

Four results

Each result names its own scope. In scores, a gate is the fixed problem set a model is scored on, so gating N means solving N of its problems; paired means the same seed and device on both arms of a comparison, and a claim says which arms are paired; a model is born when it is trained from a fresh initialization. The rest of the vocabulary is in the glossary.

Which experts you keep is the difference between 0 and 81 of 120.

The score is the number solved out of 120 generated math problems, each answer checked by symbolic equivalence, never by string match. Masking Qwen3-30B-A3B-4bit (the 30B mixture-of-experts model behind every MoE result here, from mlx-community) to the top 45.3% of its per-layer math-demand experts, ranked by how often its router selected them on math prompts, beat the paired full model at all six paired seeds: 80, 82, 81 against 63, 73, 63 at the three registered ones, pooled +14.7 against a +7 bar declared before the run. At the identical keep fraction, random and anti-demand masks score nothing at all. The effect is selection, not sparsity, and why it happens is still unexplained. Scope: one model, one keep rule, mathgen L1–3, Mac MLX; the zero-scoring controls ran at their own seed and are not paired to those arms.

Averaging independently born weights does not degrade a model. It ends it.

Six pairwise averages of independently born models gated exactly zero, at every level — not degraded, dead. Merges inside a shared-initialization lineage land in the parent band instead, and that holds at three paired seeds even when the two models share no optimizer step and no data order. The basin is chosen at initialization; everything after is basin-local.

The training floor walks down with width — and stops nowhere near the corpus limit.

The floor descends monotonically across an eight-fold width ladder and never approaches the corpus entropy at k=32 — the whole ladder buys about one token of effective context. Swapping in the opposite inductive bias, a selective state-space model, did not cross the wall either against its own paired twin on a different machine, though that arm gated only 2 of 120 and so is a weak control. The wall appears to belong to the diet.

Effective context is architecture-bound, and width does not fix it.

A truncation probe on the floor ladder's own checkpoints found something the training-loss average could not see: at deep positions every width improves by about a nat as context grows from 16 to 128 tokens, so the long-range dependency is real. Its second registered bar — that wider models separate more as context grows — did not fire: the gap is negative at k=8 and narrows again at k=128.

The record argues with itself

The 434 curated claims in FINDINGS by maturity: 47 replicated, 102 mechanism-confirmed, 223 single-seed, 57 null, 5 retracted. 62 of 434 (14.3%) are null or retracted; 223 of 434 (51.4%) are single-seed.

Nulls and retractions sit beside the wins at the same prominence, because a ledger that only records successes cannot be checked. Each claim carries exactly one maturity tag and its scope fences — device, seed count, format, regime — and those tags are part of the claim, not optional reading. The counts above are recounted from the source every time the figure is built. How a claim is made and policed, from pre-registration to amendment, is in METHODS.

Start with the curated findings, organized by evidence maturity rather than chronology. The glossary defines the vocabulary; RESULTS is the living append-only ledger every claim resolves to; REPRODUCE is the walkthrough.

What is built

The code and tests behind each system are indexed in ENGINEERING.

pip install -e ".[dev]"          # core: torch, numpy, sympy
pip install -e ".[figures,lake]" # optional: plotting, Parquet result lake

Research instruments. search/ — symbolic derivation search with explicit rewrite rules, learned evaluators, transposition memory, and a verified ZX path for circuit reduction. mathgen/ — seeded generators for calculus, linear algebra, ODEs, mechanics, and proofs, with symbolic checks built into generation. moe/ — routing anatomy: demand ranking, keep-sets, router masking, expert surgery. weightspace/ — predicting what a network computes from its parameters. quantum/ — model-Hamiltonian ground-state instruments. lab/ — the adopted instrument layer: the standard gate, the process-isolated oracle, the checkpoint catalog, merge operations. runs/ — run receipts, completion markers, trajectory instruments, the Parquet result lake. figures/ — the validated palette, the published SVG/PNG renderer backed by docs/figures.json, and the matplotlib analysis renderer for working plots.

Training and numerics. train/ — closed-system births, controlled diets, LoRA, preference objectives. common/ — device selection, string-seeded RNG, checkpoint IO. intmath — exact integer primitives, the arithmetic behind bit-identical cross-machine replay. quantize/ — sensitivity probes, closed-form bit allocation, packed integer artifacts.

Inference and systems. decoding/ — speculative and prompt-lookup decoding, sampler pipelines, constrained decoding, tree verification. cache/ — radix prefix tree, paged blocks, KV quantization, eviction. kernels/ — hand-written Metal and Triton kernels with the benchmarks they lost. codegen/ — assemble the prediction, run the program.

Local chat with the compressed Qwen3.8-27B

The qualified arm C (9.37 GB resident, generation-5 Metal kernels) serves on a 36 GB Apple Silicon Mac through scripts/qwen_chat.py. The default since VERDICT QWEN-LOW-DEFAULT-1 is thinking at the template's low reasoning effort with a 768-token budget, incremental cache reuse under an exact rendered-prefix check with full-transcript replay as the fallback, greedy decoding. The fast mode answers directly without a reasoning block. Since VERDICT QWEN-GDN-PROMOTE-0 the 48 GatedDeltaNet layers run mlx-lm's Metal recurrence kernel by default, selected per layer by name with the vendor's training flag untouched and refused under any other mlx-lm source than the qualified 0.31.3 pair; --recurrence loop selects the Python reference loop the earlier verdicts were measured on. The two paths were token-identical on every measured prompt (36 held-out comparisons, 60 v3 battery tasks, 16 fresh CLI tasks, 9 chat turns).

.venv/bin/python scripts/qwen_chat.py --interactive                 # ordinary: thinking, low effort, 768 tokens
.venv/bin/python scripts/qwen_chat.py --interactive --no-thinking   # fast: direct answers
.venv/bin/python scripts/qwen_chat.py --prompt "What is 413 * 685?" --reasoning-effort xhigh --max-tokens 1024
.venv/bin/python scripts/qwen_chat.py --interactive --replay        # full-transcript replay every turn (the reference)
.venv/bin/python scripts/qwen_chat.py --interactive --recurrence loop   # Python reference recurrence (the pre-promotion path)

What the default buys and costs, measured on purpose-built batteries (RESULTS: VERDICT QWEN-EFFORT-0, VERDICT QWEN-LOW-RELEASE-0): on the 60-task v3 confirmation battery the default configuration passed 56 of 60 against 45 for direct answers at the same 768-token budget; these are measured pass counts on a small battery whose families were chosen by the lab, not a general accuracy figure. The cost in that battery: 12,345 generated tokens and 1,470 s against 2,925 tokens and 485 s for direct answers; in four eight-turn conversations 5,202 tokens and 642 s against 533 tokens and 123 s (logs/qwen_serve/lowchat_v1.json). Three of 68 battery turns never closed their reasoning block and produced no final answer. First-token latency on the kernel default (VERDICT QWEN-GDN-PROMOTE-0, medians): 2.2 s on the 16 short CLI prompts, 8.9 s at 512 prompt tokens, 18.1 s at 1,024 and 34.9 s at 2,048, against 2.7, 13.3, 26.9 and 52.3 s on the reference loop; decode steps 94 to 96 ms against 100 to 102 ms. The prefill runs at about 57 tokens per second at every measured length; by the VERDICT QWEN-PREFILL-WALL-0 decomposition the W4 MLP projections are its largest remaining term.

Known limitations. Direct-answer mode (no thinking) collapses in longer conversations: in two studies it produced three consecutive empty replies on turns 5 to 7 of an eight-turn conversation at both 96 and 768-token budgets; the stop was the model's own decision (margins of 2 to 10 logits), not a serving defect. Thinking mode did not show this in 32 measured turns but can run a reasoning block to the budget without answering. Multi-operand arithmetic without thinking is unreliable. All measurements are greedy; the vendor recommends sampling for ordinary generation, which this program does not implement. Incremental reuse and full replay were token-identical on every measured turn, with the two paths not bit-equivalent by construction; a re-tokenisation of a stored reply can force a replay, which the session reports and which costs the prefill of the whole transcript.

Reproduce

RJOB_LOCAL=1 python -m llmopt.reproduce gravmoe-rb1

PASS means the final training-trajectory digest exactly matches the committed pin. A 1000-step integer birth replays bit-identically on a second machine, and a 200-step birth is trajectory-identical across Mac CPU, an RTX 3080, and a C++ implementation developed independently of this code, under the same operator.

Trajectory agreement is not oracle correctness: it certifies the pinned weight path and teacher-forced readouts. Free-run symbolic scoring additionally needs diet row text that is not committed, so artifact-backed arms run in an explicit trajectory-only mode. python -m llmopt.reproduce --list shows the registry.

Small text receipts, pinned checkpoints and job cards that booked verdicts cite are committed under logs/, checkpoints/ and jobs/, so a verdict's evidence travels with the repository; large artifacts stay untracked and the receipt lock records which is which.

What remains uncertain

The crest has no mechanism. Why masking the 30B MoE to its demand coalition beats full width on mathematics is unexplained. The two quantities a keep rule optimizes — coverage and recall of demanded experts — were measured not to predict even the sign of the effect.

The best current candidate is interference removal, reachable either by the demand mask or by deleting a named 80-expert carrier population, 1.3% of the bank. Both forms replicated at three fresh paired seeds, with the router measured over-inclusive at the carriers' rank class. A control run in the same session complicated it: a matched-size random fill resurrected a dead core about as well as the verbal-branch fill, but that random pool was itself ~45% verbal-branch experts. Fills that exclude the verbal branch score 0 and 7 of 120 against 16 to 55 for fills that include it. So the verbal population is necessary and recall does not organize it; what is sufficient is unmeasured.

The calibration-free packing law has a measured boundary. It holds on at-capacity house crystals and does not transport to Qwen2.5-0.5B, where max-anchored and calibrated grids exploit weight-tail structure the house crystals lack. Both sides of that boundary are n=1.

Many comparisons remain single-seed and device-scoped. The maturity tags say which. Cross-device gate comparisons are forbidden outright, and the figures above carry their own device and seed count.

Reproduction stops short of self-contained oracle scoring, because the row text cannot currently be shared. The public artifact proves the trajectory and the teacher-forced readouts, and says so.

Citing

Name the exact commit SHA and the exact verdict entry in docs/RESULTS.md that supports the claim. The ledger is living, so an unpinned citation is not reproducible. Repository metadata is in CITATION.cff.

Where to go next

• Findings: every curated claim with its maturity tag and scope.
• Engineering: what is built, with the code and tests to read first.
• Reproduce: the one-command trajectory replay and what it certifies.
• The docs map: which documents are curated, which are the record, which are generated views, and which are the lab's own working notes.

Day-to-day work is carried out by coding agents under mechanical guards (pre-registration adjudication, receipt locks, generated-document checks; see CLAUDE.md and .claude/); every claim passes those guards, and one human operator runs the lab. Charter: mathematics and physics only.

Licensed under Apache-2.0.