[yugt]

Follow-up to #24, answering the review there: the four cards calibrated, two larger Qwen3.6-35B-A3B files (Q8_0 and full-precision BF16), and DeepSeek-V4.1-Flash-1017.

Docs only. Runs at e1c2d6e (the DeepSeek end-to-end timing at this branch, same code), one GPU per Slurm job.

Qwen3.6-35B-A3B, before and after calibrate (each job: bench, calibrate, bench, bench --mtp 0, for CUDA and Vulkan):

Decode after 512, CUDA, tok/s	A100 80 GB	A30	L40S	H100
UD-Q2_K_XL, uncalibrated → calibrated	178 → 295	138 → 203	221 → 424	272 → 524 (80 GB HBM3)
Q8_0	200 → 305	126 → 209	169 → 276	298 → 532 (80 GB HBM3)
BF16	166 → 248	13 → 14	60 → 84	237 → 330 (NVL)

• Nearly all of the decode gain is MTP: calibrated with --mtp 0 decodes at the uncalibrated speed wherever the experts fit in VRAM. MTP costs 4-13% of pp 4096 prefill (23% for Q2_K_XL Vulkan on the A30).
• calibrate picked chunk 4096 everywhere except Q2_K_XL Vulkan on the A30 (2048), MTP 1-2, and MTP 0 for BF16 on the A30 and L40S, where the experts do not fit (3133 and 6657 of 10240 resident).
• Where experts do not fit, the uncalibrated run is a cold start (page cache, no routing profile yet), so part of that before/after step is not calibration. Uncalibrated pp 512 also includes one-time kernel/pipeline compilation. The doc says which rows this affects.

DeepSeek-V4.1-Flash-1017 GSQ-RCO IQ2_XXS (248.8 GB): check passes, "The capital of France is" → " Paris." on both cards, and the same greedy text on every run. With the experts in the page cache, end-to-end decode is 4.5-4.6 tok/s at the default 9 GB VRAM expert budget on an A100 80 GB and an H100 NVL, and 6.6 (A100) / 8.9 (H100 NVL) tok/s at 60 GB. Before that, reading experts from the cluster file system takes seconds to a minute per token. This is one 30-token prompt and 33 decode tokens (the model stopped there), so nothing about long contexts.

Full tables (mean, min-max of 3 repetitions), host CPUs, file hashes and commands are in docs/datacenter-nvidia-results.md. SUPPORTED_GPUS.md gets the calibrated decode per row next to the same job's --mtp 0 number; BENCHMARKS.md gets a pointer to the DeepSeek numbers.

🤖 Generated with Claude Code

[Vasyl-Trefilov]

@yugt What for you want to use RustingStrata? I mean if you think about running some bigger models, I can add them and optimize to make them way faster