From 6e8bcc954b3b50dea76972703a722b4c4f4eaf96 Mon Sep 17 00:00:00 2001 From: Guangting Yu Date: Tue, 10 Nov 2026 14:17:48 -0700 Subject: [PATCH 1/2] tests: default-model checks recognise the model by file name is_default_model compared the whole path with DEFAULT_MODEL, so on any machine where the reference GGUF is not at that path the default-model checks were skipped (the 0.05 tolerance of gpu_forward_matches_cpu_reference, the type coverage counts in tests/kernels.rs, the shapes in tests/quant.rs and tests/gguf.rs). A copy named Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf under STRATA_MODEL now counts as the default model. Co-Authored-By: Claude Fable 5.2 --- tests/common/mod.rs | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/common/mod.rs b/tests/common/mod.rs index 37f4911..2835557 100644 --- a/tests/common/mod.rs +++ b/tests/common/mod.rs @@ -8,9 +8,10 @@ pub fn model_path() -> Option { if p.exists() { Some(p) } else { eprintln!("SKIP: model file missing"); None } } -/// True for the default model; tests of its exact shapes and type mix skip those checks on any other. +/// True for the default model, wherever it is stored (`STRATA_MODEL` pointing at a copy with the same file name +/// counts); tests of its exact shapes and type mix skip those checks on any other. pub fn is_default_model(p: &std::path::Path) -> bool { - let d = p == std::path::Path::new(DEFAULT_MODEL); + let d = p.file_name().is_some() && p.file_name() == std::path::Path::new(DEFAULT_MODEL).file_name(); if !d { eprintln!("SKIP: default-model checks on {}", p.display()); } d } From 3747ee8255f9ebb9498a338a7ea16caa731894b8 Mon Sep 17 00:00:00 2001 From: Guangting Yu Date: Tue, 10 Nov 2026 14:17:50 -0700 Subject: [PATCH 2/2] docs: A100, A30, L40S and H100 test runs and benchmarks Full test suite on CUDA and Vulkan (185 passed, 0 failed on each card), bench numbers, the Vulkan shared-memory preflight, and a note on a driver hang in vkDestroyDevice that parallel tests can hit on NVIDIA 615.71.09. Co-Authored-By: Claude Fable 5.2 --- docs/README.md | 1 + docs/SUPPORTED_GPUS.md | 8 +++ docs/datacenter-nvidia-results.md | 95 +++++++++++++++++++++++++++++++ 3 files changed, 104 insertions(+) create mode 100644 docs/datacenter-nvidia-results.md diff --git a/docs/README.md b/docs/README.md index fea63d9..4f66ddf 100644 --- a/docs/README.md +++ b/docs/README.md @@ -19,3 +19,4 @@ Raw results and methods, newest work last. The headline numbers are in the [READ - [m4-results.md](m4-results.md): chunked prefill. - [m6-results.md](m6-results.md): v1.0 gate run against llama.cpp. - [V100-results.md](V100-results.md): V100 bring-up, parity and benchmarks (long; written by `tools/bench_v100.sh`). +- [datacenter-nvidia-results.md](datacenter-nvidia-results.md): A100, A30, L40S and H100 test runs and benchmarks. diff --git a/docs/SUPPORTED_GPUS.md b/docs/SUPPORTED_GPUS.md index 6c9d9a0..85ab53a 100644 --- a/docs/SUPPORTED_GPUS.md +++ b/docs/SUPPORTED_GPUS.md @@ -16,6 +16,14 @@ row. | AMD Radeon VII (Vega 20) | GCN5, wave64 | Vulkan | Runs | masanork (PR #5) | RADV Mesa 26.0.8. Each 64-lane subgroup runs as two logical 32-lane warps. See [AMD.md](AMD.md). | | AMD Radeon RX 9070 XT | RDNA 4 | Vulkan | Runs, known issues | user reports | Linux (RADV) and Windows (AMD driver). See [AMD.md](AMD.md) for the issues and what to send. | | NVIDIA Turing card | Turing, sm_75 | CUDA | Runs | x-Pepsy (PR #6) | `m16n8k8` tensor cores, no int8 GEMM, 64 KB shared memory (16-key prefill tile at head_dim 256). The GPU model was not recorded in the PR. | +| NVIDIA A100-SXM4 80 GB | Ampere, sm_80 | CUDA | Full test suite passes | yugt | pp 4096 prefill 6203 tok/s, decode 174 tok/s (MTP off, whole model in VRAM). See [datacenter-nvidia-results.md](datacenter-nvidia-results.md). | +| NVIDIA A100-SXM4 80 GB | Ampere, sm_80 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 5589 tok/s, decode 132 tok/s. | +| NVIDIA A30 24 GB | Ampere, sm_80 | CUDA | Full test suite passes | yugt | pp 4096 prefill 3544 tok/s, decode 132 tok/s. | +| NVIDIA A30 24 GB | Ampere, sm_80 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 3326 tok/s, decode 104 tok/s. | +| NVIDIA L40S 48 GB | Ada, sm_89 | CUDA | Full test suite passes | yugt | pp 4096 prefill 9030 tok/s, decode 213 tok/s. | +| NVIDIA L40S 48 GB | Ada, sm_89 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 8571 tok/s, decode 185 tok/s. | +| NVIDIA H100 NVL 94 GB, H100 80 GB | Hopper, sm_90 | CUDA | Full test suite passes | yugt | pp 4096 prefill 9131 tok/s, decode 246 tok/s (H100 NVL; the 80 GB card ran the tests only). | +| NVIDIA H100 NVL 94 GB, H100 80 GB | Hopper, sm_90 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 6428 tok/s, decode 192 tok/s (H100 NVL). | ## Compiles, not run diff --git a/docs/datacenter-nvidia-results.md b/docs/datacenter-nvidia-results.md new file mode 100644 index 0000000..6a2597f --- /dev/null +++ b/docs/datacenter-nvidia-results.md @@ -0,0 +1,95 @@ +# A100, A30, L40S and H100 results + +Four data-center NVIDIA cards on one Slurm cluster, one GPU per job (8 CPU cores, 48 GB RAM). Rocky Linux 8.10, +driver 615.71.09 (Vulkan 1.4.351), NVRTC 13.0 from the CUDA 13.0.1 toolkit, Rust 1.99.1, `target-cpu=native`. +Commit `7264e57` (v1.2.1). Model: `Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf` (sha256 `ed7cda7e...f37fb0c`); it fits in VRAM on +every card here, so all 10240 experts are resident and the expert hit rate is 100%. + +| GPU | VRAM | Arch | Host CPU | +| --- | --- | --- | --- | +| A100-SXM4-80GB | 81920 MiB | sm_80 | EPYC 7413 | +| A30 | 24576 MiB | sm_80 | EPYC 7413 | +| L40S | 46068 MiB | sm_89 | Xeon Platinum 8458P | +| H100 NVL | 95830 MiB | sm_90 | EPYC 9224 | +| H100 80GB HBM3 (tests only) | 81559 MiB | sm_90 | Xeon Platinum 8470 | + +## 2026-11-10 — benchmarks + +`rustingstrata bench --model $M --gpu --backend {cuda,vulkan} --pp 512,4096 --tg 128 --reps 3`, no `calibrate` run, MTP +off. Mean (min-max) of 3 repetitions, tok/s. + +| GPU | Backend | pp 512 prefill | decode after 512 | pp 4096 prefill | decode after 4096 | +| --- | --- | --- | --- | --- | --- | +| A100 80 GB | CUDA | 3852.3 (3767.5-3916.5) | 178.8 (178.0-179.2) | 6202.5 (6186.2-6213.7) | 173.6 (173.2-173.8) | +| A100 80 GB | Vulkan | 2353.8 (119.2-3473.5) | 120.1 (71.4-144.6) | 5588.8 (4788.0-5990.0) | 131.9 (116.5-139.7) | +| A30 | CUDA | 2261.4 (1924.2-2435.3) | 137.8 (137.3-138.0) | 3543.9 (3539.3-3548.2) | 131.9 (131.6-132.1) | +| A30 | Vulkan | 1516.8 (127.9-2239.9) | 99.7 (70.6-114.3) | 3326.0 (3061.1-3459.0) | 104.0 (94.4-108.8) | +| L40S | CUDA | 6416.6 (6198.4-6566.3) | 220.2 (219.3-220.8) | 9030.3 (9023.1-9039.5) | 213.2 (212.8-213.5) | +| L40S | Vulkan | 3887.7 (143.1-5766.5) | 166.0 (89.9-204.1) | 8570.6 (7243.4-9279.4) | 184.9 (162.3-196.3) | +| H100 NVL | CUDA | 5718.3 (5505.3-5868.2) | 253.1 (251.6-253.8) | 9130.8 (9115.2-9153.7) | 245.8 (245.1-246.1) | +| H100 NVL | Vulkan | 2967.8 (118.2-4396.9) | 174.0 (95.5-213.3) | 6428.2 (5569.7-6857.7) | 191.5 (164.5-205.1) | + +The first Vulkan repetition of `pp 512` includes pipeline compilation (the 118-143 tok/s minima), as on the V100. +The numbers are comparable within a row only: the four cards sit in hosts with different CPUs. + +Startup lines: + +``` +cuda device 0: NVIDIA A100-SXM4-80GB, sm_80 +cuda shared memory: opt-in 166912 B per block, gemvn_*: 0 B static, dynamic max 166912 B, carveout default +cuda device 0: NVIDIA A30, sm_80 +cuda shared memory: opt-in 166912 B per block, gemvn_*: 0 B static, dynamic max 166912 B, carveout default +cuda device 0: NVIDIA L40S, sm_89 +cuda shared memory: opt-in 101376 B per block, gemvn_*: 0 B static, dynamic max 101376 B, carveout default +cuda device 0: NVIDIA H100 NVL, sm_90 +cuda shared memory: opt-in 232448 B per block, gemvn_*: 0 B static, dynamic max 232448 B, carveout default + +vulkan: NVIDIA A100-SXM4-80GB (NVIDIA 615.71.09), cooperative matrix f16 yes int8 yes, conditional rendering yes, maxComputeSharedMemorySize 49152 +vulkan: NVIDIA A30 (NVIDIA 615.71.09), cooperative matrix f16 yes int8 yes, conditional rendering yes, maxComputeSharedMemorySize 49152 +vulkan: NVIDIA L40S (NVIDIA 615.71.09), cooperative matrix f16 yes int8 yes, conditional rendering yes, maxComputeSharedMemorySize 49152 +vulkan: NVIDIA H100 NVL (NVIDIA 615.71.09), cooperative matrix f16 yes int8 yes, conditional rendering yes, maxComputeSharedMemorySize 49152 +plan: weights+state+scratch 1984 MB (KV 82 MB for 4224 tokens, prefill 512 MB), expert slots 10002 MB, resident experts 10240/10240 +``` + +`rustingstrata check --backend vulkan --model $M` on each card: 96 (kernel, specialization) pairs checked, largest 49152 +bytes (`gemmi`) of 49152 allowed. + +## 2026-11-10 — test suite + +`STRATA_MODEL=$M cargo test --release --no-fail-fast -- --nocapture` on A100, A30, L40S and H100 80GB HBM3: 185 passed, +0 failed, 5 ignored on each. H100 NVL ran the same suite without `--nocapture` at `7264e57`: 185 passed, 0 failed, +5 ignored. A30 also passed with `--test-threads=1`. + +- Both backends were open in every run: each log has 48 `cuda device 0:` lines and 47 `vulkan:` device lines. The only + skips are `no glslc` (`spirv_is_fresh`) and `STRATA_TEST_BACKENDS does not select cuda`. +- The runs with `--nocapture` include the `tests/common` change that recognises the default model by file name, so the + default-model checks (the 0.05 logit tolerance of `gpu_forward_matches_cpu_reference`, the type-coverage counts in + `tests/kernels.rs`, the shapes in `tests/quant.rs` and `tests/gguf.rs`) ran and passed. Before it they were skipped + on any machine where the file is not at the path in `DEFAULT_MODEL`. + +## 2026-11-10 — parallel tests can hang in `vkDestroyDevice` on this driver + +One full `cargo test --release` on the A30 stopped in `tests/kernels.rs`: no output for five minutes, GPU utilisation +0%, one thread at 100% CPU. `gdb` on the test process: + +``` +Thread 6: pthread_join <- libnvidia-eglcore.so.615.71.09 <- vkDestroyDevice <- ::drop (src/vulkan/mod.rs:279) +Thread 15: libnvidia-eglcore.so.615.71.09 <- __nptl_deallocate_tsd <- start_thread (the thread at 100% CPU) +Thread 41: pthread_mutex_lock <- vkCreateInstance <- Vk::new (src/vulkan/mod.rs:333) <- f16_weight_uploads_as_f32 +Thread 40: pthread_mutex_lock <- vkCreateInstance <- Vk::new <- elementwise_ops_match_cpu +4 threads: __pthread_mutex_lock_full <- Inner::submit / Inner::buffer (other tests' devices) +``` + +A test thread destroying its device waits for a driver thread that spins in its thread-local destructor and never +exits; the tests that start next wait on a loader mutex in `vkCreateInstance`. + +How often, with the `kernels` test binary run 15 times per card and killed after 240 s (600 s for the serial runs): + +| Run | A30 | A100 | +| --- | --- | --- | +| parallel (default test threads) | 2 of 15 hung, the others took 34-40 s | 3 of 15 hung, the others took 31-41 s | +| `--test-threads=1` | 0 of 15 hung (51-85 s) | 0 of 15 hung (53-94 s) | + +A hung run stopped after a different number of tests each time (0 to 12 finished). L40S and H100 were not looped. The +results are not affected: every run that finished passed. Until this is solved, `cargo test --release -- +--test-threads=1` is the way to run the suite on this driver: it did not hang in these 30 runs.