diff --git a/README.md b/README.md
index 8309db1..6e4798f 100644
--- a/README.md
+++ b/README.md
@@ -58,9 +58,10 @@ IQ3_XXS, IQ3_S, IQ4_XS, IQ4_NL.
 Needs Rust (2024 edition) and one of:
 
 - an NVIDIA driver with the CUDA NVRTC library (loaded at run time). Volta (sm_70), Turing (sm_75) and Ampere or newer
-  (sm_80+) work; CUDA 13 dropped Volta, so a V100 needs the 12.x toolkit. Pascal and older use Vulkan.
-- a Vulkan 1.3 driver with shaderFloat64, int8/16/64 and subgroup size 32 or a fixed 64 (tested on RTX 3060, Radeon VII
-  and V100; Intel Arc lacks shaderFloat64).
+  (sm_80+) work; CUDA 13 dropped Volta, so a V100 needs the 12.x toolkit. Pascal and older are not supported (a GTX
+  1080 Ti runs on neither backend, see [docs/SUPPORTED_GPUS.md](docs/SUPPORTED_GPUS.md)).
+- a Vulkan 1.3 driver with shaderFloat64, shaderFloat16, int8/16/64 and subgroup size 32 or a fixed 64 (tested on RTX
+  3060, Radeon VII and V100; Intel Arc lacks shaderFloat64).
 
 `--gpu` picks CUDA when it loads, else Vulkan; `--backend` chooses. The first CUDA run compiles the kernels (~15 s) and
 caches them. Every GPU tried so far is in [docs/SUPPORTED_GPUS.md](docs/SUPPORTED_GPUS.md); AMD notes are in
diff --git a/docs/README.md b/docs/README.md
index 4f66ddf..6b6971c 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -20,3 +20,5 @@ Raw results and methods, newest work last. The headline numbers are in the [READ
 - [m6-results.md](m6-results.md): v1.0 gate run against llama.cpp.
 - [V100-results.md](V100-results.md): V100 bring-up, parity and benchmarks (long; written by `tools/bench_v100.sh`).
 - [datacenter-nvidia-results.md](datacenter-nvidia-results.md): A100, A30, L40S and H100 test runs and benchmarks.
+- [more-nvidia-results.md](more-nvidia-results.md): RTX 2080 Ti, Quadro RTX 6000, V100, A40, RTX A6000, RTX 4500 Ada
+  and A100 40 GB test runs and benchmarks; the GTX 1080 Ti does not run.
diff --git a/docs/SUPPORTED_GPUS.md b/docs/SUPPORTED_GPUS.md
index f9b5b3a..0d05510 100644
--- a/docs/SUPPORTED_GPUS.md
+++ b/docs/SUPPORTED_GPUS.md
@@ -24,6 +24,20 @@ row.
 | NVIDIA L40S 48 GB | Ada, sm_89 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 8571 tok/s, decode 185 tok/s. Calibrated in a later job: decode 317 tok/s with MTP 2, 197 with `--mtp 0`. |
 | NVIDIA H100 NVL 94 GB, H100 80 GB | Hopper, sm_90 | CUDA | Full test suite passes | yugt | pp 4096 prefill 9131 tok/s, decode 246 tok/s (H100 NVL). Calibrated in a later job on an H100 80 GB: decode 498 tok/s with MTP 2, 265 with `--mtp 0`. |
 | NVIDIA H100 NVL 94 GB, H100 80 GB | Hopper, sm_90 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 6428 tok/s, decode 192 tok/s (H100 NVL). Calibrated in a later job on an H100 80 GB: decode 363 tok/s with MTP 2, 216 with `--mtp 0`. |
+| NVIDIA GeForce RTX 2080 Ti 11 GB | Turing, sm_75 | CUDA | Full test suite passes | yugt | 8213-8334 of 10240 experts resident (10496 with MTP). pp 4096 prefill 2714 tok/s, decode 140 tok/s with `--mtp 0`; calibrated decode 197 tok/s with MTP 1. See [more-nvidia-results.md](more-nvidia-results.md). |
+| NVIDIA GeForce RTX 2080 Ti 11 GB | Turing, sm_75 | Vulkan | Runs, test suite passes with `STRATA_VK_F16_TRUNC=1` | yugt | Cooperative matrices (f16 and int8). The f16 conversion truncates as on the V100: the variable only tells the kernel tests to expect that, and 4 of them fail without it. pp 4096 prefill 2630 tok/s, decode 126 tok/s with `--mtp 0`; calibrated decode 158 tok/s with MTP 1. |
+| NVIDIA Quadro RTX 6000 24 GB | Turing, sm_75 | CUDA | Full test suite passes | yugt | pp 4096 prefill 2795 tok/s, decode 122 tok/s with `--mtp 0`; calibrated decode 179 tok/s with MTP 1. |
+| NVIDIA Quadro RTX 6000 24 GB | Turing, sm_75 | Vulkan | Runs, test suite passes with `STRATA_VK_F16_TRUNC=1` | yugt | Cooperative matrices (f16 and int8). The f16 conversion truncates as on the V100: the variable only tells the kernel tests to expect that, and 4 of them fail without it. pp 4096 prefill 2719 tok/s, decode 109 tok/s with `--mtp 0`; calibrated decode 143 tok/s with MTP 1. |
+| NVIDIA Tesla V100-SXM2 16 GB and 32 GB | Volta, sm_70 | CUDA | Full test suite passes | yugt | NVRTC 12.6, driver 580.95.05. 16 GB card: pp 4096 prefill 3145 tok/s, decode 128 tok/s with `--mtp 0`; calibrated decode 193 tok/s with MTP 1. See [more-nvidia-results.md](more-nvidia-results.md). |
+| NVIDIA Tesla V100-SXM2 16 GB and 32 GB | Volta, sm_70 | Vulkan | Runs, test suite passes with `STRATA_VK_F16_TRUNC=1` | yugt | No cooperative matrices. The variable only tells the kernel tests to expect the truncating f16 conversion; 4 of them fail without it. 16 GB card: pp 4096 prefill 1676 tok/s, decode 120 tok/s with `--mtp 0`; calibrated decode 170 tok/s with MTP 2. |
+| NVIDIA A100-PCIE 40 GB | Ampere, sm_80 | CUDA | Full test suite passes | yugt | pp 4096 prefill 5877 tok/s, decode 169 tok/s with `--mtp 0`; calibrated decode 275 tok/s with MTP 1. See [more-nvidia-results.md](more-nvidia-results.md). |
+| NVIDIA A100-PCIE 40 GB | Ampere, sm_80 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 5639 tok/s, decode 136 tok/s with `--mtp 0`; calibrated decode 223 tok/s with MTP 2. |
+| NVIDIA RTX A6000 48 GB | Ampere, sm_86 | CUDA | Full test suite passes | yugt | pp 4096 prefill 5094 tok/s, decode 188 tok/s with `--mtp 0`; calibrated decode 329 tok/s with MTP 2. |
+| NVIDIA RTX A6000 48 GB | Ampere, sm_86 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 5052 tok/s, decode 165 tok/s with `--mtp 0`; calibrated decode 252 tok/s with MTP 2. |
+| NVIDIA A40 48 GB | Ampere, sm_86 | CUDA | Full test suite passes | yugt | pp 4096 prefill 4805 tok/s, decode 159 tok/s with `--mtp 0`; calibrated decode 289 tok/s with MTP 2. |
+| NVIDIA A40 48 GB | Ampere, sm_86 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 4796 tok/s, decode 141 tok/s with `--mtp 0`; calibrated decode 222 tok/s with MTP 2. |
+| NVIDIA RTX 4500 Ada 24 GB | Ada, sm_89 | CUDA | Full test suite passes | yugt | pp 4096 prefill 4934 tok/s, decode 153 tok/s with `--mtp 0`; calibrated decode 273 tok/s with MTP 2. |
+| NVIDIA RTX 4500 Ada 24 GB | Ada, sm_89 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 4979 tok/s, decode 142 tok/s with `--mtp 0`; calibrated decode 221 tok/s with MTP 2. |
 
 ## Compiles, not run
 
@@ -37,7 +51,7 @@ row.
 | GPU | Why |
 | --- | --- |
 | Intel Arc | Vulkan driver has no `shaderFloat64`. |
-| NVIDIA Pascal and older (CUDA) | No tensor cores. Use `--backend vulkan`; not tested. |
+| NVIDIA Pascal and older | CUDA: no tensor cores; on a GTX 1080 Ti (sm_61) the kernels do not compile (`__nanosleep` is undefined). Vulkan: the GTX 1080 Ti on driver 580.95.05 has no `shaderFloat16`, so no device is usable. See [more-nvidia-results.md](more-nvidia-results.md). |
 | GPUs without Vulkan 1.3 and subgroup size 32 or 64 | The Vulkan backend needs both. |
 
 ## Architecture tiers (CUDA)
diff --git a/docs/more-nvidia-results.md b/docs/more-nvidia-results.md
new file mode 100644
index 0000000..c46631a
--- /dev/null
+++ b/docs/more-nvidia-results.md
@@ -0,0 +1,168 @@
+# RTX 2080 Ti, Quadro RTX 6000, V100, A40, RTX A6000, RTX 4500 Ada, A100 40 GB and GTX 1080 Ti results
+
+A second Slurm cluster (not the one in [datacenter-nvidia-results.md](datacenter-nvidia-results.md)), one GPU per job
+with 8 CPU cores and 48 GB RAM. Rocky Linux 8.10, Rust 1.99.1, `target-cpu=native`, commit `f47e370`. NVRTC 13.0 from
+the CUDA 13.0.1 toolkit, except 12.6 from CUDA 12.6.1 on the V100s and in the GTX 1080 Ti CUDA attempt. Model:
+`Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf` (sha256 `ed7cda7e...f37fb0c`). Numbers are comparable within one job only: the cards
+sit in hosts with different CPUs.
+
+| GPU | VRAM | Arch | Driver | Host CPU (bench job) |
+| --- | --- | --- | --- | --- |
+| A100-PCIE-40GB | 40960 MiB | sm_80 | 615.71.09 | Xeon Gold 6330 |
+| RTX A6000 | 49140 MiB | sm_86 | 615.71.09 | EPYC 7413 |
+| A40 | 46068 MiB | sm_86 | 615.71.09 | Xeon Gold 6330 |
+| RTX 4500 Ada Generation | 24570 MiB | sm_89 | 615.71.09 | Xeon Gold 6246 |
+| Quadro RTX 6000 | 23040 MiB | sm_75 | 615.71.09 | Xeon Gold 6230R |
+| GeForce RTX 2080 Ti | 11264 MiB | sm_75 | 615.71.09 | EPYC 7402P |
+| Tesla V100-SXM2-16GB | 16384 MiB | sm_70 | 580.95.05 | Xeon Silver 4114 |
+| Tesla V100-SXM2-32GB | 32768 MiB | sm_70 | 580.95.05 | Xeon Gold 6148 |
+| GeForce GTX 1080 Ti | 11264 MiB | sm_61 | 580.95.05 | Xeon Gold 5120 |
+
+## 2026-11-11 — what does not pass
+
+- **GTX 1080 Ti (Pascal) runs on neither backend.** `--backend vulkan`: `no usable Vulkan device 0 (0 usable; NVIDIA
+  GeForce GTX 1080 Ti: missing shaderFloat16)` for `bench`, `calibrate` and `check`. `--backend cuda` with NVRTC 12.6:
+  the startup line is `cuda device 0: NVIDIA GeForce GTX 1080 Ti, sm_61 (Volta, m8n8k4)`, then `NVRTC compile of
+  kernels.cu failed` with `default_program(2321): error: identifier "__nanosleep" is undefined` (the only error; the
+  rest are warnings). `cargo test` reports 185 passed there, but that job had no CUDA toolkit loaded (`SKIP: cuda:
+  CUDA init: the NVRTC library (libnvrtc.so) is not installed`) and no usable Vulkan device, so the GPU tests
+  skipped; it is not a pass for the card. The suite was not run on this card with NVRTC present.
+- **Four Vulkan kernel tests fail on Turing and Volta unless `STRATA_VK_F16_TRUNC=1` is set.** On the RTX 2080 Ti, the
+  Quadro RTX 6000 and both V100s, `cargo test` is 181 passed, 4 failed, 5 ignored; with `STRATA_VK_F16_TRUNC=1` it is
+  185 passed, 0 failed on all four. The variable changes nothing in the engine: it tells the kernel tests to expect
+  an f16 conversion that truncates (`tests/kernels.rs`). This was known for the V100; the two Turing cards have
+  cooperative matrices (f16 and int8) and driver 615.71.09 and need it too. The Ampere and Ada cards on the same
+  driver pass without it. The variable was passed to those four jobs with `sbatch --export=ALL`; the logs do not
+  echo it. The runs with and without it were on different nodes of the same card type, except for the Quadro RTX
+  6000 (same node). The failures, with the same values on both Turing cards and (in parentheses) on both V100s:
+
+  | Test | Message |
+  | --- | --- |
+  | `f16_weight_uploads_as_f32` | `vulkan gemm_dense rel err 0.0000665` (0.0000654) |
+  | `gemm_matches_reference_for_every_type` | `IQ2_XS gemm (blk.0.ffn_gate_exps.weight): rel err 0.000512` (0.000511) |
+  | `gemv_partial_superblocks_and_q2_0` | `vulkan Q8_0 cols 320: gemm rel err 0.000534` (0.000533) |
+  | `vulkan_type_matrix` | `type matrix failures`, 4 |
+
+## 2026-11-11 — test suite
+
+`STRATA_MODEL=$M cargo test --release --no-fail-fast -- --test-threads=1 --nocapture`, one job per card (separate from
+the bench job, sometimes on another node of the same card type).
+
+| GPU | Result | With `STRATA_VK_F16_TRUNC=1` |
+| --- | --- | --- |
+| A100 40 GB | 185 passed, 0 failed, 5 ignored | not run |
+| RTX A6000 | 185 passed, 0 failed, 5 ignored | not run |
+| A40 | 185 passed, 0 failed, 5 ignored | not run |
+| RTX 4500 Ada | 185 passed, 0 failed, 5 ignored | not run |
+| Quadro RTX 6000 | 181 passed, 4 failed, 5 ignored | 185 passed, 0 failed, 5 ignored |
+| RTX 2080 Ti | 181 passed, 4 failed, 5 ignored | 185 passed, 0 failed, 5 ignored |
+| V100 16 GB | 181 passed, 4 failed, 5 ignored | 185 passed, 0 failed, 5 ignored |
+| V100 32 GB | 181 passed, 4 failed, 5 ignored | 185 passed, 0 failed, 5 ignored |
+
+Both backends were open in every run in this table: each log has 48 `cuda device 0:` and 47 `vulkan:` device lines.
+The skips are `no glslc` (`spirv_is_fresh`), `STRATA_TEST_BACKENDS does not select cuda`, the two DeepSeek tests
+(`STRATA_DS_MODEL` unset) and, with NVRTC 13.0, `nvrtc_compiles_full_source_for_sm70`.
+No run hung; all were serial (`--test-threads=1`), so this says nothing new about the parallel `vkDestroyDevice` hang.
+
+## 2026-11-11 — benchmarks before and after `calibrate`
+
+Each job builds, then runs on one card, with a new `XDG_CACHE_HOME` per job:
+
+```sh
+rustingstrata bench --model $M --gpu --backend cuda --pp 512,4096 --tg 128 --reps 3      # uncalibrated
+rustingstrata calibrate --model $M --backend cuda
+rustingstrata bench --model $M --gpu --backend cuda --pp 512,4096 --tg 128 --reps 3      # calibrated
+rustingstrata bench --model $M --gpu --backend cuda --pp 512,4096 --tg 128 --reps 3 --mtp 0
+# the same four steps with --backend vulkan, then
+rustingstrata check --model $M --backend vulkan
+```
+
+What `calibrate` chose (chunk / MTP drafts / short threshold):
+
+| Backend | A100 40 GB | RTX A6000 | A40 | RTX 4500 Ada | Quadro RTX 6000 | RTX 2080 Ti | V100 16 GB | V100 32 GB |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| CUDA | 4096 / 1 / 8 | 4096 / 2 / 8 | 4096 / 2 / 8 | 4096 / 2 / 8 | 4096 / 1 / 8 | 4096 / 1 / 8 | 4096 / 1 / 8 | 4096 / 1 / 8 |
+| Vulkan | 4096 / 2 / 8 | 4096 / 2 / 8 | 4096 / 2 / 8 | 4096 / 2 / 8 | 4096 / 1 / 8 | 2048 / 1 / 8 | 4096 / 2 / 32 | 4096 / 2 / 32 |
+
+Mean (min-max) of 3 repetitions, tok/s:
+
+| GPU | Backend | Run | MTP | Resident experts | pp 512 prefill | decode after 512 | pp 4096 prefill | decode after 4096 |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| A100 40 GB | CUDA | uncalibrated | 0 | 10240/10240 | 3306.1 (2120.6-3921.3) | 173.7 (171.2-175.1) | 5999.1 (5987.1-6010.4) | 168.9 (168.4-169.2) |
+| A100 40 GB | CUDA | calibrated | 1 | 10496/10496 | 3507.8 (3066.0-3741.0) | 288.5 (285.8-289.9) | 5509.5 (5503.8-5517.0) | 274.6 (273.4-275.4) |
+| A100 40 GB | CUDA | calibrated, `--mtp 0` | 0 | 10240/10240 | 3640.3 (3148.9-3907.5) | 174.4 (173.4-175.0) | 5877.3 (5870.5-5887.3) | 168.9 (168.6-169.1) |
+| A100 40 GB | Vulkan | uncalibrated | 0 | 10240/10240 | 2237.2 (118.6-3310.1) | 117.5 (72.6-139.9) | 5263.8 (4518.1-5640.4) | 127.1 (112.0-134.8) |
+| A100 40 GB | Vulkan | calibrated | 2 | 10496/10496 | 2739.2 (1984.9-3132.6) | 249.3 (242.5-254.1) | 5266.9 (5260.1-5278.8) | 222.8 (176.4-246.3) |
+| A100 40 GB | Vulkan | calibrated, `--mtp 0` | 0 | 10240/10240 | 2978.0 (2421.2-3270.4) | 140.8 (140.0-141.6) | 5639.1 (5601.2-5660.8) | 135.9 (135.9-136.0) |
+| RTX A6000 | CUDA | uncalibrated | 0 | 10240/10240 | 3459.3 (3071.2-3662.8) | 194.4 (193.5-195.0) | 5170.1 (5167.0-5172.5) | 188.5 (188.3-188.8) |
+| RTX A6000 | CUDA | calibrated | 2 | 10496/10496 | 3490.0 (3457.3-3522.8) | 346.2 (344.6-347.2) | 4813.7 (4800.0-4832.9) | 328.9 (325.6-330.7) |
+| RTX A6000 | CUDA | calibrated, `--mtp 0` | 0 | 10240/10240 | 3626.6 (3595.0-3647.2) | 193.6 (192.5-194.2) | 5093.6 (5082.3-5102.7) | 187.9 (187.2-188.4) |
+| RTX A6000 | Vulkan | uncalibrated | 0 | 10240/10240 | 2258.2 (133.1-3340.5) | 143.0 (87.5-170.8) | 4775.7 (4170.4-5086.4) | 152.8 (134.2-164.8) |
+| RTX A6000 | Vulkan | calibrated | 2 | 10496/10496 | 2983.3 (2782.1-3088.4) | 282.5 (266.9-291.2) | 4782.3 (4749.4-4815.8) | 251.8 (196.5-281.7) |
+| RTX A6000 | Vulkan | calibrated, `--mtp 0` | 0 | 10240/10240 | 3140.0 (2973.9-3296.0) | 163.7 (153.2-169.9) | 5051.9 (5027.7-5066.1) | 164.6 (163.2-166.2) |
+| A40 | CUDA | uncalibrated | 0 | 10240/10240 | 3399.6 (3263.6-3473.5) | 163.9 (162.8-164.5) | 4790.9 (4776.8-4799.6) | 159.3 (158.9-159.5) |
+| A40 | CUDA | calibrated | 2 | 10496/10496 | 3316.4 (3299.4-3341.2) | 303.6 (302.2-304.5) | 4579.1 (4560.0-4593.7) | 289.3 (286.3-291.1) |
+| A40 | CUDA | calibrated, `--mtp 0` | 0 | 10240/10240 | 3467.6 (3458.3-3486.0) | 164.1 (163.3-164.6) | 4805.2 (4802.8-4809.6) | 158.9 (157.7-159.6) |
+| A40 | Vulkan | uncalibrated | 0 | 10240/10240 | 2127.1 (117.4-3137.2) | 120.4 (69.9-145.7) | 4513.4 (3953.1-4795.7) | 132.6 (115.1-141.4) |
+| A40 | Vulkan | calibrated | 2 | 10496/10496 | 2924.5 (2758.5-3029.8) | 252.5 (245.0-256.3) | 4539.1 (4517.9-4551.1) | 222.1 (174.4-246.5) |
+| A40 | Vulkan | calibrated, `--mtp 0` | 0 | 10240/10240 | 3060.7 (2936.0-3146.0) | 145.2 (144.3-145.8) | 4795.5 (4779.3-4807.4) | 141.2 (141.1-141.4) |
+| RTX 4500 Ada | CUDA | uncalibrated | 0 | 10240/10240 | 3627.5 (3331.3-3785.7) | 159.6 (158.8-160.0) | 5056.2 (5052.3-5063.3) | 152.8 (152.5-153.0) |
+| RTX 4500 Ada | CUDA | calibrated | 2 | 10496/10496 | 3552.4 (3525.9-3598.8) | 284.6 (283.3-285.2) | 4631.0 (4625.1-4634.1) | 272.5 (270.0-273.9) |
+| RTX 4500 Ada | CUDA | calibrated, `--mtp 0` | 0 | 10240/10240 | 3704.7 (3684.5-3730.9) | 159.3 (158.6-159.7) | 4933.8 (4923.7-4940.2) | 152.6 (152.2-152.8) |
+| RTX 4500 Ada | Vulkan | uncalibrated | 0 | 10240/10240 | 2237.7 (121.6-3316.5) | 124.9 (74.8-150.3) | 4743.8 (4150.8-5045.6) | 134.7 (118.4-143.4) |
+| RTX 4500 Ada | Vulkan | calibrated | 2 | 10496/10496 | 2906.3 (2736.9-2998.3) | 243.5 (238.8-246.0) | 4644.8 (4599.1-4670.9) | 221.1 (173.5-245.0) |
+| RTX 4500 Ada | Vulkan | calibrated, `--mtp 0` | 0 | 10240/10240 | 3137.4 (3031.9-3201.8) | 147.1 (146.3-147.6) | 4979.4 (4916.3-5011.6) | 141.8 (140.6-142.8) |
+| Quadro RTX 6000 | CUDA | uncalibrated | 0 | 10240/10240 | 1924.8 (1727.6-2024.5) | 126.4 (125.9-126.7) | 2822.2 (2818.7-2824.3) | 122.2 (121.9-122.4) |
+| Quadro RTX 6000 | CUDA | calibrated | 1 | 10496/10496 | 1900.1 (1791.8-1962.7) | 186.6 (185.9-187.0) | 2660.2 (2654.7-2664.9) | 178.6 (177.8-178.9) |
+| Quadro RTX 6000 | CUDA | calibrated, `--mtp 0` | 0 | 10240/10240 | 1978.4 (1889.7-2029.1) | 126.4 (125.7-126.7) | 2794.6 (2791.2-2798.2) | 122.3 (122.0-122.5) |
+| Quadro RTX 6000 | Vulkan | uncalibrated | 0 | 10240/10240 | 1320.7 (113.2-1934.6) | 95.5 (64.8-111.3) | 2636.5 (2431.1-2743.2) | 102.7 (92.0-108.1) |
+| Quadro RTX 6000 | Vulkan | calibrated | 1 | 10496/10496 | 1572.0 (1065.5-1875.4) | 156.8 (154.4-158.9) | 2557.8 (2552.5-2562.6) | 142.5 (122.0-152.8) |
+| Quadro RTX 6000 | Vulkan | calibrated, `--mtp 0` | 0 | 10240/10240 | 1872.6 (1688.9-1977.9) | 112.3 (112.1-112.6) | 2719.4 (2715.7-2724.0) | 109.0 (108.7-109.2) |
+| RTX 2080 Ti | CUDA | uncalibrated | 0 | 8334/10240 | 1448.6 (267.6-2095.7) | 141.8 (137.7-143.9) | 2748.6 (2600.3-2823.4) | 140.8 (140.2-141.5) |
+| RTX 2080 Ti | CUDA | calibrated | 1 | 8213/10496 | 1898.6 (1852.0-1974.4) | 207.4 (206.8-207.7) | 2594.8 (2593.8-2595.6) | 197.2 (196.0-198.1) |
+| RTX 2080 Ti | CUDA | calibrated, `--mtp 0` | 0 | 8334/10240 | 2023.3 (2018.7-2030.1) | 145.0 (144.3-145.5) | 2713.5 (2712.0-2715.0) | 139.7 (139.4-140.0) |
+| RTX 2080 Ti | Vulkan | uncalibrated | 0 | 8482/10240 | 1317.5 (108.0-1947.8) | 107.7 (60.5-131.4) | 2620.6 (2381.8-2743.3) | 118.7 (103.1-126.7) |
+| RTX 2080 Ti | Vulkan | calibrated | 1 | 8583/10496 | 1777.4 (1685.9-1861.5) | 178.0 (175.9-179.0) | 1864.3 (1471.9-2442.3) | 158.0 (130.9-171.6) |
+| RTX 2080 Ti | Vulkan | calibrated, `--mtp 0` | 0 | 8712/10240 | 1934.8 (1764.2-2051.2) | 130.8 (130.3-131.0) | 2629.7 (2608.8-2642.7) | 126.3 (126.0-126.7) |
+| V100 16 GB | CUDA | uncalibrated | 0 | 10240/10240 | 2194.5 (2053.1-2266.2) | 133.2 (132.4-133.6) | 3199.5 (3197.4-3203.2) | 128.1 (127.7-128.3) |
+| V100 16 GB | CUDA | calibrated | 1 | 10496/10496 | 2103.6 (1953.7-2200.6) | 202.6 (201.0-203.4) | 2964.6 (2956.9-2973.9) | 192.5 (191.5-193.0) |
+| V100 16 GB | CUDA | calibrated, `--mtp 0` | 0 | 10240/10240 | 2253.3 (2233.9-2275.7) | 133.2 (132.3-133.8) | 3144.9 (3138.7-3154.1) | 128.1 (127.7-128.4) |
+| V100 16 GB | Vulkan | uncalibrated | 0 | 10240/10240 | 571.0 (97.2-809.5) | 98.4 (43.1-126.1) | 1682.0 (1679.5-1685.3) | 107.8 (84.0-119.8) |
+| V100 16 GB | Vulkan | calibrated | 2 | 10496/10496 | 766.8 (711.0-794.9) | 209.2 (204.3-211.8) | 1597.7 (1596.9-1599.2) | 170.4 (120.0-195.8) |
+| V100 16 GB | Vulkan | calibrated, `--mtp 0` | 0 | 10240/10240 | 804.6 (796.3-809.8) | 124.9 (124.1-125.4) | 1676.3 (1675.2-1678.0) | 119.7 (119.6-119.9) |
+| V100 32 GB | CUDA | uncalibrated | 0 | 10240/10240 | 1978.8 (1527.4-2205.0) | 125.4 (124.4-126.0) | 3164.0 (3159.2-3167.9) | 120.9 (120.6-121.1) |
+| V100 32 GB | CUDA | calibrated | 1 | 10496/10496 | 2124.0 (2109.9-2149.0) | 192.3 (191.3-193.0) | 2951.1 (2948.3-2956.0) | 182.6 (181.5-183.2) |
+| V100 32 GB | CUDA | calibrated, `--mtp 0` | 0 | 10240/10240 | 2203.4 (2190.7-2217.5) | 125.5 (124.6-126.0) | 3117.6 (3113.5-3123.9) | 120.9 (120.6-121.1) |
+| V100 32 GB | Vulkan | uncalibrated | 0 | 10240/10240 | 586.7 (150.8-806.1) | 102.4 (64.5-121.3) | 1666.7 (1664.5-1668.6) | 107.5 (91.1-115.8) |
+| V100 32 GB | Vulkan | calibrated | 2 | 10496/10496 | 784.8 (774.4-790.6) | 203.0 (198.6-205.3) | 1584.6 (1579.7-1587.2) | 169.4 (130.5-189.0) |
+| V100 32 GB | Vulkan | calibrated, `--mtp 0` | 0 | 10240/10240 | 798.3 (791.1-802.4) | 120.9 (120.0-121.3) | 1664.7 (1662.1-1667.0) | 115.6 (115.6-115.7) |
+
+- The model fits in VRAM everywhere except the RTX 2080 Ti (11 GB), where 8213-8712 experts are resident and the
+  expert hit rate is 99.7-99.9% (uncalibrated CUDA: 637 misses, 1878 cache swaps; calibrated `--mtp 0`: 231 and 9).
+- As on the other cluster, calibrated with `--mtp 0` decodes at the uncalibrated CUDA speed (within 0.5%; 2.3% on the
+  RTX 2080 Ti, where the uncalibrated run starts cold: 637 misses against 231), and the gain is MTP:
+  decode after 512 with MTP on is 1.43-1.85x the `--mtp 0` run with CUDA (RTX 2080 Ti 1.43x, A40 1.85x).
+- The V100 Vulkan prefill is about half its CUDA prefill (pp 4096: 1676 against 3145 tok/s on the 16 GB card): no
+  cooperative matrices there, so the `_dp` kernels run. Decode is close (120 against 128 tok/s with `--mtp 0`).
+- The first Vulkan repetition of the uncalibrated `pp 512` includes pipeline compilation (minima of 97-151 tok/s).
+
+Startup lines:
+
+```
+cuda device 0: NVIDIA A100-PCIE-40GB, sm_80
+cuda shared memory: opt-in 166912 B per block, attn_decode_gqa*: 2112 B static, dynamic max 73728 B, carveout default
+cuda device 0: NVIDIA RTX A6000, sm_86          (the A40 and RTX 4500 Ada, sm_89, print the same shared-memory line)
+cuda shared memory: opt-in 101376 B per block, attn_decode_gqa*: 2112 B static, dynamic max 73728 B, carveout default
+cuda device 0: Quadro RTX 6000, sm_75 (Turing, m16n8k8)          (same suffix and next line for the GeForce RTX 2080 Ti)
+cuda shared memory: opt-in 65536 B per block, attn_decode_gqa*: 2112 B static, dynamic max 63424 B, carveout default
+cuda device 0: Tesla V100-SXM2-16GB, sm_70 (Volta, m8n8k4)          (same for the 32 GB card)
+cuda shared memory: opt-in 98304 B per block, attn_decode_gqa*: 2112 B static, dynamic max 73728 B, carveout default
+
+vulkan: NVIDIA A100-PCIE-40GB (NVIDIA 615.71.09), cooperative matrix f16 yes int8 yes, conditional rendering yes, maxComputeSharedMemorySize 49152
+vulkan: NVIDIA GeForce RTX 2080 Ti (NVIDIA 615.71.09), cooperative matrix f16 yes int8 yes, conditional rendering yes, maxComputeSharedMemorySize 49152
+vulkan: Quadro RTX 6000 (NVIDIA 615.71.09), cooperative matrix f16 yes int8 yes, conditional rendering yes, maxComputeSharedMemorySize 49152
+vulkan: Tesla V100-SXM2-16GB (NVIDIA 580.95.05), cooperative matrix f16 no int8 no, conditional rendering yes, maxComputeSharedMemorySize 49152
+```
+
+`rustingstrata check --backend vulkan`: 96 (kernel, specialization) pairs, largest 49152 bytes (`gemmi`) of 49152
+allowed, on the six cards with cooperative matrices; 84 pairs, largest 49152 bytes (`gemmi_dp`), on both V100s.