Skip to content

docs: RTX 2080 Ti, Quadro RTX 6000, V100, A40, RTX A6000, RTX 4500 Ada and A100 40 GB results; GTX 1080 Ti does not run #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yugt wants to merge 1 commit into
base: main
Choose a base branch
from
+189 −4
Open

docs: RTX 2080 Ti, Quadro RTX 6000, V100, A40, RTX A6000, RTX 4500 Ada and A100 40 GB results; GTX 1080 Ti does not run #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,9 +58,10 @@ IQ3_XXS, IQ3_S, IQ4_XS, IQ4_NL.
Needs Rust (2024 edition) and one of:

- an NVIDIA driver with the CUDA NVRTC library (loaded at run time). Volta (sm_70), Turing (sm_75) and Ampere or newer
(sm_80+) work; CUDA 13 dropped Volta, so a V100 needs the 12.x toolkit. Pascal and older use Vulkan.
- a Vulkan 1.3 driver with shaderFloat64, int8/16/64 and subgroup size 32 or a fixed 64 (tested on RTX 3060, Radeon VII
and V100; Intel Arc lacks shaderFloat64).
(sm_80+) work; CUDA 13 dropped Volta, so a V100 needs the 12.x toolkit. Pascal and older are not supported (a GTX
1080 Ti runs on neither backend, see [docs/SUPPORTED_GPUS.md](docs/SUPPORTED_GPUS.md)).
- a Vulkan 1.3 driver with shaderFloat64, shaderFloat16, int8/16/64 and subgroup size 32 or a fixed 64 (tested on RTX
3060, Radeon VII and V100; Intel Arc lacks shaderFloat64).

`--gpu` picks CUDA when it loads, else Vulkan; `--backend` chooses. The first CUDA run compiles the kernels (~15 s) and
caches them. Every GPU tried so far is in [docs/SUPPORTED_GPUS.md](docs/SUPPORTED_GPUS.md); AMD notes are in
Expand Down
2 changes: 2 additions & 0 deletions docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,5 @@ Raw results and methods, newest work last. The headline numbers are in the [READ
- [m6-results.md](m6-results.md): v1.0 gate run against llama.cpp.
- [V100-results.md](V100-results.md): V100 bring-up, parity and benchmarks (long; written by `tools/bench_v100.sh`).
- [datacenter-nvidia-results.md](datacenter-nvidia-results.md): A100, A30, L40S and H100 test runs and benchmarks.
- [more-nvidia-results.md](more-nvidia-results.md): RTX 2080 Ti, Quadro RTX 6000, V100, A40, RTX A6000, RTX 4500 Ada
and A100 40 GB test runs and benchmarks; the GTX 1080 Ti does not run.
16 changes: 15 additions & 1 deletion docs/SUPPORTED_GPUS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,20 @@ row.
| NVIDIA L40S 48 GB | Ada, sm_89 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 8571 tok/s, decode 185 tok/s. Calibrated in a later job: decode 317 tok/s with MTP 2, 197 with `--mtp 0`. |
| NVIDIA H100 NVL 94 GB, H100 80 GB | Hopper, sm_90 | CUDA | Full test suite passes | yugt | pp 4096 prefill 9131 tok/s, decode 246 tok/s (H100 NVL). Calibrated in a later job on an H100 80 GB: decode 498 tok/s with MTP 2, 265 with `--mtp 0`. |
| NVIDIA H100 NVL 94 GB, H100 80 GB | Hopper, sm_90 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 6428 tok/s, decode 192 tok/s (H100 NVL). Calibrated in a later job on an H100 80 GB: decode 363 tok/s with MTP 2, 216 with `--mtp 0`. |
| NVIDIA GeForce RTX 2080 Ti 11 GB | Turing, sm_75 | CUDA | Full test suite passes | yugt | 8213-8334 of 10240 experts resident (10496 with MTP). pp 4096 prefill 2714 tok/s, decode 140 tok/s with `--mtp 0`; calibrated decode 197 tok/s with MTP 1. See [more-nvidia-results.md](more-nvidia-results.md). |
| NVIDIA GeForce RTX 2080 Ti 11 GB | Turing, sm_75 | Vulkan | Runs, test suite passes with `STRATA_VK_F16_TRUNC=1` | yugt | Cooperative matrices (f16 and int8). The f16 conversion truncates as on the V100: the variable only tells the kernel tests to expect that, and 4 of them fail without it. pp 4096 prefill 2630 tok/s, decode 126 tok/s with `--mtp 0`; calibrated decode 158 tok/s with MTP 1. |
| NVIDIA Quadro RTX 6000 24 GB | Turing, sm_75 | CUDA | Full test suite passes | yugt | pp 4096 prefill 2795 tok/s, decode 122 tok/s with `--mtp 0`; calibrated decode 179 tok/s with MTP 1. |
| NVIDIA Quadro RTX 6000 24 GB | Turing, sm_75 | Vulkan | Runs, test suite passes with `STRATA_VK_F16_TRUNC=1` | yugt | Cooperative matrices (f16 and int8). The f16 conversion truncates as on the V100: the variable only tells the kernel tests to expect that, and 4 of them fail without it. pp 4096 prefill 2719 tok/s, decode 109 tok/s with `--mtp 0`; calibrated decode 143 tok/s with MTP 1. |
| NVIDIA Tesla V100-SXM2 16 GB and 32 GB | Volta, sm_70 | CUDA | Full test suite passes | yugt | NVRTC 12.6, driver 580.95.05. 16 GB card: pp 4096 prefill 3145 tok/s, decode 128 tok/s with `--mtp 0`; calibrated decode 193 tok/s with MTP 1. See [more-nvidia-results.md](more-nvidia-results.md). |
| NVIDIA Tesla V100-SXM2 16 GB and 32 GB | Volta, sm_70 | Vulkan | Runs, test suite passes with `STRATA_VK_F16_TRUNC=1` | yugt | No cooperative matrices. The variable only tells the kernel tests to expect the truncating f16 conversion; 4 of them fail without it. 16 GB card: pp 4096 prefill 1676 tok/s, decode 120 tok/s with `--mtp 0`; calibrated decode 170 tok/s with MTP 2. |
| NVIDIA A100-PCIE 40 GB | Ampere, sm_80 | CUDA | Full test suite passes | yugt | pp 4096 prefill 5877 tok/s, decode 169 tok/s with `--mtp 0`; calibrated decode 275 tok/s with MTP 1. See [more-nvidia-results.md](more-nvidia-results.md). |
| NVIDIA A100-PCIE 40 GB | Ampere, sm_80 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 5639 tok/s, decode 136 tok/s with `--mtp 0`; calibrated decode 223 tok/s with MTP 2. |
| NVIDIA RTX A6000 48 GB | Ampere, sm_86 | CUDA | Full test suite passes | yugt | pp 4096 prefill 5094 tok/s, decode 188 tok/s with `--mtp 0`; calibrated decode 329 tok/s with MTP 2. |
| NVIDIA RTX A6000 48 GB | Ampere, sm_86 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 5052 tok/s, decode 165 tok/s with `--mtp 0`; calibrated decode 252 tok/s with MTP 2. |
| NVIDIA A40 48 GB | Ampere, sm_86 | CUDA | Full test suite passes | yugt | pp 4096 prefill 4805 tok/s, decode 159 tok/s with `--mtp 0`; calibrated decode 289 tok/s with MTP 2. |
| NVIDIA A40 48 GB | Ampere, sm_86 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 4796 tok/s, decode 141 tok/s with `--mtp 0`; calibrated decode 222 tok/s with MTP 2. |
| NVIDIA RTX 4500 Ada 24 GB | Ada, sm_89 | CUDA | Full test suite passes | yugt | pp 4096 prefill 4934 tok/s, decode 153 tok/s with `--mtp 0`; calibrated decode 273 tok/s with MTP 2. |
| NVIDIA RTX 4500 Ada 24 GB | Ada, sm_89 | Vulkan | Full test suite passes | yugt | Cooperative matrices (f16 and int8). pp 4096 prefill 4979 tok/s, decode 142 tok/s with `--mtp 0`; calibrated decode 221 tok/s with MTP 2. |

## Compiles, not run

Expand All @@ -37,7 +51,7 @@ row.
| GPU | Why |
| --- | --- |
| Intel Arc | Vulkan driver has no `shaderFloat64`. |
| NVIDIA Pascal and older (CUDA) | No tensor cores. Use `--backend vulkan`; not tested. |
| NVIDIA Pascal and older | CUDA: no tensor cores; on a GTX 1080 Ti (sm_61) the kernels do not compile (`__nanosleep` is undefined). Vulkan: the GTX 1080 Ti on driver 580.95.05 has no `shaderFloat16`, so no device is usable. See [more-nvidia-results.md](more-nvidia-results.md). |
| GPUs without Vulkan 1.3 and subgroup size 32 or 64 | The Vulkan backend needs both. |

## Architecture tiers (CUDA)
Expand Down
Loading
Loading