You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(server): add an RTX PRO 6000 32K thinking-bounded RLM profile for Qwen3.8-27B-FP8 - #949
Adds rtx-pro-6000-32k-thinking-bounded-rlm (and its identical -no-spec grammar twin) to Qwen/Qwen3.8-27B-FP8: a lane for recursive-language-model (RLM) workloads, where a driver model makes many short calls and many traces run at once.
The shipped rtx-pro-6000-256k profiles run one request at a time (--max-running-requests 1, decode graphs to 1). An RLM call never needs the 256K window, and a single trace keeps only about one request in flight, so a GPU only gets busy when many traces share it.
What the profile is
The rtx-pro-6000-256k-no-spec launch with these changes, and nothing else:
the model card's thinking recipe: 1.0 / 0.95 / 20 / 0.0, as h100-256k-thinking
Measurements
One RTX PRO 6000 serving this configuration through sie-server (v0.11.0 image), with requests replayed from RLM runs on agent-trace failure attribution: about 20 requests per trace, 9K prompt and 2.4K completion tokens on average. Each level was measured over a 10-minute window after a 2-minute warm-up.
Requests in flight
8
16
32
48
64
Completed requests per minute
6.6
10.8
14.8
15.8
16.5
Output tokens/s
280
426
586
643
630
GPU-minutes per trace
2.97
1.82
1.32
1.24
1.19
The 8-in-flight point was taken with the same settings capped at 8 running requests. A real run of 100 traces, 32 at a time on one GPU, used 1.27 GPU-minutes per trace, with no timeouts or rejected requests; its answers matched two earlier runs at 8 per GPU within run-to-run variation.
Not measured individually: the larger admission budget (it made no difference at 32 in flight), and prefill chunk size or an FP8 KV cache, which stay at the RTX defaults.
Tests
test_qwen38_generation_profiles.py: the profile set; a new test that pins the profile as the RTX no-spec launch plus exactly the changes above; grammar requests stay on the lane.
test_generation_mode_profiles.py: the new pair is listed with Qwen3.8's thinking profiles.
mise run test -- packages/sie_server/tests/config (1,738 passed), plus test_remote_profile_window.py, test_config_delta_rejection.py, processors/test_grammar_prewarm.py and test_model_yaml_filenames.py; mise run lint and mise run typecheck pass.
Summary by CodeRabbit
New Features
Added a Qwen 3.8 27B FP8 profile for RTX Pro 6000 with bounded thinking and a non-speculative grammar variant.
The profile supports context lengths up to 32,768 tokens, outputs up to 8,192 tokens, and thinking up to 4,096 tokens. It can handle up to 64 concurrent requests.
…cursive-language-model calls on Qwen3.8-27B-FP8
A recursive language model (RLM) drives a model through many short calls: each fits 32K tokens, and a
trace needs about 20 of them, mostly sequential within the trace. The shipped RTX PRO 6000 profiles run
one request at a time, so many traces queue behind each other.
rtx-pro-6000-32k-thinking-bounded-rlm is the rtx-pro-6000-256k launch without speculation plus the
strict-thinking adapter and qwen3 reasoning parser with a hard 4,096-token thinking bound, a 32,768-token
window, up to 64 running requests (decode CUDA graphs to 64) and a 1,048,576-token admission budget.
Sampling is the model card's thinking recipe. Its -no-spec twin is identical.
Measured on one RTX PRO 6000 with replayed RLM requests: 2.97 GPU-minutes per trace at 8 requests in
flight, 1.32 at 32 and 1.19 at 64; a real run of 100 traces, 32 at a time, used 1.27.
No actionable comments were generated in the recent review.
Walkthrough
Adds an RTX PRO 6000 bounded-thinking profile for Qwen3.8-27B-FP8, with a no-spec grammar alias. Updates profile associations and tests for its configuration, generated variants, and grammar resolution.
Changes: RTX bounded-thinking profile
Layer / File(s)
Summary
Profile definition and grammar alias packages/sie_server/models/Qwen__Qwen3.8-27B-FP8.yaml
Adds an RTX PRO 6000 profile with Qwen reasoning parsing, a 4,096-token thinking limit, 32,768-token sequence limit, and non-speculative execution settings. Adds a no-spec grammar alias that extends the profile without overrides.
Profile registration and validation packages/sie_server/tests/config/test_generation_mode_profiles.py, packages/sie_server/tests/config/test_qwen38_generation_profiles.py
Adds the new profiles to expected associations and tests their configuration, generated variants, and grammar resolution.
Possibly related PRs
superlinked/sie#832: Adds profiles to the same model configuration and changes the same expected profile set in test_qwen38_generation_profiles.py.
Suggested reviewers:krisztian-gajdar, mamayer19
Merge Risk: Minimal · up to e0aa4
This adds a new opt-in RTX PRO 6000 bounded-thinking profile for Qwen3.8-27B-FP8 and a grammar alias, with tests covering both. No concrete merge-blocking risk was found.
Pre-merge checks: 4
Check name
Status
Explanation
Description Check
Check skipped - CodeRabbit’s high-level summary is enabled.
Title check
The title names the main change: adding an RTX PRO 6000 32K thinking-bounded RLM profile for Qwen3.8-27B-FP8. The raw summary confirms the new profile, its no-spec grammar alias, and the tests coverin…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title names the main change: adding an RTX PRO 6000 32K thinking-bounded RLM profile for Qwen3.8-27B-FP8. The raw summary confirms the new profile, its no-spec grammar alias, and the tests covering them. The title is specific and accurate. Its length (86 characters, 12 words) is acceptable, though it is slightly long.
Finishing Touches
Generate unit tests (beta)
Commit to this branch
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: CHILL
Plan: Team
Run ID: 9c9f6b32-04f7-433b-a4f4-d36d2468c50a
Commits
Reviewing files that changed from the base of the PR and between 4ebd4b8 and e0aa4c1.
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Summary
Adds
rtx-pro-6000-32k-thinking-bounded-rlm(and its identical-no-specgrammar twin) toQwen/Qwen3.8-27B-FP8: a lane for recursive-language-model (RLM) workloads, where a driver model makes many short calls and many traces run at once.The shipped
rtx-pro-6000-256kprofiles run one request at a time (--max-running-requests 1, decode graphs to 1). An RLM call never needs the 256K window, and a single trace keeps only about one request in flight, so a GPU only gets busy when many traces share it.What the profile is
The
rtx-pro-6000-256k-no-speclaunch with these changes, and nothing else:rtx-pro-6000-256k-no-specmax_seq_length--max-running-requests/--cuda-graph-max-bs-decodekv_budget_tokens(SIE admission)SGLangCuda13AdapterSGLangStrictThinkingAdapter,reasoning_parser: qwen3,enable_thinking: trueSGLANG_MAX_THINK_TOKENS: "4096"max_output_tokensh100-256k-thinkingMeasurements
One RTX PRO 6000 serving this configuration through
sie-server(v0.11.0 image), with requests replayed from RLM runs on agent-trace failure attribution: about 20 requests per trace, 9K prompt and 2.4K completion tokens on average. Each level was measured over a 10-minute window after a 2-minute warm-up.The 8-in-flight point was taken with the same settings capped at 8 running requests. A real run of 100 traces, 32 at a time on one GPU, used 1.27 GPU-minutes per trace, with no timeouts or rejected requests; its answers matched two earlier runs at 8 per GPU within run-to-run variation.
Not measured individually: the larger admission budget (it made no difference at 32 in flight), and prefill chunk size or an FP8 KV cache, which stay at the RTX defaults.
Tests
test_qwen38_generation_profiles.py: the profile set; a new test that pins the profile as the RTX no-spec launch plus exactly the changes above; grammar requests stay on the lane.test_generation_mode_profiles.py: the new pair is listed with Qwen3.8's thinking profiles.mise run test -- packages/sie_server/tests/config(1,738 passed), plustest_remote_profile_window.py,test_config_delta_rejection.py,processors/test_grammar_prewarm.pyandtest_model_yaml_filenames.py;mise run lintandmise run typecheckpass.Summary by CodeRabbit