Skip to content

fix(deployer): reject missing accelerator mappings - #43

Merged
thxCode merged 1 commit into
gpustack:mainfrom
yxf0314:issue/6755-ascend-device-mask
Nov 10, 2026
Merged

thxCode merged 1 commit into
gpustack:mainfrom
yxf0314:issue/6755-ascend-device-mask

Conversation

@yxf0314

@yxf0314 yxf0314 commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Publish device mappings only after detection completes and retry empty results on the next lookup. Reject Docker workloads when their requested accelerator mapping is missing so a privileged container cannot start with an empty device mask.

Ref gpustack/gpustack#6755

Validation

  • 273 deployer tests passed. make lint passed.
  • On Ascend 910B2 with the real Docker API, the forced overlap reproduced the empty prefill mask in 5 of 5 runs on Runtime 0.2.7.post1. All 5 runs with the candidate wheel produced backend masks 2,3 for prefill and 0,1 for decode.
  • An empty discovery result blocked container creation. A subsequent request on the same deployer succeeded once discovery recovered.

Hardware validation covered Docker container configuration. Full vLLM PD inference is pending.

Copilot AI balanced review requested due to automatic review settings November 10, 2026 12:10

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces thread-safe device material detection and caching across the deployer modules. It adds a locking mechanism to prevent race conditions during device detection, updates the Docker container creation logic to handle device mapping more robustly, and includes comprehensive tests to verify concurrent device detection and error handling. I have no feedback to provide as there were no review comments.

@yxf0314
yxf0314 requested a review from thxCode November 10, 2026 12:12

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Empty device discovery allows privileged accelerator requests to pass with fallback-only UNKNOWN_* masks, permitting container creation without enforcing the requested device selection.

1 open issue
What changed in this PR

This PR addresses missing accelerator mappings in GPUStack Runtime’s deployment layer, targeting the incorrect Ascend device selection reported in gpustack/gpustack#6755.

Changes:

  • Publish completed device mappings under a lock and retry discovery after empty results.
  • Reject empty Docker device mappings and adapt Kubernetes’ policy-specific cache.
  • Add tests for missing Ascend mappings, discovery recovery, and concurrent lookups.
File Description
tests/​gpustack_runtime/​deployer/​test_visible_devices_ordering.py Adds rejection, retry, and concurrent mapping tests.
gpustack_runtime/​deployer/​kubernetes.py Adapts policy-specific caching to the new material lookup flow.
gpustack_runtime/​deployer/​docker.py Consolidates device requests and rejects empty mappings.
gpustack_runtime/​deployer/​__types__.py Synchronizes cache initialization and leaves empty discovery results uncached.

Review effort: Balanced

Comment thread gpustack_runtime/deployer/docker.py Resolved
- Publish completed device mappings atomically so concurrent workload creation sees consistent results.
- Retry empty device discovery results and preserve policy-specific Kubernetes device mapping caches.
- Reject Docker accelerator requests without mappings and cover missing devices and concurrent initialization.

Signed-off-by: yxf <xf.ye@gpustack.ai>
@yxf0314
yxf0314 force-pushed the issue/6755-ascend-device-mask branch from 7665adb to c2ddfb8 Compare November 10, 2026 12:16

@thxCode thxCode left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thxCode
thxCode merged commit 42d5150 into gpustack:main Nov 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants