Repository navigation
fix(TT-PA-SMALL-WINDOW): the PA device decode committed a bf16 shadow on an f32-contract out — PaOutTypedForCommit; the small-window defect closes - #3571
Open
lu-zero wants to merge 4 commits into
Conversation
…w defect — deterministic garbage at seq=2/5/7, isolated at last The DFlash2 degeneracy hunt's 20+ engine-only waves could not isolate the compute class; the W22 wave's grown harness did: one raw RAC write plus one raw PA decode yields deterministic garbage at seq=2 and seq=5/7 (identical bytes), clean at the neighboring seq values, with the Copy-seeded arm clean everywhere and the RAC's writes landing in all 40 reps — the defect is on the PA read/decode side at small seq, and the engine's DFlash2 degeneracy is this defect reached through the engine's allocator/program-cache state. This commit files the row: W1 characterize the boundary and commit the focused reproduction, W2 root-cause (our small-window branch vs an upstream tt-metal SDPA defect — the indexed_fill precedent says the substrate carries real write-path bugs), W3 fix + the gates that finally produce the stage-C row's verdict number: the acceptance multiple. Records only. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5-3-flash [maki]
… on an f32-contract out — the host write width, not the kernel The DFlash2 degeneracy hunt's isolated reproduction (67d041d76 on row/QWEN38-DFLASH2-STAGE-C: one raw RAC write + one raw PA decode, garbage at seq=2/5/7, clean elsewhere) is characterized, root-caused and fixed. The defect is OURS, in the commit path; the upstream tt-metal sdpa_decode kernel, the RAC write and the page walk are all exonerated. Characterization (docs/bench-evidence/tt-pa-small-window-20261116/ README.md, all logs flat in that directory): the garbage is always in the second KV group's heads (2/3), never bf16-rounded (low 16 bits nonzero — never computed), and its bytes are recycled memory (RISC-V instructions, RTA words, NaNs, plausible previous outputs). Which seq values garbage is NOT intrinsic to cur_pos: the RAC-seeded sweep garbages at seq=1 only, the Copy-seeded sweep at 31/64 seqs, the fresh-cache-per-seq arm at 2/5, the program-cache-disabled arm at 1/3/6/7/8 — each in a fresh process. The stage-C seq-parity set (2/5/7) is that process's state fingerprint; the invariants travel, the set does not. The Copy-seeded arm garbaging falsifies the RAC write-visibility hypothesis. Root cause: TryPagedAttentionDeviceDecode's identity-order commit path moved the bf16 sdpa_decode output onto the caller's output slot as the device shadow. EnsureHostBytes writes the host bytes at the SHADOW's dtype width (the FLINT slot-serve rule: the shadow's dtype is authoritative, never the slot's registered capacity), so an f32-contract out received 512x2 B in a 512x4 B buffer. The f32 consumer then read elements 0..255 as mis-paired bf16s (plausible values, small oracle deltas — read as "clean") and elements 256..511 (heads 2/3) as recycled host bytes — the garbage. The VT_TT_PA_DUMP bisect shows the raw sdpa_decode output clean at every call including the garbage ones, which convicts our commit path and exonerates the kernel. The mechanism check re-reads the same buffer as bf16 pairs and matches the oracle to ~1e-3 on every garbage row while the f32 read shows 1e38. Falsified on the way, recorded so they stay falsified: the upstream tt-metal SDPA kernel (dev_out clean); the program-cache-hit run-args class (the defect is worse with the cache disabled, and the buffer bindings re-patch on every hit at this pin); the writer kernel's unfenced Blackhole L1-cache read of the compute-packed output (the fence was applied to the pin tree, changed nothing — the garbage bits stayed identical — and was reverted). The fix: PaOutTypedForCommit — reconcile the commit's shadow dtype with OUT's contract before CommitDeviceLogical2D, in the decode arm and the prefill arm alike (the chunked-SDPA pieces carry the same violation). f32/bf16 typecast on device; f16 has no ttnn dtype at this pin and declines to the host scatter, which honors every dtype through StoreElemF32 — as do the existing small-D tests the TILE gate declines. The engine's default paths pass bf16 outs (dense_attn_block adt, qwen3_dflash.cpp:1012) so the width mismatch is latent there; the registered VT_QWEN3_ATTN_F32=1 f32 A/B path and every f32-out caller hit it. Capture-safe by the W4 doctrine: the eager warmup step runs the same out dtype, warming the typecast program before any capture replays it. Gates: the focused RAC-seeded sweep red-before (garbage at seq=1/64) green-after 0 garbage / 64, max_abs 0.002238 (the bf16 envelope); all five characterization arms green after the fix (5 passed, 0 failed); the existing PA suites (5 cases, the device-vs-host oracle arms and the pure-prefill arm) and the RAC bit-exact arm unchanged. The characterization sweep, the payload-tracking arms and the program-cache toggle hook (ProgramCacheToggleForTest) land as the row's focused tests. The stage-C consumption gates (Q4_K_M KV64 drafter-ON 3 reps token identity + TPOT vs 6607, UD-Q6_K_XL KV8 vs b36bc61a + TPOT vs 5340.9, drafter-OFF 6/6) run from /tmp/q38-stage-c after this branch merges there and are recorded on the issue. ISSUE-LOCAL-01M7JHW5N40PWJCRG5BAKQN7Y8 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5-3-flash [maki]
…ent in the env doc The env-doc gate (test_shipped_tree_is_fully_covered) requires every shipped env var documented; the previous commit shipped the instrument without its row. Read-only, default-off, byte-identical when unset. ISSUE-LOCAL-01M7JHW5N40PWJCRG5BAKQN7Y8 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5-3-flash [maki]
… exposed latent defects The verdict: the small-window defect was the PA device decode committing a bf16 shadow on an f32-contract out — the same commit-dtype-mismatch class the Q8_K and embed rows fixed on their paths — fixed by PaOutTypedForCommit (typecast the SDPA output to OUT's dtype before CommitDeviceLogical2D, both decode and prefill arms; capture-safe by the warmup doctrine). The characterization's seq-parity framing is falsified as the defect's property: the boundary moves with process state, so the stage-C seq set (2/5/7) was that process's fingerprint; the invariants that travel are second-KV-group heads, never-bf16-rounded recycled bytes (RISC-V instructions, RTA words, NaNs). The program-cache (worse disabled) and L1-fence (changed nothing) hypotheses are falsified and recorded. Gates: focused reproduction red-before -> green-after (0 garbage / 64, max_abs 0.002238 = the bf16 envelope); all five characterization arms green (w3-all-arms-GREEN.log); the PA suites unchanged (GQA reference, device-vs-host oracle, pure-prefill, RAC bit-exact); failing-SETS compared on the stage-C consumption run. Two latent defects the characterization exposed are filed as owed issues rather than absorbed: the batched-decode RAC device lane's second-slot second-KV-head garbage under rich suite state (01M7JYHQJEANK9Z6S0SJWKXWBC), and the PA decode's value-level garbage with the program cache DISABLED at seq>=2 (01M7JYJ1WDJJ3M7E23VQCYS6KC). The row closes; the stage-C row's acceptance gate consumes this fix. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5-3-flash [maki]
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The DFlash2 degeneracy hunt's isolated reproduction (one raw RAC write
at 5/7) is root-caused and fixed: the PA device decode committed a bf16
shadow on an f32-contract out — the same commit-dtype-mismatch class the
Q8_K and embed rows fixed on their paths. The fix (PaOutTypedForCommit,
tenstorrent_paged.cpp): typecast the SDPA output to OUT's dtype before
CommitDeviceLogical2D, in BOTH the decode and prefill arms; capture-safe
by the warmup doctrine (the eager warmup runs the same out dtype,
warming the typecast program).
The characterization's seq-parity framing is falsified as the defect's
property: the boundary moves with process state, so the stage-C seq set
(2/5/7) was that process's fingerprint; the invariants that travel are
second-KV-group heads, never-bf16-rounded recycled bytes (RISC-V
instructions, RTA words, NaNs). The program-cache hypothesis (worse
disabled) and the L1-fence candidate (changed nothing) are falsified and
recorded.
Gates: focused reproduction red-before -> green-after (0 garbage / 64,
max_abs 0.002238 = the bf16 envelope); all five characterization arms
green; the PA suites unchanged (GQA reference, device-vs-host oracle,
pure-prefill, RAC bit-exact); failing-SETS compared on the stage-C
consumption run.
Two latent defects the characterization exposed are filed as owed issues
rather than absorbed: the batched-decode RAC device lane's second-slot
second-KV-head garbage under rich suite state
(ISSUE-LOCAL-01M7JYHQJEANK9Z6S0SJWKXWBC), and the PA decode's value-level
garbage with the program cache DISABLED at seq>=2
(ISSUE-LOCAL-01M7JYJ1WDJJ3M7E23VQCYS6KC). The row closes; the stage-C
row's acceptance gate consumes this fix — the DFlash2 acceptance
multiple (the row's verdict number) is the next measurement.
VT_TT_PA_DUMP (the bisection instrument) documented in docs/ENVIRONMENT.md.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5-3-flash [maki]