Skip to content

fix(TT-PA-SMALL-WINDOW): the PA device decode committed a bf16 shadow on an f32-contract out — PaOutTypedForCommit; the small-window defect closes - #3571

Open
lu-zero wants to merge 4 commits into
localai-org:mainfrom
lu-zero:row/TT-PA-SMALL-WINDOW
Open

lu-zero wants to merge 4 commits into
localai-org:mainfrom
lu-zero:row/TT-PA-SMALL-WINDOW

Conversation

@lu-zero

@lu-zero lu-zero commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

The DFlash2 degeneracy hunt's isolated reproduction (one raw RAC write

  • one raw PA decode: deterministic garbage at seq=2/5/7, identical bytes
    at 5/7) is root-caused and fixed: the PA device decode committed a bf16
    shadow on an f32-contract out — the same commit-dtype-mismatch class the
    Q8_K and embed rows fixed on their paths. The fix (PaOutTypedForCommit,
    tenstorrent_paged.cpp): typecast the SDPA output to OUT's dtype before
    CommitDeviceLogical2D, in BOTH the decode and prefill arms; capture-safe
    by the warmup doctrine (the eager warmup runs the same out dtype,
    warming the typecast program).

The characterization's seq-parity framing is falsified as the defect's
property: the boundary moves with process state, so the stage-C seq set
(2/5/7) was that process's fingerprint; the invariants that travel are
second-KV-group heads, never-bf16-rounded recycled bytes (RISC-V
instructions, RTA words, NaNs). The program-cache hypothesis (worse
disabled) and the L1-fence candidate (changed nothing) are falsified and
recorded.

Gates: focused reproduction red-before -> green-after (0 garbage / 64,
max_abs 0.002238 = the bf16 envelope); all five characterization arms
green; the PA suites unchanged (GQA reference, device-vs-host oracle,
pure-prefill, RAC bit-exact); failing-SETS compared on the stage-C
consumption run.

Two latent defects the characterization exposed are filed as owed issues
rather than absorbed: the batched-decode RAC device lane's second-slot
second-KV-head garbage under rich suite state
(ISSUE-LOCAL-01M7JYHQJEANK9Z6S0SJWKXWBC), and the PA decode's value-level
garbage with the program cache DISABLED at seq>=2
(ISSUE-LOCAL-01M7JYJ1WDJJ3M7E23VQCYS6KC). The row closes; the stage-C
row's acceptance gate consumes this fix — the DFlash2 acceptance
multiple (the row's verdict number) is the next measurement.

VT_TT_PA_DUMP (the bisection instrument) documented in docs/ENVIRONMENT.md.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5-3-flash [maki]

…w defect — deterministic garbage at seq=2/5/7, isolated at last

The DFlash2 degeneracy hunt's 20+ engine-only waves could not isolate
the compute class; the W22 wave's grown harness did: one raw RAC write
plus one raw PA decode yields deterministic garbage at seq=2 and
seq=5/7 (identical bytes), clean at the neighboring seq values, with
the Copy-seeded arm clean everywhere and the RAC's writes landing in
all 40 reps — the defect is on the PA read/decode side at small seq,
and the engine's DFlash2 degeneracy is this defect reached through the
engine's allocator/program-cache state. This commit files the row:
W1 characterize the boundary and commit the focused reproduction, W2
root-cause (our small-window branch vs an upstream tt-metal SDPA
defect — the indexed_fill precedent says the substrate carries real
write-path bugs), W3 fix + the gates that finally produce the stage-C
row's verdict number: the acceptance multiple. Records only.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5-3-flash [maki]
… on an f32-contract out — the host write width, not the kernel

The DFlash2 degeneracy hunt's isolated reproduction (67d041d76 on
row/QWEN38-DFLASH2-STAGE-C: one raw RAC write + one raw PA decode,
garbage at seq=2/5/7, clean elsewhere) is characterized, root-caused and
fixed. The defect is OURS, in the commit path; the upstream tt-metal
sdpa_decode kernel, the RAC write and the page walk are all exonerated.

Characterization (docs/bench-evidence/tt-pa-small-window-20261116/
README.md, all logs flat in that directory): the garbage is always in the
second KV group's heads (2/3), never bf16-rounded (low 16 bits nonzero —
never computed), and its bytes are recycled memory (RISC-V instructions,
RTA words, NaNs, plausible previous outputs). Which seq values garbage is
NOT intrinsic to cur_pos: the RAC-seeded sweep garbages at seq=1 only,
the Copy-seeded sweep at 31/64 seqs, the fresh-cache-per-seq arm at 2/5,
the program-cache-disabled arm at 1/3/6/7/8 — each in a fresh process.
The stage-C seq-parity set (2/5/7) is that process's state fingerprint;
the invariants travel, the set does not. The Copy-seeded arm garbaging
falsifies the RAC write-visibility hypothesis.

Root cause: TryPagedAttentionDeviceDecode's identity-order commit path
moved the bf16 sdpa_decode output onto the caller's output slot as the
device shadow. EnsureHostBytes writes the host bytes at the SHADOW's
dtype width (the FLINT slot-serve rule: the shadow's dtype is
authoritative, never the slot's registered capacity), so an f32-contract
out received 512x2 B in a 512x4 B buffer. The f32 consumer then read
elements 0..255 as mis-paired bf16s (plausible values, small oracle
deltas — read as "clean") and elements 256..511 (heads 2/3) as recycled
host bytes — the garbage. The VT_TT_PA_DUMP bisect shows the raw
sdpa_decode output clean at every call including the garbage ones, which
convicts our commit path and exonerates the kernel. The mechanism check
re-reads the same buffer as bf16 pairs and matches the oracle to ~1e-3
on every garbage row while the f32 read shows 1e38.

Falsified on the way, recorded so they stay falsified: the upstream
tt-metal SDPA kernel (dev_out clean); the program-cache-hit run-args
class (the defect is worse with the cache disabled, and the buffer
bindings re-patch on every hit at this pin); the writer kernel's
unfenced Blackhole L1-cache read of the compute-packed output (the
fence was applied to the pin tree, changed nothing — the garbage bits
stayed identical — and was reverted).

The fix: PaOutTypedForCommit — reconcile the commit's shadow dtype with
OUT's contract before CommitDeviceLogical2D, in the decode arm and the
prefill arm alike (the chunked-SDPA pieces carry the same violation).
f32/bf16 typecast on device; f16 has no ttnn dtype at this pin and
declines to the host scatter, which honors every dtype through
StoreElemF32 — as do the existing small-D tests the TILE gate declines.
The engine's default paths pass bf16 outs (dense_attn_block adt,
qwen3_dflash.cpp:1012) so the width mismatch is latent there; the
registered VT_QWEN3_ATTN_F32=1 f32 A/B path and every f32-out caller
hit it. Capture-safe by the W4 doctrine: the eager warmup step runs the
same out dtype, warming the typecast program before any capture
replays it.

Gates: the focused RAC-seeded sweep red-before (garbage at seq=1/64)
green-after 0 garbage / 64, max_abs 0.002238 (the bf16 envelope); all
five characterization arms green after the fix (5 passed, 0 failed);
the existing PA suites (5 cases, the device-vs-host oracle arms and the
pure-prefill arm) and the RAC bit-exact arm unchanged. The
characterization sweep, the payload-tracking arms and the program-cache
toggle hook (ProgramCacheToggleForTest) land as the row's focused
tests. The stage-C consumption gates (Q4_K_M KV64 drafter-ON 3 reps
token identity + TPOT vs 6607, UD-Q6_K_XL KV8 vs b36bc61a + TPOT vs
5340.9, drafter-OFF 6/6) run from /tmp/q38-stage-c after this branch
merges there and are recorded on the issue.

ISSUE-LOCAL-01M7JHW5N40PWJCRG5BAKQN7Y8

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5-3-flash [maki]
…ent in the env doc

The env-doc gate (test_shipped_tree_is_fully_covered) requires every
shipped env var documented; the previous commit shipped the instrument
without its row. Read-only, default-off, byte-identical when unset.

ISSUE-LOCAL-01M7JHW5N40PWJCRG5BAKQN7Y8

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5-3-flash [maki]
… exposed latent defects

The verdict: the small-window defect was the PA device decode
committing a bf16 shadow on an f32-contract out — the same
commit-dtype-mismatch class the Q8_K and embed rows fixed on their
paths — fixed by PaOutTypedForCommit (typecast the SDPA output to
OUT's dtype before CommitDeviceLogical2D, both decode and prefill
arms; capture-safe by the warmup doctrine). The characterization's
seq-parity framing is falsified as the defect's property: the boundary
moves with process state, so the stage-C seq set (2/5/7) was that
process's fingerprint; the invariants that travel are second-KV-group
heads, never-bf16-rounded recycled bytes (RISC-V instructions, RTA
words, NaNs). The program-cache (worse disabled) and L1-fence
(changed nothing) hypotheses are falsified and recorded.

Gates: focused reproduction red-before -> green-after (0 garbage / 64,
max_abs 0.002238 = the bf16 envelope); all five characterization arms
green (w3-all-arms-GREEN.log); the PA suites unchanged (GQA reference,
device-vs-host oracle, pure-prefill, RAC bit-exact); failing-SETS
compared on the stage-C consumption run.

Two latent defects the characterization exposed are filed as owed
issues rather than absorbed: the batched-decode RAC device lane's
second-slot second-KV-head garbage under rich suite state
(01M7JYHQJEANK9Z6S0SJWKXWBC), and the PA decode's value-level garbage
with the program cache DISABLED at seq>=2
(01M7JYJ1WDJJ3M7E23VQCYS6KC). The row closes; the stage-C row's
acceptance gate consumes this fix.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5-3-flash [maki]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant