Skip to content

tcu: start row-major A and K-major B at the descriptor's in-row offset - #475

Merged
tinebp merged 1 commit into
masterfrom
fix/tcu-inrow-offset
Nov 15, 2026
Merged

tinebp merged 1 commit into
masterfrom
fix/tcu-inrow-offset

Conversation

@jel221

@jel221 jel221 commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

Human comment:
When WGMMA does full-stripe LMEM reads on row-major A or k-major B, if the LMEM address of the operand didn't start in the full-row boundary (happens when you step along K) rounds down to the nearest row. This adds the offset within the row so that WGMMA reads the correct data.

AI-generated comment:

Fixes the RTL half of #454: a WGMMA descriptor whose base address is not at the start of an LMEM bank row read the wrong data on the RTL while SimX read it correctly. Third of the standalone fixes in #457's review (§8, item 3).

The bug

LMEM is read one bank row at a time (NUM_THREADS × XLEN/8 bytes: 32 B at NT=8, XLEN=32). The tile buffers took the descriptor's base, kept only its bank-row index and dropped its offset inside the row:

  • VX_tcu_abuf (row-major A, ldm != 0): row_word_off_init = step_k × row_k_words, with no in-row term.
  • VX_tcu_bbuf (K-major B): km_word_off_init likewise.

SimX reads A and B from the exact base, so a base such as 0x210 (16 B into a 32 B row) passed on SimX and read every row from the wrong columns on rtlsim and FPGA, with no error. Advancing a descriptor along K by one K-slice (tileK × sizeof(input_t), 16 B for fp16 at NT=8) is the standard pattern and lands mid-row on every other step.

Change

  • VX_tcu_abuf.sv: the row-major starting word offset adds the base's in-row offset ((base >> 2) mod BANK_ROW_WORDS), latched on the first compute uop with the other descriptor fields.
  • VX_tcu_bbuf.sv: the K-major starting word offset adds the same.
  • SimX is unchanged: it already reads the exact base, and its fetch model counts one LMEM read per A row (row-major) and per B row (K-major), as the fixed RTL does.

Block-major layouts are unaffected (their stripes are bank-row aligned by construction), and an aligned base adds zero, so existing kernels see no change in results or cycles.

Scope

The fix is exact when each row's K-slice stays inside one bank row, which holds whenever the base and the row stride are whole K-slices, e.g. a descriptor advanced along K. A base or stride that is not a multiple of the 2-word fetch slice can still cross a bank row and lose words on the RTL; no alignment rule or assert is added for that case.

Tests

  • sgemm_tcu_wg gains layout knobs, in K-slices (tileK elements) so one setting means the same at any NT, type or XLEN:
    • A_PAD_K / A_OFF_K: pad row-major A rows and start them that many K-slices in (WGMMA_RMAJOR_A).
    • WGMMA_KMAJOR_B with B_PAD_K / B_OFF_K: stage B K-major, padded and offset the same way.
  • New CI cases, SimX and rtlsim, XLEN 32 and 64:
    • tensor_wg:sgemm_tcu_wg-a-offset: SS, row-major A, one K-slice in.
    • tensor_wg:sgemm_tcu_wg-b-offset: SS, K-major B, one K-slice in.

Change impact: R2

ci/change_impact.py: R2 (path:rtl, evidence:simx-rtl-lockstep).

  • SimX/RTL lockstep: no SimX change is needed (SimX was already right), and cycles are unchanged for aligned descriptors. Every perf_gate and model_parity case passed unchanged in CI.
  • Golden data: none changed.

Evidence

NT=8, fp16, SS, XLEN=32:

Layout master rtlsim fixed rtlsim fixed SimX
row-major A, aligned pass pass pass
row-major A, 1 K-slice in wrong results pass pass
K-major B, aligned pass pass pass
K-major B, 1 K-slice in wrong results pass pass
both offset 512 / 512 wrong pass pass
  • run: the two new CI cases through pytest on SimX and rtlsim (XLEN 32).
  • run: ci/testcase.py lint.
  • run (CI, 43613123419): 114 jobs pass. The one failure is tests (logic_gate, rtlsim, 32): tensor: dsp 267 -> 283 (+6.0%). It is identical on master (e.g. run 43199063171) and is in a build this change does not touch.
  • not run: FPGA hardware.

@jel221
jel221 requested a review from tinebp November 15, 2026 02:36
@tinebp
tinebp merged commit 57d9b4b into master Nov 15, 2026
114 of 118 checks passed
@tinebp
tinebp deleted the fix/tcu-inrow-offset branch November 15, 2026 04:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants