Repository navigation
tcu: start row-major A and K-major B at the descriptor's in-row offset - #475
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Human comment:
When WGMMA does full-stripe LMEM reads on row-major A or k-major B, if the LMEM address of the operand didn't start in the full-row boundary (happens when you step along K) rounds down to the nearest row. This adds the offset within the row so that WGMMA reads the correct data.
AI-generated comment:
Fixes the RTL half of #454: a WGMMA descriptor whose base address is not at the start of an LMEM bank row read the wrong data on the RTL while SimX read it correctly. Third of the standalone fixes in #457's review (§8, item 3).
The bug
LMEM is read one bank row at a time (
NUM_THREADS × XLEN/8bytes: 32 B at NT=8, XLEN=32). The tile buffers took the descriptor's base, kept only its bank-row index and dropped its offset inside the row:VX_tcu_abuf(row-major A,ldm != 0):row_word_off_init = step_k × row_k_words, with no in-row term.VX_tcu_bbuf(K-major B):km_word_off_initlikewise.SimX reads A and B from the exact base, so a base such as
0x210(16 B into a 32 B row) passed on SimX and read every row from the wrong columns on rtlsim and FPGA, with no error. Advancing a descriptor along K by one K-slice (tileK × sizeof(input_t), 16 B for fp16 at NT=8) is the standard pattern and lands mid-row on every other step.Change
VX_tcu_abuf.sv: the row-major starting word offset adds the base's in-row offset ((base >> 2) mod BANK_ROW_WORDS), latched on the first compute uop with the other descriptor fields.VX_tcu_bbuf.sv: the K-major starting word offset adds the same.Block-major layouts are unaffected (their stripes are bank-row aligned by construction), and an aligned base adds zero, so existing kernels see no change in results or cycles.
Scope
The fix is exact when each row's K-slice stays inside one bank row, which holds whenever the base and the row stride are whole K-slices, e.g. a descriptor advanced along K. A base or stride that is not a multiple of the 2-word fetch slice can still cross a bank row and lose words on the RTL; no alignment rule or assert is added for that case.
Tests
sgemm_tcu_wggains layout knobs, in K-slices (tileKelements) so one setting means the same at any NT, type or XLEN:A_PAD_K/A_OFF_K: pad row-major A rows and start them that many K-slices in (WGMMA_RMAJOR_A).WGMMA_KMAJOR_BwithB_PAD_K/B_OFF_K: stage B K-major, padded and offset the same way.tensor_wg:sgemm_tcu_wg-a-offset: SS, row-major A, one K-slice in.tensor_wg:sgemm_tcu_wg-b-offset: SS, K-major B, one K-slice in.Change impact: R2
ci/change_impact.py: R2 (path:rtl,evidence:simx-rtl-lockstep).perf_gateandmodel_paritycase passed unchanged in CI.Evidence
NT=8, fp16, SS, XLEN=32:
ci/testcase.py lint.tests (logic_gate, rtlsim, 32):tensor: dsp 267 -> 283 (+6.0%). It is identical on master (e.g. run 43199063171) and is in a build this change does not touch.