Skip to content

Write each wildcard element as it is walked - #59

Merged
Mearman merged 1 commit into
mainfrom
perf/encode-walk
Oct 21, 2026
Merged

Mearman merged 1 commit into
mainfrom
perf/encode-walk

Conversation

@Mearman

@Mearman Mearman commented Oct 21, 2026

Copy link
Copy Markdown
Member

A wildcard rule used to expand every read path into one site record per element, hold a joined-index map per further read path, and build the whole output list before the first write, which kept several hundred bytes per element beyond the result and hit the heap limit at tens of millions of elements (about 800 bytes per element measured on an eight-rule graph crosswalk, and the table-size abort in the issue). The governing path is now walked one element at a time with the trail held once, the other read paths are read at the element's indexes through the existing single-location reader, and each element's results are written before the next element is visited, so nothing per element outlives the element. Absence and element-pairing failures keep their order, because every path is walked for its count before the write walk begins, exactly as the previous frontier design reported them. The regression test pins the retained growth per element on a five-read-path shape.

Fixes #57.

A wildcard rule expanded every read path into one site record per
element, held a joined-index map per further read path, and built the
whole output list before the first write, so an encode over a large
array kept several hundred bytes per element beyond its result and ran
out of memory at tens of millions of elements. The governing path is
now walked one element at a time with the trail held once, the other
read paths are read at the element's indexes, and each element's
results are written before the next is visited. Absence and
element-pairing failures keep their order: every path is walked for
its count first, so an absent path is reported before a misaligned
one, exactly as before.
@Mearman
Mearman marked this pull request as ready for review October 21, 2026 13:31
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
🔒 Security Review ⚠️ Failed 2026-10-21T13:32:08.229502Z
We couldn’t complete this review. Comment @codex security review to try again.
9553b3b Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@Mearman

Mearman commented Oct 21, 2026

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins.

1 similar comment
@chatgpt-codex-connector

Copy link
Copy Markdown

This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins.

@Mearman
Mearman merged commit 8d9c544 into main Oct 21, 2026
10 checks passed
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 0.8.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

@Mearman
Mearman deleted the perf/encode-walk branch October 21, 2026 23:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

encode holds hundreds of bytes per array element beyond its result

1 participant