Skip to content

Read Arrow rows directly from vectors with per-batch converters - #628

Open
vuanhphung wants to merge 3 commits into
mainfrom
perf/row-convert
Open

vuanhphung wants to merge 3 commits into
mainfrom
perf/row-convert

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Arrow object materialization spends most CPU repeatedly dispatching types and expanding StructRow proxies: a decoded-batch profile attributes 76% of sampled time to the recursive converter. Both Thrift and kernel results use this path.

Read top-level vectors directly into final rows and resolve scalar converters once per batch. Keep nested conversion and the legacy object path for duplicate/special names unchanged. Values, precision options, logging, disabled materialization and IPC/fetching behavior stay the same.

Verification: 1,831 unit tests pass (agent markers cleared; the unrelated default-environment user-agent failure also reproduces on main), all static checks pass, and 47 live e2e cases plus type/limit/lifecycle checksum matrices pass. Fresh-process live medians below use five interleaved pairs on Node 24; p95/p99 and raw runs are in the factory handoff. Decoded DOUBLE/object CPU: 480 → 45 ms/million rows. Array-mode heap peaks can increase with GC slack; bounded-heap runs still complete.

Live query / row mode Thrift main → branch Kernel main → branch
DOUBLE 30M / object 20.52 → 5.30 s 23.58 → 5.22 s
DOUBLE 30M / array 8.15 → 5.87 s 8.13 → 5.99 s
lineitem 2M / object 29.60 → 20.30 s 27.49 → 18.35 s
lineitem 2M / array 21.58 → 19.71 s 20.65 → 16.75 s

Adversarial review: converged in round 1; gains and parity independently reproduced, fallback/precision/binary tests strengthened and mutation-checked, array-mode memory caveat documented.

This pull request and its description were written by Isaac.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung
vuanhphung marked this pull request as ready for review November 19, 2026 09:25

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

This branch was successfully deployed

1 active deployment
azure-prod — fb040d8e Deployed Nov 19, 2026 by vuanhphung via e2e-test (24.22.1, 24_22_1) #2144
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant