Skip to content

Avoid decimal strings for safe Arrow decimal values - #633

Draft
vuanhphung wants to merge 1 commit into
perf/row-convertfrom
perf/row-convert-2
Draft

vuanhphung wants to merge 1 commit into
perf/row-convertfrom
perf/row-convert-2

Conversation

@vuanhphung

Copy link
Copy Markdown
Collaborator

Stacked on #628 (perf/row-convert); retarget to main after it merges.

Arrow Decimal128 → number currently formats an intermediate decimal string for every value. Profiles attribute 60% of decoded mixed-batch time and 6.67 seconds of a live kernel lineitem 2M run to this conversion.

Read the unscaled integer directly from its existing 32-bit words only when the upper words are valid sign extensions and the result is exactly representable. Keep the original conversion for larger values and precision-preserving output. The shared helper also covers nested/legacy rows; BIGINT, IPC, defaults, logging and telemetry are unchanged.

Verification: 42 focused unit tests and static checks pass; full-unit environment issue reproduces on main. Five interleaved fresh-process pairs on Node 24.22.1, decoded batches (no network):

Scenario #628 → this PR, median p95/p99
Mixed 16 columns / object 4844 → 1365 ms 5330 → 1482 ms
Decimal / object 868 → 100 ms 917 → 104 ms
Nested decimals / object 3900 → 1980 ms 4023 → 2043 ms

Live paired results and complete correctness checks are in progress. Factory reproduction scripts and raw runs are in ~/factory/harness/ and ~/factory/runs/; full handoff will be in ~/factory/perf-evidence.md.

Adversarial review: pending independent factory review; keep this PR in draft.

This pull request and its description were written by Isaac.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant