Repository navigation
GH-3043: Ignore V1 level encodings for omitted streams - #3867
Open
skyon0522-ai wants to merge 1 commit into
Open
skyon0522-ai wants to merge 1 commit into
skyon0522-ai wants to merge 1 commit into
Conversation
Use the existing zero reader when a column's maximum repetition or definition level is zero. Such a column has no corresponding encoded level stream, so its unused header encoding must not select a physical value reader or consume value bytes. Preserve encoding dispatch for nonzero levels and add synthetic V1 regressions for required, optional and repeated columns across two pages. Generated-by: OpenAI Codex
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3043.
Rationale for this change
The attached nanoparquet file uses PLAIN in the V1 repetition/definition
encoding fields for flat required columns. Neither level stream is present.
readPageV1nevertheless selects a reader from each header encoding, whichcan consume value bytes or fail at
ValuesReader.readIntegerbefore thefirst record is returned. The format specification
specifies that these zero-maximum level streams are omitted.
What changes are included in this PR?
Select the existing
ZeroIntegerValuesReaderindependently when thecorresponding maximum level is zero. Keep the existing encoding dispatch
for nonzero levels. The regression uses the upstream V1 writer to generate
two pages, replaces only unused level-encoding fields, and checks exact
values and levels for required, optional and repeated columns.
Are these changes tested?
Validation on Linux (Ubuntu 24.04, Java 17.0.20.1, Maven 3.9.11):
code. The two controls and seven existing cases passed. All 13 passed
with the fix.
parquet-columnreactor test run passed all 1,094 tests.the issue attachment through both reader APIs and a generated control.
eight reactor modules, and
git diff --checkpassed.The entire root reactor, native Windows and macOS were not tested.
Are there any user-facing changes?
Files such as the issue attachment become readable when a V1 header uses
PLAIN for an omitted level stream.
OpenAI Codex assisted with the implementation and tests.