Skip to content

Encode if GC candidate is dir or file - #6638

Open
keith-turner wants to merge 2 commits into
apache:mainfrom
keith-turner:gc-candidates
Open

keith-turner wants to merge 2 commits into
apache:mainfrom
keith-turner:gc-candidates

Conversation

@keith-turner

@keith-turner keith-turner commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

The GC codes examines candidate paths looking for the depth of the
literal tables in the path to determine if a candidate is a dir or
file. If a candidate path does not have tables then its not possible
to know if a candidate is a dir or a file.

This change encodes if a candidate is a dir or file when writing it to
the metadata table because its known at that point. The encoding
replaces the current value of skewed which was only used for 2.1 to
4.0 upgrade but is no longer needed.

The GcCandidate type was only used to read candidates from the metadata
table. Changed the code to also use that type to write to the metadata
table. This change causes a good bit of diffs, but it was needed to
support adding the type.

Added a new GcCandidates type that is responsible for removing in use
candidates. This consolidates code that used to be spread throughout
the GC code. The code is also much simpler than the code it replaced
because the type information in the candidate allows the code to be more
direct.

While working on this noticed a race condition between the delete table
code and GC deletion of in use candidates. The table deletion code was
inserting candidates and then deleting tablet metadata. If the GC ran
after candidate insertion but before deleting tablet metadata, then it
could conclude some candidates are in use and delete them. However,
nothing may ever insert these candidates again and then the files would
never be deleted. Modified the delete table code to insert GC candidates
into a temporary area that is ignored by GC, delete the medata table
entries, and then copy GC candidates to the place where GC reads from.
Made this change in the commit because I thought it was causing a test
to fail, but discovered the test failure was caused by another problem
that I will open an issue about.

The GC codes examines candidate paths looking for the depth of the
literal `tables` in the path to determine if a candidate is a dir or
file.  If a candidate path does not have `tables` then its not possible
to know if a candidate is a dir or a file.

This change encodes if a candidate is a dir or file when writing it to
the metadata table because its known at that point. The encoding
replaces the current value of `skewed` which was only used for 2.1 to
4.0 upgrade but is no longer needed.

The GcCandidate type was only used to read candidates from the metadata
table. Changed the code to also use that type to write to the metadata
table. This change causes a good bit of diffs, but it was needed to
support adding the type.

Added a new GcCandidates type that is responsible for removing in use
candidates.  This consolidates code that used to be spread throughout
the GC code. The code is also much simpler than the code it replaced
because the type information in the candidate allows the code to be more
direct.

While working on this noticed a race condition between the delete table
code and GC deletion of in use candidates. The table deletion code was
inserting candidates and then deleting tablet metadata. If the GC ran
after candidate insertion but before deleting tablet metadata, then it
could conclude some candidates are in use and delete them. However,
nothing may ever insert these candidates again and then the files would
never be deleted. Modified the delete table code to insert GC candidates
into a temporary area that is ignored by GC, delete the medata table
entries, and then copy GC candidates to the place where GC reads from.
Made this change in the commit because I thought it was causing a test
to fail, but discovered the test failure was caused by another problem
that I will open an issue about.
@keith-turner
keith-turner marked this pull request as ready for review November 13, 2026 02:17

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants