Repository navigation
Encode if GC candidate is dir or file - #6638
Open
keith-turner wants to merge 2 commits into
Open
keith-turner wants to merge 2 commits into
keith-turner wants to merge 2 commits into
Conversation
dlmarion
reviewed
Nov 9, 2026
The GC codes examines candidate paths looking for the depth of the literal `tables` in the path to determine if a candidate is a dir or file. If a candidate path does not have `tables` then its not possible to know if a candidate is a dir or a file. This change encodes if a candidate is a dir or file when writing it to the metadata table because its known at that point. The encoding replaces the current value of `skewed` which was only used for 2.1 to 4.0 upgrade but is no longer needed. The GcCandidate type was only used to read candidates from the metadata table. Changed the code to also use that type to write to the metadata table. This change causes a good bit of diffs, but it was needed to support adding the type. Added a new GcCandidates type that is responsible for removing in use candidates. This consolidates code that used to be spread throughout the GC code. The code is also much simpler than the code it replaced because the type information in the candidate allows the code to be more direct. While working on this noticed a race condition between the delete table code and GC deletion of in use candidates. The table deletion code was inserting candidates and then deleting tablet metadata. If the GC ran after candidate insertion but before deleting tablet metadata, then it could conclude some candidates are in use and delete them. However, nothing may ever insert these candidates again and then the files would never be deleted. Modified the delete table code to insert GC candidates into a temporary area that is ignored by GC, delete the medata table entries, and then copy GC candidates to the place where GC reads from. Made this change in the commit because I thought it was causing a test to fail, but discovered the test failure was caused by another problem that I will open an issue about.
keith-turner
force-pushed
the
gc-candidates
branch
from
November 13, 2026 02:15
83fe4a3 to
dfc0d8f
Compare
keith-turner
marked this pull request as ready for review
November 13, 2026 02:17
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The GC codes examines candidate paths looking for the depth of the
literal
tablesin the path to determine if a candidate is a dir orfile. If a candidate path does not have
tablesthen its not possibleto know if a candidate is a dir or a file.
This change encodes if a candidate is a dir or file when writing it to
the metadata table because its known at that point. The encoding
replaces the current value of
skewedwhich was only used for 2.1 to4.0 upgrade but is no longer needed.
The GcCandidate type was only used to read candidates from the metadata
table. Changed the code to also use that type to write to the metadata
table. This change causes a good bit of diffs, but it was needed to
support adding the type.
Added a new GcCandidates type that is responsible for removing in use
candidates. This consolidates code that used to be spread throughout
the GC code. The code is also much simpler than the code it replaced
because the type information in the candidate allows the code to be more
direct.
While working on this noticed a race condition between the delete table
code and GC deletion of in use candidates. The table deletion code was
inserting candidates and then deleting tablet metadata. If the GC ran
after candidate insertion but before deleting tablet metadata, then it
could conclude some candidates are in use and delete them. However,
nothing may ever insert these candidates again and then the files would
never be deleted. Modified the delete table code to insert GC candidates
into a temporary area that is ignored by GC, delete the medata table
entries, and then copy GC candidates to the place where GC reads from.
Made this change in the commit because I thought it was causing a test
to fail, but discovered the test failure was caused by another problem
that I will open an issue about.