Repository navigation
Conversation
erezrokah
self-requested a review
October 14, 2026 09:39
Member
|
Can we flush pending checkpoints while the source is idle and split large checkpoint batches to stay within the gRPC message limit? |
Member
Author
|
@erezrokah Both done in 118eac4:
Checkpoint messages from the source are not split yet: a single |
- Validate checkpoint column types instead of panicking on an unexpected version type - Report a pipeline's error instead of a cancelled commit when a pipeline fails during a commit - Name the destination whose flush failed - Log when the source sends checkpoints, instead of claiming they are enabled up front - Test the checkpoint flow through syncConnectionV3 with fake plugins, and the switch that enables it
Member
|
Could we apply |
Member
Author
|
@erezrokah Done in 30ee29e: the receive-error path now goes through the same check (renamed to |
erezrokah
approved these changes
Oct 14, 2026
- Build and send state records one at a time, and count every column in the record size estimate - Report a duplicate state column as such, and route receive errors through receiveError - Test that a flush waits for pipeline delivery, idle commits through syncConnectionV3, backend errors across several records, and prefix matching of the state backend
- Flush every destination that has not failed yet at the end of the sync, so later failures skip delete-stale too - Keep syncing other sources when a sync finishes without saving its state, and report every such source at the end - Treat a failed state backend flush like a failed destination flush - Report destination commit failures together with hard flush errors - Prefer the pipelines' errors over the receive loop's instead of filtering each loop error - Word the error differently for sources that sent no checkpoints - Warn about skipped delete-stale only for destinations that run it - Bump plugin-pb-go to the regenerated flush_error wording
- Skip delete-stale only for the tables a destination failed to write, when it names them - Ignore the end of a write stream after the destination reported a failed flush, and close every destination before returning - Keep the unsaved state errors of earlier sources when a later source fails - Pin the preference of pipeline errors over the receive loop's in a helper with a deterministic test - Warn append mode destinations that the next sync appends the unsaved rows again - Stop UpdateTableNames from removing tables from its input, which skipped delete-stale on every destination after the first - Bump plugin-pb-go to the failed_tables field
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When every destination supports flush, the CLI flushes them before it writes each source state checkpoint to the backend.
Tested with a local SQLite sync: after a source crash, the stored cursor never passed the committed rows.
Depends on cloudquery/plugin-pb#42 and cloudquery/plugin-pb-go#710:
go.modreplaces plugin-pb-go with that branch until it is released.Checkpoints stay off with transformers or destinations without flush support.
If a destination or the state backend reports a failed flush, the CLI saves no more state for that source, finishes the sync and exits with an error. Other sources still sync.
A destination that ends its stream after it reports the failure still lets the sync finish.
Side effects of this policy:
Also fixes delete-stale running only on the first
overwrite-delete-staledestination:UpdateTableNamesremoved the tables from its input map.