Skip to content

Expose checkpoint progress and buffer replica checkpoint writes - #680

Merged
matrixarkai merged 3 commits into
mainfrom
replica-checkpoint-observability
Nov 18, 2026
Merged

matrixarkai merged 3 commits into
mainfrom
replica-checkpoint-observability

Conversation

@matrixarkai

@matrixarkai matrixarkai commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Standalone replicas now expose bounded binary protobuf journal status and an explicit checkpoint request. Clients verify publication against both the requested epoch and journal identity, so a replaced or reopened legacy journal cannot satisfy an old checkpoint ticket. Memory-only and Raft nodes refuse these local maintenance operations; journal I/O failures fence the replica.

Default replica checkpoints now coalesce fragmented encoder output with a fixed 64 KiB buffer and explicitly flush it before rewriting the checkpoint header, synchronizing the file, and publishing the replacement. Stored bytes, checksums, synchronization, and recovery rules remain unchanged. Final write and flush failures propagate to the existing failure fence.

The paired corpus test drains warmup and final checkpoints before sampling Linux data-process I/O. It reports acknowledgment latency and QPS separately from the full cycle, includes kernel writes and cancellations, and checks retained acknowledged pages on every replica after restart. Reads use resident payloads. The timed requests directly address data replicas; metadata publication QPS and Windows device I/O are outside their scope.

In one local WSL release run with 64 MiB of background payload per data replica, 16 clients and 512 fixed-suffix appends, the three data processes made 12,001 write calls, versus about 921,000 in three preceding unbuffered runs. Kernel write bytes remained near 658 MB. CPU ticks were 542, versus 705–839 before buffering. These short stress phases include completed checkpoints and use a 1 MiB checkpoint interval, three acknowledged copies and one-copy reads. Timing varied substantially; this is not a reliable latency or sustained-QPS claim.

Validation covers canonical bounded codecs, both layouts over TCP, checkpoint identity across restart, fragmented reads, short/interrupted writes, final write/flush failures, changed stream length, library tests, Star replication through metadata failover and full process restart, zero-filled WAL storage, paired release measurements at 1/16/64 MiB per replica, feature combinations, documentation, and the CI test-name inventory.

The segmented layout remains opt-in, retains old segments, and still rewrites its complete metadata catalog. Buffering reduces default checkpoint syscall overhead; it does not remove full-corpus payload rewriting. Flat append latency, incremental metadata checkpoints, and bounded storage retention remain future work.

@matrixarkai matrixarkai changed the title Expose replica checkpoint progress and measure completed I/O Expose checkpoint progress and buffer replica checkpoint writes Nov 18, 2026
@matrixarkai
matrixarkai merged commit e726472 into main Nov 18, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants