Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -605,13 +605,34 @@ jobs:
compact_owned_append_keeps_publication_bounded_and_recovers_all_replicas \
owned_append_keeps_data_writes_bounded_and_recovers_all_replicas \
star_corpus_scale::star_corpus_workload_verifies_durable_reads_writes_and_appends \
star_corpus_scale::checkpoint_corpus_measurement::checkpoint_corpus_measurement_drains_both_layouts_and_recovers \
star_corpus_scale::checkpoint_interference::checkpoint_interference_preserves_three_durable_append_copies \
star_recovery_profiles::star_repair_and_append_work_across_local_recovery_profiles \
segmented_star_replicas_keep_objects_through_metadata_failover_and_full_restart
do
grep -Fxq "test star_append_amplification::${test_name} ... ok" /tmp/star-append-cost.log
done

- name: checkpoint buffering preserves bytes and propagates final flush failures
run: |
set -o pipefail
cargo test --locked --all-features --lib journal_checkpoint_buffer_ -- --test-threads=1 | tee /tmp/checkpoint-buffer.log
grep -q '^test result: ok. 3 passed; 0 failed;' /tmp/checkpoint-buffer.log
for test_name in journal_checkpoint_buffer_preserves_fragmented_bytes_and_checksum journal_checkpoint_buffer_final_write_and_flush_errors_propagate journal_checkpoint_buffer_rejects_a_changed_stream_length
do
grep -Fq "${test_name} ... ok" /tmp/checkpoint-buffer.log
done

- name: standalone journal control validates bounded progress and publication
run: |
set -o pipefail
cargo test --locked --all-features --lib journal_control_ -- --test-threads=1 | tee /tmp/journal-control.log
grep -q '^test result: ok. 3 passed; 0 failed;' /tmp/journal-control.log
for test_name in journal_control_codecs_are_canonical_bounded_and_versioned journal_control_tcp_checkpoint_reports_published_epoch_and_recovers_tail journal_control_memory_refusal_does_not_fence_ordinary_requests
do
grep -Fq "${test_name} ... ok" /tmp/journal-control.log
done

- name: journaled append decisions preserve binary outcomes and recovery
run: |
set -o pipefail
Expand Down
81 changes: 81 additions & 0 deletions docs/replica-segmented-journal.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,3 +94,84 @@ The process test starts three metadata nodes and three chunk servers, confirms
six distinct live PIDs with their own storage paths and established sockets, writes
objects through Star replication, changes metadata leadership, appends data, kills
and restarts all six processes, and reads every object from every data replica.

## Standalone checkpoint control

`RemoteReplicaClient::journal_status()` reports bounded journal progress without
walking pages or reading payloads. `checkpoint_journal()` synchronizes preceding
mutations and starts a checkpoint. Both commands address the connected standalone
replica directly; they are refused on memory-only nodes and Raft groups. A busy
checkpoint is refused without fencing an otherwise healthy node. I/O failures
fence the node, including failures observed while polling background publication.

The canonical binary protobuf control request is bounded to 32 bytes and its
status reply to 256 bytes. Unknown actions, versions, duplicate fields, unknown
fields, and noncanonical encodings are refused. Status includes a 16-byte journal
identity, layout, started and published epochs, in-flight state, queued record
count, uncheckpointed logical bytes, committed journal bytes, and segment count.
Committed journal bytes exclude preallocated space and ancillary files; they are
not a measurement of storage I/O. Legacy epochs and identity last until reopen;
segmented identity and committed epochs survive restart.

Quiesce writers, wait for any current checkpoint, request a checkpoint, then poll
`status.checkpoint_is_published(&requested)` and `!status.checkpoint_in_flight`.
The helper refuses a changed journal identity or layout. A quiescent completed
barrier must also report zero pending records and zero checkpoint debt. A status
observation can reap a completed worker but does not append journal records.

## Corpus I/O measurement

The paired local WSL measurement uses three metadata processes and three data
processes, with three acknowledged copies and one-copy reads. Its timed requests
use the data replica API directly; metadata publication QPS is outside that scope.
Every phase drains warmup checkpoints before sampling data-node `/proc/PID/io`.
Writes and appends then include completed automatic checkpoints plus a final
explicit checkpoint. Request QPS and latency describe acknowledgment; cycle QPS
and CPU include checkpoint drain. Read phases do not create extra checkpoints.

The output includes kernel write bytes, cancelled write bytes, their signed
difference, kernel read bytes, and all-file-descriptor write characters/calls.
The latter include sockets. These Linux process counters are not Windows device
I/O or allocation counts. All live payloads remain cached by the service. The
1 MiB checkpoint threshold is a stress configuration, not a default change.
One paired run is diagnostic evidence, not a sustained throughput guarantee.

```sh
cargo test --locked --all-features --lib -j1 journal_control_ -- --test-threads=1
cargo test --locked --all-features --test network_metadata_consensus -j1 checkpoint_corpus_measurement_drains_both_layouts_and_recovers -- --test-threads=1
cargo test --locked --release --all-features --test network_metadata_consensus -j1 measure_checkpoint_io_at_growing_corpora -- --ignored --nocapture --test-threads=1
```

The functional case covers both layouts and full process restart. The explicit
release measurement pairs 1, 16, and 64 MiB background payloads per data node,
16 clients, and 512 requests per phase, using the same foreground workload and
rotating layout order. Every read checks its complete payload; retained first
and last written pages and cold pages are checked on every replica after restart.

One release run at commit `aae965b1` in local WSL produced these append results. Each phase appends
2 MiB of logical data, with three acknowledged copies. Kernel writes sum all
three data processes through the completed final checkpoint.

| Background payload per replica | Layout | Append p99 | Cycle QPS | Kernel write bytes / logical byte |
| --- | --- | ---: | ---: | ---: |
| 1 MiB | legacy | 105 ms | 320 | 27.68 |
| 1 MiB | segmented | 58 ms | 396 | 5.95 |
| 16 MiB | legacy | 688 ms | 116 | 95.77 |
| 16 MiB | segmented | 67 ms | 373 | 8.40 |
| 64 MiB | legacy | 1919 ms | 58 | 313.57 |
| 64 MiB | segmented | 157 ms | 193 | 16.47 |

These are short stress phases on one machine, with layout order rotated by corpus
size and no cross-host validation. Segmented latency and amplification still grow;
this run does not establish flat latency or a production throughput guarantee.
All cached read phases recorded zero kernel write bytes. Safe reclamation and an
incremental metadata checkpoint remain necessary before default activation.

## Legacy checkpoint stream buffering

The legacy checkpoint writer coalesces fragmented encoder reads in a fixed 64 KiB
output buffer. It flushes that buffer before rewriting the checkpoint header,
synchronizing the file, and atomically publishing it. Flush errors propagate to
the replica's existing failure fence. This changes neither stored bytes nor the
full-corpus payload rewrite; it targets small-write syscall overhead. The table
above predates this buffering change.
8 changes: 7 additions & 1 deletion src/cluster/chunk_consensus.rs
Original file line number Diff line number Diff line change
Expand Up @@ -392,6 +392,12 @@ impl ChunkConsensus {
self.failed.load(Ordering::SeqCst)
}
pub(super) fn request(&self, request: ReplicaRequest) -> Result<ReplicaReply, ObjectError> {
if matches!(request, ReplicaRequest::JournalControl(_)) {
return Err(ObjectError::PreconditionFailed(
"journal control requires standalone replica storage".into(),
));
}

let request = match request {
ReplicaRequest::WriteChunkBatchIfSequence { writes } => {
return self.request_batch(writes)
Expand Down Expand Up @@ -1003,7 +1009,7 @@ fn restore_snapshot(
fn read_only(request: &ReplicaRequest) -> bool {
match request {
ReplicaRequest::ReadNamespaceFence { .. } => true,
ReplicaRequest::FenceNamespace(_) => false,
ReplicaRequest::FenceNamespace(_) | ReplicaRequest::JournalControl(_) => false,
ReplicaRequest::Membership(command) => command.is_read(),
ReplicaRequest::ReadChunk { .. }
| ReplicaRequest::ReadChunkBatch { .. }
Expand Down
1 change: 1 addition & 0 deletions src/cluster/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,7 @@ mod replica_image;
mod replica_journal;
#[cfg(feature = "replica-server")]
mod replica_journal_backend;
pub mod replica_journal_control;
#[cfg(feature = "replica-server")]
mod replica_segmented_journal;

Expand Down
110 changes: 110 additions & 0 deletions src/cluster/replica_checkpoint_stream_tests.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
// SPDX-License-Identifier: Apache-2.0
// Copyright 2026 MatrixArkAI
use super::*;

struct Fragmented<'a> {
bytes: &'a [u8],
}
impl Read for Fragmented<'_> {
fn read(&mut self, output: &mut [u8]) -> io::Result<usize> {
let count = output.len().min(31).min(self.bytes.len());
output[..count].copy_from_slice(&self.bytes[..count]);
self.bytes = &self.bytes[count..];
Ok(count)
}
}
#[derive(Default)]
struct Sink {
bytes: Vec<u8>,
calls: usize,
short: bool,
fail_write: bool,
fail_flush: bool,
interrupted: bool,
}
impl Write for Sink {
fn write(&mut self, bytes: &[u8]) -> io::Result<usize> {
self.calls += 1;
if self.fail_write {
return Err(io::Error::other("checkpoint write failure"));
}
if self.interrupted {
self.interrupted = false;
return Err(io::ErrorKind::Interrupted.into());
}
let count = if self.short {
bytes.len().min(7)
} else {
bytes.len()
};
self.bytes.extend_from_slice(&bytes[..count]);
Ok(count)
}
fn flush(&mut self) -> io::Result<()> {
if self.fail_flush {
Err(io::Error::other("checkpoint flush failure"))
} else {
Ok(())
}
}
}
#[test]
fn journal_checkpoint_buffer_preserves_fragmented_bytes_and_checksum() {
let bytes = (0..1024 * 1024 + 17)
.map(|n| (n % 251) as u8)
.collect::<Vec<_>>();
for short in [false, true] {
let mut sink = Sink {
short,
interrupted: true,
..Default::default()
};
let checksum = write_checkpoint_stream(
&mut Fragmented { bytes: &bytes },
bytes.len() as u64,
&mut sink,
)
.unwrap();
assert_eq!(sink.bytes, bytes);
assert_eq!(checksum, fnv1a64(&bytes));
if !short {
assert!(
sink.calls < 32,
"fragmented writes were not coalesced: {}",
sink.calls
);
}
}
}
#[test]
fn journal_checkpoint_buffer_final_write_and_flush_errors_propagate() {
// Everything fits in the buffer: only an explicit final flush exposes these errors.
let bytes = [0x5a; 1024];
for fail_write in [false, true] {
let mut sink = Sink {
fail_write,
fail_flush: !fail_write,
..Default::default()
};
let error = write_checkpoint_stream(&mut bytes.as_slice(), bytes.len() as u64, &mut sink)
.unwrap_err();
assert_eq!(error.kind(), io::ErrorKind::Other);
assert_eq!(
error.to_string(),
if fail_write {
"checkpoint write failure"
} else {
"checkpoint flush failure"
}
);
}
}
#[test]
fn journal_checkpoint_buffer_rejects_a_changed_stream_length() {
for expected in [3, 5] {
let mut sink = Sink::default();
let error =
write_checkpoint_stream(&mut b"data".as_slice(), expected, &mut sink).unwrap_err();
assert_eq!(error.kind(), io::ErrorKind::InvalidData);
}
}
61 changes: 56 additions & 5 deletions src/cluster/replica_journal.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ const MAGIC_ZERO_TAIL: &[u8; 4] = b"MOZ2";
const FRAME_END: u8 = 0xA5;

pub(super) struct ReplicaJournal {
journal_id: [u8; 16],
checkpoint_epoch: u64,
file: File,
// Stable across atomic journal replacement; retain the journal lock too.
_lock: Option<File>,
Expand Down Expand Up @@ -107,7 +109,25 @@ fn outcome_header(payload: &[u8], chain: bool) -> io::Result<[u8; HEADER_LEN]> {
Ok(header)
}

fn new_journal_id() -> io::Result<[u8; 16]> {
let mut id = [0; 16];
ring::rand::SecureRandom::fill(&ring::rand::SystemRandom::new(), &mut id)
.map_err(|_| invalid("journal identity generation failed"))?;
Ok(id)
}

impl ReplicaJournal {
pub(super) fn journal_id(&self) -> &[u8] {
&self.journal_id
}
pub(super) fn checkpoint_epoch(&self) -> u64 {
self.checkpoint_epoch
}

pub(super) fn checkpoint_bytes(&self) -> u64 {
self.bytes_since_checkpoint
}

#[cfg(test)]
pub(super) fn open(path: &Path, service: &mut MatrixReplicaService) -> io::Result<Self> {
Self::open_with_limits(path, service, usize::MAX)
Expand Down Expand Up @@ -318,6 +338,8 @@ impl ReplicaJournal {
file.seek(SeekFrom::End(0))?;
}
let mut journal = Self {
journal_id: new_journal_id()?,
checkpoint_epoch: 0,
file,
_lock: Some(lock),
path,
Expand Down Expand Up @@ -607,6 +629,11 @@ impl ReplicaJournal {

#[cfg(unix)]
pub(super) fn checkpoint(&mut self, service: &MatrixReplicaService) -> io::Result<()> {
let epoch = self
.checkpoint_epoch
.checked_add(1)
.ok_or_else(|| invalid("checkpoint epoch overflow"))?;

let mut stream = super::replica_image::capture(service.export_snapshot(), (1, 1))
.map_err(invalid)?
.ok_or_else(|| invalid("replica checkpoint exceeds stream bounds"))?;
Expand All @@ -620,11 +647,7 @@ impl ReplicaJournal {
.open(&temporary)?;
fs2::FileExt::try_lock_exclusive(&replacement)?;
replacement.write_all(&[0; BASE_HEADER_LEN])?;
let mut output = Digest::new(&mut replacement);
if io::copy(&mut stream, &mut output)? != length {
return Err(invalid("replica checkpoint stream length changed"));
}
let checksum = output.hash;
let checksum = write_checkpoint_stream(&mut stream, length, &mut replacement)?;
replacement.seek(SeekFrom::Start(0))?;
replacement.write_all(&base_header(length, checksum))?;
replacement.sync_all()?;
Expand All @@ -639,6 +662,7 @@ impl ReplicaJournal {
self.bytes_since_checkpoint = 0;
File::open(self.path.parent().unwrap())?.sync_all()?;
self.mark_initialized()?;
self.checkpoint_epoch = epoch;
Ok(())
}

Expand All @@ -651,6 +675,31 @@ impl ReplicaJournal {
}
}

// The encoder may yield individual metadata fields as short reads. Coalesce
// those fragments without collecting the checkpoint, and flush before its header
// is rewritten or its file is synchronized and published.
#[cfg(unix)]
fn write_checkpoint_stream(
stream: &mut impl Read,
length: u64,
target: &mut impl Write,
) -> io::Result<u64> {
let mut buffered = std::io::BufWriter::with_capacity(64 * 1024, target);
let checksum = {
let mut output = Digest::new(&mut buffered);
if io::copy(stream, &mut output)? != length {
return Err(invalid("replica checkpoint stream length changed"));
}
output.hash
};
buffered.flush()?;
Ok(checksum)
}

#[cfg(all(test, unix))]
#[path = "replica_checkpoint_stream_tests.rs"]
mod checkpoint_stream_tests;

const BASE_HEADER_LEN: usize = 32;
fn sibling(path: &Path, suffix: &str) -> PathBuf {
let mut name = path.as_os_str().to_owned();
Expand Down Expand Up @@ -754,6 +803,8 @@ impl<T: Write> Write for Digest<T> {
impl ReplicaJournal {
pub(super) fn read_only_for_test(path: &Path) -> Self {
Self {
journal_id: new_journal_id().unwrap(),
checkpoint_epoch: 0,
file: File::open(path).unwrap(),
_lock: None,
path: path.to_path_buf(),
Expand Down
Loading
Loading