Skip to content

The cluster feature set is metadata state, carried by the snapshot - #681

Merged
matrixarkai merged 3 commits into
mainfrom
the-cluster-feature-set-is-metadata-state
Nov 18, 2026
Merged

matrixarkai merged 3 commits into
mainfrom
the-cluster-feature-set-is-metadata-state

Conversation

@matrixarkai

Copy link
Copy Markdown
Owner

Reversible: adds a field that is empty everywhere today. A snapshot with no feature set is byte for byte what it was; nothing sets the field yet, so no node writes the new snapshot version. Rolling back this build leaves every existing file readable. (The part that raises a feature, 4a-2, is separate and will state exactly what an old node does with it.)

Why

Compact append (outcome v11), reserve-next (v13) and the chunk-granular manifest (format 3) each need every metadata voter and chunkserver to read a new shape before anything emits it. Today the only protection is a documented rule ("upgrade voters before gateways"); a skewed pair is not detected, and an old voter that meets a shape it cannot read fails to apply the entry rather than skipping it. This is part 4a-1 of one capability mechanism shared by all of them and by the gateway recovery records.

What

  • src/cluster_features.rs: ClusterFeatures { raised, declared }, the feature bits (compact append, reserve-next, chunk manifest, a reserved byte for the gateway recovery records), has, common_support, bounds (at most 1024 declared voters, no voter id 0).
    • declared: voter id to the bits that voter reads, to be written by the voter itself through the log (4a-2).
    • raised: the bits the group may emit, to be changed only by a committed command that the leader accepts only while every declared voter supports them (4a-2).
  • The state lives in the metadata snapshot (MetaSnapshot and MatrixMetaService), so it is exported, installed (a late-joining voter gets it with the snapshot), checkpointed and restarted with everything else.
  • Encoding: version 21 is written only when the set is not empty. It has the same roster shape as version 20, a presence flag for the manifest catalog (so an empty catalog is representable), and the set before the checksum. An empty set keeps the version it had (18, 19 or 20) and the same bytes. The checksum covers the set only when it is not empty, so no existing checksum changes.
  • Readers: this build reads 1 through 21 (including the streamed reader). A build without this change refuses version 21 at its magic check, so an old node never half-reads a snapshot that carries a set.

Tests (src/cluster_features_tests.rs, 7)

Empty set keeps the old magic and bytes and an old snapshot reads as empty; round trip through both readers (bytes and streamed); a late joiner installs the snapshot and holds the set, and a restart keeps it; a declaration alone is state; every damaged byte near the set is refused and truncations fail; each of the raised bits and a declaration changes the checksum; an invalid table (voter 0, too many voters) is refused; has and common_support.
Planted defects, each caught (undone by edit, tree clean after): the encoder drops the set, the checksum ignores the set, install does not copy the set, the decoder reads the set as empty, validation skipped.

Numbers

No data path changed. Snapshot size with an empty set: unchanged (0 bytes). With a set: 8 bytes (raised) + 8 (count) + 16 per declared voter + 8 (catalog flag), once per snapshot.

🤖 Generated with Claude Code

superhaiou and others added 3 commits November 17, 2026 21:25
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…t guards

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@matrixarkai
matrixarkai merged commit 104c9c2 into main Nov 18, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants