Skip to content

MOD-19328 Shared strings in CBOR documents - #25

Merged
AvivDavid23 merged 10 commits into
masterfrom
rdb_strings_firstuse
Nov 2, 2026
Merged

AvivDavid23 merged 10 commits into
masterfrom
rdb_strings_firstuse

Conversation

@AvivDavid23

@AvivDavid23 AvivDavid23 commented Oct 25, 2026 •

Copy link
Copy Markdown

Shared strings in CBOR documents

Adds an opt-in stream form of the CBOR encoding in which a string that several
values share is written once and referred to by id afterwards.

  • StreamEncoder writes a heap string of 8 bytes or more with at least two owners as a
    definition (CBOR tag 6). Later occurrences are references (tag 7) by implicit,
    sequential id. It keys on the address of the shared allocation, never on the bytes,
    so its memory grows with the number of distinct strings. AddressDictionary is the
    default and ContentDictionary suits callers whose strings do not outlive the call.
    DocUse reports how many definitions and references one document used.
  • StreamDecoder reads the documents of a stream in order and rebuilds the table. It
    decodes tags and object keys straight from the CBOR reader, without a Value.
  • A document can also be encoded on its own, so that a string that repeats inside it is
    written once, with no other record needed to read it.
  • encode_into appends to a buffer. StreamStrings lets an external manager's
    serializer use the same dictionary.
  • Existing encode and decode keep their format and no public item is removed.

Also in this PR

  • fix: a plain CBOR array was accepted as an object key in the stream decoder.
  • fix: IValue::new_ptr did pointer arithmetic on a null pointer for inline integers
    (found by the new fuzz target).
  • CI: a --features thread_safe test step and the new fuzz target.

Testing

cargo test -- --test-threads=1 passes with default features and --features thread_safe
(the lib tests share global state). fuzz_cbor_stream_decode, fuzz_cbor_roundtrip and
fuzz_cbor_decode ran a few million executions each with no crash.


Note

Medium Risk
Rewrites the default CBOR encode/decode pipeline and changes IString equality/hash semantics, though compatibility tests and an unchanged standalone format mitigate regression risk; the new stream format adds ordering and lifetime constraints callers must honor.

Overview
Adds an opt-in CBOR string stream for ordered multi-document payloads: StreamEncoder / StreamDecoder dedupe heap strings (≥8 bytes, ≥2 owners by default) using CBOR tag 6 definitions and tag 7 references, with table budgets, poison/reset semantics, encode_with / StreamStrings, and pluggable AddressDictionary vs ContentDictionary. Standalone encode / decode stay the default wire format; encode_into appends plain CBOR.

The CBOR path is rewired through Serde (no intermediate Value tree on the hot path) so stream logic can compile out of plain encoding and memory use stays bounded; stream decode handles tags and keys directly from the reader.

Supporting changes: IString::heap_identity for address-based sharing, equality/hash/order that treat same text across cache resets consistently, safer string-cache drops after reinit_shared_string_cache, and IValue::new_ptr using wrapping_add for inline integers.

CI/docs: README stream section, fuzz_cbor_stream_decode, thread_safe test job, rust-analyzer in toolchain.

Reviewed by Cursor Bugbot for commit a51ca4c. Bugbot is set up for automated code reviews on this repo. Configure here.

gabsow commented Oct 26, 2026

Copy link
Copy Markdown

One correctness issue to fix before approval: cache reset breaks lookups of retained object keys with thread_safe enabled.

The new thread-safe reset wiring clears the intern table while existing strings remain alive. Interning the same text afterwards creates a different allocation, but IString::eq still compares allocation identity. This makes an existing object miss a lookup for identical key text.

Reproduced on the current head:

use ijson::{IObject, IString, IValue};

#[test]
fn retained_object_keys_still_match_after_cache_reset() {
    ijson::init_shared_string_cache(true).unwrap();
    let key = IString::from("retained object key");
    let mut object = IObject::new();
    object.insert(key.clone(), 42).unwrap();

    ijson::reinit_shared_string_cache();

    let replacement = IString::from("retained object key");
    assert_eq!(object.get(&replacement), Some(&IValue::from(42)));
}

With --features thread_safe, the assertion gets None instead of Some(42). The same test passes with default features. A second check that encodes a retained string, resets the cache, and decodes it also fails equality under thread_safe, despite identical text.

Please preserve content-based key/value semantics across cache resets and add regression coverage for both retained-object lookup and the encode/decode case. Equality, ordering, and hashing need to remain consistent, including for IValue; changing IString::Hash alone does not address the pointer-only equality.

The existing suites passed locally with both feature configurations. The current cache-reset test checks replacement lifetime/drop behavior, but does not exercise these lookup and equality cases. This finding is separate from the shared-string wire-format design.

@AvivDavid23
AvivDavid23 requested a review from gabsow October 26, 2026 11:33

gabsow commented Oct 26, 2026

Copy link
Copy Markdown

Follow-up to my cache-reset finding: the correctness blocker is addressed by a5f374f.

I reviewed the change and verified that:

  • IString retains the pointer-equality fast path and compares text when allocations differ after reset.
  • String-valued IValue hashes use the same content-based semantics as equality.
  • The new regressions cover retained-object lookup, encode/decode across reset, equality/ordering/hash consistency, and hash-set lookup.

Local validation on that exact commit:

  • cargo test --locked -- --test-threads=1: 181 passed, including documentation tests.
  • cargo test --locked --features thread_safe -- --test-threads=1: 176 passed, including documentation tests.
  • cargo fmt -- --check: passed.

The GitHub CI run now also reports a successful build/test job and all five fuzz jobs passing. I have no remaining objection from this cache-reset finding. RedisJSON/JSON-HDT integration validation remains separate from this library fix.

@gabsow gabsow left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at a5f374f. The cache-reset equality/hash blocker is fixed and the added regressions cover retained-key lookup, round-trip equality, ordering, and hash-set lookup. Local tests passed with default features (181) and thread_safe (176), including documentation tests; formatting passed. The build/test job and all five fuzz jobs are green.

RedisJSON/JSON-HDT integration and Enterprise validation remain tracked separately on the RedisJSON PR.

@gabsow

gabsow commented Oct 26, 2026

Copy link
Copy Markdown

Independent review at a5f374fd3f996e7775ca216626a6f7ffc9dc864e: approval stands

The cache-reset fix holds: equality, ordering and hashing agree, and the suites pass with and without thread_safe. The points below are optional follow-ups; none blocks this PR.

  • Address reuse. AddressDictionary cannot detect a freed string whose address is reused by a
    string of the same length within one stream (stream.rs:94-99, 289-303). A system-allocator
    test produced silently wrong output in 192 of 200 rounds. RedisJSON's current callers are sound:
    fork child, blocking SAVE, and a fresh encoder per DUMP. However, AddressDictionary is also the
    default for external save_with managers. Consider a cheap fingerprint check on a hit, or make
    the borrowing contract unsafe.
  • Committed id count after a failed document. next_id() and stats().defs count definitions
    from a failed document that the reader never saw. RedisJSON is safe, because a poisoned encoder
    only writes marker 0. Exposing the committed id count would make the README's framing rule safe
    for other users of the library.
  • Futile reset. ContentDictionary requests a reset when one string is larger than the entire
    budget, which wipes the useful entries for nothing. Only request it if the string would fit in an
    empty table.
  • Unbounded decode depth. decode(_, false) has no recursion limit. This is pre-existing
    behavior, but RedisJSON passes false by default (json-legacy-rdb-depth-limit no), including
    for RESTORE payloads. Consider a finite ceiling in both modes.
  • Cost of the equality change. It costs something only under thread_safe: a missed lookup
    takes about 2× longer in an 8-key object. Default builds already fell back to comparing text at
    base.

@AvivDavid23
AvivDavid23 merged commit 6484ffa into master Nov 2, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants