Repository navigation
MOD-19328 Shared strings in CBOR documents - #25
Conversation
|
One correctness issue to fix before approval: cache reset breaks lookups of retained object keys with The new thread-safe reset wiring clears the intern table while existing strings remain alive. Interning the same text afterwards creates a different allocation, but Reproduced on the current head: use ijson::{IObject, IString, IValue};
#[test]
fn retained_object_keys_still_match_after_cache_reset() {
ijson::init_shared_string_cache(true).unwrap();
let key = IString::from("retained object key");
let mut object = IObject::new();
object.insert(key.clone(), 42).unwrap();
ijson::reinit_shared_string_cache();
let replacement = IString::from("retained object key");
assert_eq!(object.get(&replacement), Some(&IValue::from(42)));
}With Please preserve content-based key/value semantics across cache resets and add regression coverage for both retained-object lookup and the encode/decode case. Equality, ordering, and hashing need to remain consistent, including for The existing suites passed locally with both feature configurations. The current cache-reset test checks replacement lifetime/drop behavior, but does not exercise these lookup and equality cases. This finding is separate from the shared-string wire-format design. |
|
Follow-up to my cache-reset finding: the correctness blocker is addressed by a5f374f. I reviewed the change and verified that:
Local validation on that exact commit:
The GitHub CI run now also reports a successful build/test job and all five fuzz jobs passing. I have no remaining objection from this cache-reset finding. RedisJSON/JSON-HDT integration validation remains separate from this library fix. |
gabsow
left a comment
There was a problem hiding this comment.
Approved at a5f374f. The cache-reset equality/hash blocker is fixed and the added regressions cover retained-key lookup, round-trip equality, ordering, and hash-set lookup. Local tests passed with default features (181) and thread_safe (176), including documentation tests; formatting passed. The build/test job and all five fuzz jobs are green.
RedisJSON/JSON-HDT integration and Enterprise validation remain tracked separately on the RedisJSON PR.
Independent review at
|
Shared strings in CBOR documents
Adds an opt-in stream form of the CBOR encoding in which a string that several
values share is written once and referred to by id afterwards.
StreamEncoderwrites a heap string of 8 bytes or more with at least two owners as adefinition (CBOR tag 6). Later occurrences are references (tag 7) by implicit,
sequential id. It keys on the address of the shared allocation, never on the bytes,
so its memory grows with the number of distinct strings.
AddressDictionaryis thedefault and
ContentDictionarysuits callers whose strings do not outlive the call.DocUsereports how many definitions and references one document used.StreamDecoderreads the documents of a stream in order and rebuilds the table. Itdecodes tags and object keys straight from the CBOR reader, without a
Value.written once, with no other record needed to read it.
encode_intoappends to a buffer.StreamStringslets an external manager'sserializer use the same dictionary.
encodeanddecodekeep their format and no public item is removed.Also in this PR
fix:a plain CBOR array was accepted as an object key in the stream decoder.fix:IValue::new_ptrdid pointer arithmetic on a null pointer for inline integers(found by the new fuzz target).
--features thread_safetest step and the new fuzz target.Testing
cargo test -- --test-threads=1passes with default features and--features thread_safe(the lib tests share global state).
fuzz_cbor_stream_decode,fuzz_cbor_roundtripandfuzz_cbor_decoderan a few million executions each with no crash.Note
Medium Risk
Rewrites the default CBOR encode/decode pipeline and changes IString equality/hash semantics, though compatibility tests and an unchanged standalone format mitigate regression risk; the new stream format adds ordering and lifetime constraints callers must honor.
Overview
Adds an opt-in CBOR string stream for ordered multi-document payloads:
StreamEncoder/StreamDecoderdedupe heap strings (≥8 bytes, ≥2 owners by default) using CBOR tag 6 definitions and tag 7 references, with table budgets, poison/reset semantics,encode_with/StreamStrings, and pluggableAddressDictionaryvsContentDictionary. Standaloneencode/decodestay the default wire format;encode_intoappends plain CBOR.The CBOR path is rewired through Serde (no intermediate
Valuetree on the hot path) so stream logic can compile out of plain encoding and memory use stays bounded; stream decode handles tags and keys directly from the reader.Supporting changes:
IString::heap_identityfor address-based sharing, equality/hash/order that treat same text across cache resets consistently, safer string-cache drops afterreinit_shared_string_cache, andIValue::new_ptrusingwrapping_addfor inline integers.CI/docs: README stream section,
fuzz_cbor_stream_decode,thread_safetest job, rust-analyzer in toolchain.Reviewed by Cursor Bugbot for commit a51ca4c. Bugbot is set up for automated code reviews on this repo. Configure here.