Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/actions/build_and_test/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,9 @@ runs:
command: test
args: -- --test-threads=1

- name: Test (thread_safe)
uses: actions-rs/cargo@v1
with:
command: test
args: --features thread_safe -- --test-threads=1

1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ jobs:
- fuzz_json_de
- fuzz_cbor_decode
- fuzz_cbor_roundtrip
- fuzz_cbor_stream_decode
- fuzz_object_ops
steps:
- name: Checkout repository
Expand Down
127 changes: 127 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,133 @@ case that is important in the original JSON.

Removing from an `IObject` will disrupt the insertion order.

### String streams

`StreamEncoder` shares strings across the documents of one ordered stream. A
string with at least two owners and at least 8 bytes is
written inline once, wrapped in CBOR tag 6 (a definition), and every later
occurrence is written as tag 7 plus an ID (a reference). IDs are implicit: the
n-th definition of the stream is ID n, so `StreamDecoder` rebuilds the table by
decoding the documents in the order they were encoded. A reference costs 2, 3, 4
or 6 bytes for IDs below 24, 256, 65,536 and 2^32. Typed numeric arrays keep their
existing CBOR tags, and `encode`/`decode` and the compressed APIs keep their
standalone formats.

```rust
use ijson::{IValue, StreamDecoder, StreamEncoder, StreamEncoderConfig};

let mut encoder = StreamEncoder::new(StreamEncoderConfig {
table_budget_bytes: 64 << 20,
});
let first: IValue = serde_json::from_str(r#"{"k":"a string stored in two documents"}"#).unwrap();
let second: IValue = serde_json::from_str(r#"{"k":"a string stored in two documents"}"#).unwrap();

let (mut first_bytes, mut second_bytes) = (Vec::new(), Vec::new());
encoder.encode_into(&mut first_bytes, &first); // defines the string
encoder.encode_into(&mut second_bytes, &second); // references it

let mut decoder = StreamDecoder::new();
assert_eq!(decoder.decode(&first_bytes, true).unwrap(), first);
assert_eq!(decoder.decode(&second_bytes, true).unwrap(), second);
```

The encoder recognizes a string by the address of its shared allocation, so it
neither hashes nor copies string bytes and only reads them. The caller must keep
all defined string allocations alive and at the same address until the stream
ends. The encoder holds no owned handles. A freed string whose address another
string of the same length takes over goes unnoticed, and the reader gets the old
text; only a string of another length at a registered address is noticed, which
poisons the encoder: that string and every later one is written inline. The table budget bounds the
encoder's memory: when one more definition
would exceed it, new strings stay inline while earlier ones keep being
referenced. If a document that defined strings is not written, call `poison()`;
every later document is then plain CBOR. `StreamEncoder::next_id` before a
document and `StreamDecoder::len` before reading it must agree, which lets a
framing layer detect skipped, repeated or reordered documents.

To reuse a full table, call `StreamEncoder::reset_if_needed()` between documents
that may define strings. It clears the table only after an eligible definition
missed the table budget, preserving its allocation. Signal that reset in your
framing and call `StreamDecoder::reset()` before decoding the next document
that introduces definitions. Inline documents can precede that marker; they do
not use the dictionary. Both dictionaries then assign IDs from 0 again.

With resets, use cumulative `stats().defs` for framing positions rather than the
current table length, and require a reset-bearing record to introduce a definition.
Otherwise, skipping a reset-only record could silently reuse old IDs. Statistics
stay cumulative across the entire stream. Reader resets release old dictionary handles while loaded
values keep their own. Neither reset API clears a poisoned or failed state.

### Standalone documents

A fresh `StreamEncoder` and a fresh `StreamDecoder` per document make that document stand
alone: ids start at 0, nothing else is needed to read it, and strings that repeat inside the
document are defined once. A string only counts as repeated here if it has two or more owners,
so a document that holds a shared string once also gets a definition that nothing references.
Write the plain bytes when the definitions do not pay: each definition costs 1 byte and
`stats().saved_bytes` is what the references saved, so keep the stream form only when
`saved_bytes > defs`.

```rust
use ijson::{IValue, StreamDecoder, StreamEncoder, StreamEncoderConfig};

let value: IValue = serde_json::from_str(
r#"[{"a repeated object key": 1}, {"a repeated object key": 2}]"#,
).unwrap();
let mut bytes = Vec::new();
let mut encoder = StreamEncoder::new(StreamEncoderConfig { table_budget_bytes: 1 << 20 });
encoder.encode_into(&mut bytes, &value);
let stats = encoder.stats();
assert!(stats.saved_bytes > stats.defs);
assert_eq!(bytes.len() as u64, ijson::encode(&value).len() as u64 - stats.saved_bytes + stats.defs);
assert_eq!(StreamDecoder::new().decode(&bytes, true).unwrap(), value);
```

#### Other value representations

`StreamEncoder<D>` accepts any implementation of `StringDictionary`, defaulting
to `AddressDictionary`. `ContentDictionary` matches ordinary `&str` values by
content and owns copies, so sources can be freed after each document. Its budget
includes copied UTF-8 bytes, table storage and simultaneous growth allocations.
It admits all strings of at least 8 bytes, including strings that occur only once.

```rust
use ijson::{ContentDictionary, StreamDecoder, StreamEncoder, StreamEncoderConfig};

let mut encoder = StreamEncoder::with_dictionary(
StreamEncoderConfig { table_budget_bytes: 4096 },
ContentDictionary::default(),
);
let mut bytes = Vec::new();
encoder.encode_with(&mut bytes, |strings, out| {
ciborium::into_writer(&strings.string("text from another value representation"), out)
}).unwrap();
assert_eq!(
StreamDecoder::new().decode(&bytes, true).unwrap().as_string().unwrap().as_str(),
"text from another value representation",
);
```

A custom Serde serializer wraps each participating string value and object key
with `strings.string(...)`. Wrapping does not register a definition; serialization
does. Serialize one complete CBOR document directly in wire order, without
pre-serializing, discarding or reordering encoded pieces. Propagate errors:
`encode_with` discards appended bytes and poisons the encoder on failure.
Unwrapped strings remain ordinary inline CBOR.

A custom dictionary supplies identity, eligibility, storage and allocation
accounting. Its input is a `StreamString`: text plus an optional interned allocation
identity and owner count. Ordinary string slices have no allocation identity.
A dictionary that borrows its strings, like `AddressDictionary`, needs the caller to keep the
source storage alive; one that owns copies, like `ContentDictionary`, does not.
Choose one dictionary for the stream. Pass a fresh dictionary, or ensure retained
allocations already fit the budget. `Box<dyn StringDictionary>` supports choosing
the implementation at initialization without runtime type checks.

All dictionaries use the same DEF/REF format and existing `StreamDecoder`.
The decoder reconstructs its own `IString` table from definitions; it does not
need the writer's cache or value representation.

## Technical details

### `IValue`
Expand Down
7 changes: 7 additions & 0 deletions fuzz/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,13 @@ test = false
doc = false
bench = false

[[bin]]
name = "fuzz_cbor_stream_decode"
path = "fuzz_targets/fuzz_cbor_stream_decode.rs"
test = false
doc = false
bench = false

[[bin]]
name = "fuzz_object_ops"
path = "fuzz_targets/fuzz_object_ops.rs"
Expand Down
46 changes: 46 additions & 0 deletions fuzz/fuzz_targets/fuzz_cbor_stream_decode.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#![no_main]

use arbitrary::Arbitrary;
use ijson::StreamDecoder;
use libfuzzer_sys::fuzz_target;

// One document of the stream, decoded with or without the nesting limit.
#[derive(Arbitrary, Debug)]
struct Document {
limit_depth: bool,
bytes: Vec<u8>,
}

// A stream document that defines three strings, so that the input can also
// use the references 0 to 2.
fn decoder_with_definitions() -> StreamDecoder {
let mut document = vec![0x83];
for text in ["aaaaaaaa", "bbbbbbbb", "cccccccc"] {
document.extend_from_slice(&[0xC6, 0x68]);
document.extend_from_slice(text.as_bytes());
}
let mut decoder = StreamDecoder::new();
decoder
.decode(&document, true)
.expect("the definitions are a valid stream document");
decoder
}

// Reads the documents in order with one decoder, as a loader does: the table
// grows from document to document, and after a document that failed the decoder
// must refuse every other one.
fuzz_target!(|documents: Vec<Document>| {
let mut decoder = decoder_with_definitions();
let mut failed = false;
for document in documents {
let known = decoder.len();
let result = decoder.decode(&document.bytes, document.limit_depth);
assert!(
!failed || result.is_err(),
"a decoder that failed decoded another document"
);
failed |= result.is_err();
assert!(decoder.len() >= known, "the table shrank");
assert_eq!(decoder.len() as u64, decoder.stats().defs);
}
});
1 change: 1 addition & 0 deletions rust-toolchain.toml
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
[toolchain]
channel = "1.92"
components = ["rust-analyzer"]
Loading
Loading