Skip to content

feat(database): Db.stream streams rows with decoding - #94

Merged
Makisuo merged 5 commits into
MapleTechLabs:mainfrom
artile:feat/db-stream
Nov 2, 2026
Merged

Makisuo merged 5 commits into
MapleTechLabs:mainfrom
artile:feat/db-stream

Conversation

@artile

@artile artile commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Closes #70

@coderabbitai

coderabbitai Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 49 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 215d0ea0-7f13-4b32-bdbd-cbea61dc0f00
Commits

Reviewing files that changed from the base of the PR and between b57faf1 and 2b3b146.

Files selected for processing: 12
  • .changeset/db-stream.md
  • docs/database.md
  • docs/reference.md
  • src/database.ts
  • src/database/database.test-d.ts
  • src/database/database.ts
  • src/database/stream.test.ts
  • src/database/tracing.ts
  • src/sqlite/d1.test.ts
  • src/sqlite/sqlite.test.ts
  • tests/clickhouse-http.clickhouse.test.ts
  • tests/database.clickhouse.test.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@artile
artile marked this pull request as ready for review October 25, 2026 15:01
Makisuo and others added 2 commits November 2, 2026 16:33
Resolves the database.ts, tracing.ts, database.test-d.ts and reference.md
conflicts by keeping stream beside afterCommit. Effect 4.0.6 no longer names
its statement span sql.execute; the stream span test finds it by attribute,
as tracing.test.ts on main does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Effect's node:sqlite and D1 drivers implement executeStream as
Stream.die("executeStream not implemented"), so Db.stream was a defect on
sqliteDialect and d1Dialect. A SQLite-family dialect now reads the whole
result as run does and emits it row by row. Adds node:sqlite, D1 (Miniflare)
and live ClickHouse (Effect's client and ClickhouseHttpClient) stream tests,
and an afterCommit test beside a stream in a transaction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Makisuo

Makisuo commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Merge-readiness pass (2b3b146):

  • Merged main (45ad2c8). The conflicts were in database.ts (imports), tracing.ts (the header comment from main, plus Stream for traceStream), database.test-d.ts and docs/reference.md. The resolution keeps stream alongside afterCommit. Effect 4.0.6 no longer names its statement span sql.execute, so the span test now finds that span by its effect.sql.method attribute, as tracing.test.ts on main does.
  • Fix for SQLite/D1. @effect/sql-sqlite-node and @effect/sql-d1 implement executeStream as Stream.die("executeStream not implemented"), so Db.stream was a defect on sqliteDialect and d1Dialect. A SQLite-family dialect (functions === "sqlite") now reads the whole result the way run does, with the same span, guard and tenant check, and emits it row by row. The docs and changeset say so.
  • New tests:
    • node:sqlite streaming, with integer booleans decoded.
    • D1 streaming on Miniflare, where the tenant guard still refuses an unscoped stream.
    • Live ClickHouse Db.stream through Effect's client and through ClickhouseHttpClient: UInt64 is decoded per row, and Stream.take stops an unbounded system.numbers scan.
    • An afterCommit hook queued next to a stream inside transaction runs only after the commit.

Verified locally:

  • bun run build and bun run typecheck are clean.
  • bun run test against live ClickHouse 26.8 with PGlite, node:sqlite and Miniflare D1, including fuzz at FUZZ_SCALE=0.3, passed 74 files and 1442 tests. The doc citation, export and doc example checks also pass.
  • bun run test:package passes.

CI run 41472824161 did not need approval. If it fails only on the known fuzz flakes, they are fixed separately in #131: ClickHouse/Postgres division by zero in generated fixtures …, or compiled row codec retains generated values.

Open design choice for the maintainer: on Postgres, Effect's pg driver streams with a cursor, but PGlite buffers. On SQLite/D1 the stream is always buffered, so stream there gives the API shape without the memory benefit.

@Makisuo
Makisuo merged commit 1e13356 into MapleTechLabs:main Nov 2, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Db.stream: stream rows with decoding

2 participants