Skip to content

feat(kernel): expose HTTP/2 opt-out - #1070

Open
aakash-saravanan-db wants to merge 6 commits into
mainfrom
aakash-saravanan-db/feat/http2
Open

aakash-saravanan-db wants to merge 6 commits into
mainfrom
aakash-saravanan-db/feat/http2

Conversation

@aakash-saravanan-db

@aakash-saravanan-db aakash-saravanan-db commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Feature or gap

Direct kernel-owned HTTPS requests negotiate HTTP/2 with HTTP/1.1 fallback. Nonempty HTTP_PROXY, HTTPS_PROXY, ALL_PROXY or lowercase equivalents keep the entire kernel client on HTTP/1.1.

What this PR addresses

Add the kernel-only boolean enable_http2 connection option. Omission preserves the kernel default; False forces HTTP/1.1 for every kernel-owned HTTPS request. Invalid values raise ValueError before native session construction.

Dependencies

Requires the native HTTP/2 support in kernel #483. Requires a kernel build exposing Session(enable_http2=...). The driver and matching kernel ship in the coordinated release.

Tests and validation

Validation status: blocked. Matching kernel integration and the full metadata compatibility matrix remain open.

All 288 kernel-client unit tests pass, including boolean validation through connect() and explicit-false forwarding. Final source-built native transport and DBSQL live full-result checks pass for omitted, true and false settings. A rerun on an additional supported warehouse type timed out; the warehouse was observed in STARTING state afterward. Five-pair connection/query and full 250,000-row result-drain comparisons on DBSQL against Python Thrift meet parity. Full wildcard metadata comparator validation remains incomplete.

This pull request and its description were written by Isaac.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High · 2 Low

Mostly good, with 1 medium concern. Leaving enable_http2 unset keeps working on older wheels, but setting it explicitly on a wheel that lacks the option fails with a vague error instead of a clear NotSupportedError.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.

def test_open_session_preserves_http2_option(monkeypatch, enabled):
captured = {}

def fake_session(**kwargs):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The new tests only check kwarg pass-through to a fake Session, so they can't detect a real kernel that ignores the option or negotiates the wrong protocol. Add a realkernel e2e test that checks HTTP/2 negotiation, the explicit opt-out and the proxy fallback.

🔀 Found by GPT-6.1 Terra · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shared kernel owns the TLS negotiation, opt-out and proxy tests; this adapter owns omitted/true/false forwarding. The matching PyO3 binding now also tests all three mappings in its existing HTTP-config helper. Actual source-built Python transport checks verify negotiated HTTP versions. These cover the distinct boundaries without duplicating the shared TLS suite in this repository.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 21cda69: tests/ only has fake-Session forwarding tests (test_kernel_client.py:411-450). No real-wheel test shows the installed kernel accepts and honors enable_http2.

if value is None:
return {}
if not isinstance(value, bool):
raise ValueError("enable_http2 must be a bool")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — _kernel_http2_kwargs passes an explicit enable_http2 to Session without calling _kernel_session_accepts_kwarg. An older kernel wheel then fails with a TypeError, which is wrapped into a vague OperationalError. Gate it like _kernel_geospatial_kwargs and raise NotSupportedError saying a newer databricks-sql-kernel is required.

🔀 Found by both reviewers · confirmed by the judge

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This driver release requires a matching kernel binding exposing Session(enable_http2=...). The dependency and blocked merge/release status are explicit in the PR description. Compatibility with an older kernel wheel is not part of this change; the coordinated release must integrate the matching native artifacts before this option ships.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at 21cda69: the pin is still ^1.4.0 (pyproject.toml:73), and explicit enable_http2 reaches Session without the _kernel_session_accepts_kwarg check (client.py:221-226).

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High · 1 Medium

Mostly good, with 1 medium concern: an explicit enable_http2 reaches older kernel wheels without the capability check that geospatial uses. Validation and forwarding look correct. Nit: the new README paragraph sits between the install heading and its install line.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.

Other findings

  • 🟡 Medium — The new tests only capture kwargs in a fake Session, so they cannot detect a native transport that ignores the flag or incorrectly negotiates HTTP/2 through a proxy. Add a real-kernel regression that verifies negotiated protocols for omitted, true and false settings, including proxy fallback. (Found by GPT-6.1 Terra · the judge couldn't confirm it)

**geospatial_kwargs,
**max_connections_kwargs,
**http_headers_kwargs,
**self._http2_kwargs,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — An explicit enable_http2 is forwarded to the native Session with no capability check. Geospatial and max_connections are gated on _kernel_session_accepts_kwarg, but this option isn't. The floor is ^1.4.0 and the lock pins 1.4.0, so an older wheel can fail with a TypeError. Gate it with _kernel_session_accepts_kwarg("enable_http2") and raise NotSupportedError, or raise the kernel floor.

🔀 Found by both reviewers · confirmed by the judge

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).

Verdict: 1 High · 1 Medium

The forwarding and validation look correct and are tested, with one medium concern: an explicit enable_http2 is not gated for older kernel wheels.

This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.

Other findings

  • 🟡 Medium — The new test only captures kwargs in a permissive fake, so CI cannot detect a missing native argument or verify HTTP/2 negotiation, fallback, and proxy opt-out. Add a regression using the matching real kernel that exercises the settings and checks the negotiated transport. (Found by GPT-6.1 Terra · the judge couldn't confirm it)

**geospatial_kwargs,
**max_connections_kwargs,
**http_headers_kwargs,
**self._http2_kwargs,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High — enable_http2 is passed to the native Session without _kernel_session_accepts_kwarg, unlike geospatial and max_connections. An older kernel wheel that lacks the parameter fails with a generic TypeError/DatabaseError. Gate it and raise NotSupportedError naming the required kernel version, and bump the pin once the matching kernel ships.

🔀 Found by both reviewers · confirmed by the judge

This branch was previously deployed

1 inactive deployment
azure-prod — ccb6cecb Deployed Nov 19, 2026 by peco-review-bot[bot] via followup #2495
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant