Repository navigation
feat(kernel): expose HTTP/2 opt-out - #1070
aakash-saravanan-db wants to merge 6 commits into
Conversation
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 2 Low
Mostly good, with 1 medium concern. Leaving enable_http2 unset keeps working on older wheels, but setting it explicitly on a wheel that lacks the option fails with a vague error instead of a clear NotSupportedError.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.
| def test_open_session_preserves_http2_option(monkeypatch, enabled): | ||
| captured = {} | ||
|
|
||
| def fake_session(**kwargs): |
There was a problem hiding this comment.
🔵 Low — The new tests only check kwarg pass-through to a fake Session, so they can't detect a real kernel that ignores the option or negotiates the wrong protocol. Add a realkernel e2e test that checks HTTP/2 negotiation, the explicit opt-out and the proxy fallback.
🔀 Found by GPT-6.1 Terra · confirmed by the judge
There was a problem hiding this comment.
The shared kernel owns the TLS negotiation, opt-out and proxy tests; this adapter owns omitted/true/false forwarding. The matching PyO3 binding now also tests all three mappings in its existing HTTP-config helper. Actual source-built Python transport checks verify negotiated HTTP versions. These cover the distinct boundaries without duplicating the shared TLS suite in this repository.
There was a problem hiding this comment.
Still open at 21cda69: tests/ only has fake-Session forwarding tests (test_kernel_client.py:411-450). No real-wheel test shows the installed kernel accepts and honors enable_http2.
| if value is None: | ||
| return {} | ||
| if not isinstance(value, bool): | ||
| raise ValueError("enable_http2 must be a bool") |
There was a problem hiding this comment.
🟠 High — _kernel_http2_kwargs passes an explicit enable_http2 to Session without calling _kernel_session_accepts_kwarg. An older kernel wheel then fails with a TypeError, which is wrapped into a vague OperationalError. Gate it like _kernel_geospatial_kwargs and raise NotSupportedError saying a newer databricks-sql-kernel is required.
🔀 Found by both reviewers · confirmed by the judge
There was a problem hiding this comment.
This driver release requires a matching kernel binding exposing Session(enable_http2=...). The dependency and blocked merge/release status are explicit in the PR description. Compatibility with an older kernel wheel is not part of this change; the coordinated release must integrate the matching native artifacts before this option ships.
There was a problem hiding this comment.
Still open at 21cda69: the pin is still ^1.4.0 (pyproject.toml:73), and explicit enable_http2 reaches Session without the _kernel_session_accepts_kwarg check (client.py:221-226).
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Medium
Mostly good, with 1 medium concern: an explicit enable_http2 reaches older kernel wheels without the capability check that geospatial uses. Validation and forwarding look correct. Nit: the new README paragraph sits between the install heading and its install line.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.
Other findings
- 🟡 Medium — The new tests only capture kwargs in a fake Session, so they cannot detect a native transport that ignores the flag or incorrectly negotiates HTTP/2 through a proxy. Add a real-kernel regression that verifies negotiated protocols for omitted, true and false settings, including proxy fallback. (Found by GPT-6.1 Terra · the judge couldn't confirm it)
| **geospatial_kwargs, | ||
| **max_connections_kwargs, | ||
| **http_headers_kwargs, | ||
| **self._http2_kwargs, |
There was a problem hiding this comment.
🟠 High — An explicit enable_http2 is forwarded to the native Session with no capability check. Geospatial and max_connections are gated on _kernel_session_accepts_kwarg, but this option isn't. The floor is ^1.4.0 and the lock pins 1.4.0, so an older wheel can fail with a TypeError. Gate it with _kernel_session_accepts_kwarg("enable_http2") and raise NotSupportedError, or raise the kernel floor.
🔀 Found by both reviewers · confirmed by the judge
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Terra), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Medium
The forwarding and validation look correct and are tested, with one medium concern: an explicit enable_http2 is not gated for older kernel wheels.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Terra added findings it doesn't mention — see the findings below.
Other findings
- 🟡 Medium — The new test only captures kwargs in a permissive fake, so CI cannot detect a missing native argument or verify HTTP/2 negotiation, fallback, and proxy opt-out. Add a regression using the matching real kernel that exercises the settings and checks the negotiated transport. (Found by GPT-6.1 Terra · the judge couldn't confirm it)
| **geospatial_kwargs, | ||
| **max_connections_kwargs, | ||
| **http_headers_kwargs, | ||
| **self._http2_kwargs, |
There was a problem hiding this comment.
🟠 High — enable_http2 is passed to the native Session without _kernel_session_accepts_kwarg, unlike geospatial and max_connections. An older kernel wheel that lacks the parameter fails with a generic TypeError/DatabaseError. Gate it and raise NotSupportedError naming the required kernel version, and bump the pin once the matching kernel ships.
🔀 Found by both reviewers · confirmed by the judge
Feature or gap
Direct kernel-owned HTTPS requests negotiate HTTP/2 with HTTP/1.1 fallback. Nonempty HTTP_PROXY, HTTPS_PROXY, ALL_PROXY or lowercase equivalents keep the entire kernel client on HTTP/1.1.
What this PR addresses
Add the kernel-only boolean enable_http2 connection option. Omission preserves the kernel default; False forces HTTP/1.1 for every kernel-owned HTTPS request. Invalid values raise ValueError before native session construction.
Dependencies
Requires the native HTTP/2 support in kernel #483. Requires a kernel build exposing Session(enable_http2=...). The driver and matching kernel ship in the coordinated release.
Tests and validation
Validation status: blocked. Matching kernel integration and the full metadata compatibility matrix remain open.
All 288 kernel-client unit tests pass, including boolean validation through connect() and explicit-false forwarding. Final source-built native transport and DBSQL live full-result checks pass for omitted, true and false settings. A rerun on an additional supported warehouse type timed out; the warehouse was observed in STARTING state afterward. Five-pair connection/query and full 250,000-row result-drain comparisons on DBSQL against Python Thrift meet parity. Full wildcard metadata comparator validation remains incomplete.
This pull request and its description were written by Isaac.