Skip to content

Drop undecodable codings from Accept-Encoding - #2381

Open
pavel-ptashyts wants to merge 6 commits into
AsyncHttpClient:mainfrom
maygemdev:claude/accept-encoding-filter-113031
Open

pavel-ptashyts wants to merge 6 commits into
AsyncHttpClient:mainfrom
maygemdev:claude/accept-encoding-filter-113031

Conversation

@pavel-ptashyts

@pavel-ptashyts pavel-ptashyts commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

With automatic decompression on, a caller-set Accept-Encoding is meant to lose br and zstd when their native libraries are missing. Three bugs defeated that, so br (and in some cases zstd) was still offered:

  • The zstd filter ran on the original value and overwrote the Brotli result. With both libraries missing (both are optional dependencies) br came back. This dates from support zstd compression #1950.
  • Each filter only cut a trailing ", br" or ", zstd". The browser-style gzip, deflate, br, zstd kept br even with the filters chained, and so did a coding listed first, without a space, in another case or with a weight.
  • Only the first Accept-Encoding field was read, and whenever either library was missing the header was replaced by that field alone, dropping the caller's other fields.

Impact: a server that picks br or zstd sends a body AHC cannot decode. PullDecompression leaves such a body as it is, so the handler receives compressed bytes where it expects decoded ones.

Change

  • HttpUtils.filterOutBrotliFromAcceptEncoding / filterOutZstdFromAcceptEncoding now drop every list element that offers the coding, wherever it is and in any case or spacing (RFC 9110 section 12.5.3). Signatures are unchanged; the value is returned as is when nothing is dropped. Dropping the only element leaves an empty value, which accepts no content coding.
  • An element with weight 0 (br;q=0, BR ; Q = 0.000) refuses the coding and is kept: dropping it next to * would let the server pick the coding, while main left *, br;q=0 alone.
  • When the list has a * that offers codings, the dropped elements are replaced by one <coding>;q=0 refusal, so *, zstd;q=0.5 becomes *, zstd;q=0 rather than *, which would raise zstd to the wildcard's weight.
  • NettyRequestFactory moves the check into a package-private filterOutUndecodableAcceptEncodings(headers, brotliAvailable, zstdAvailable) that chains the filters on the running value and filters all fields as one comma-joined list (RFC 9110 section 5.3), without joining when there is one field. It returns at once when both libraries are available and sets the header only when something was dropped.
  • The factory reads Brotli.isAvailable() and Zstd.isAvailable() once, in its constructor; a package-private constructor takes both flags so tests can drive newNettyRequest with either library missing.

No public API change.

Tests

Brotli and zstd are both on the test classpath, so NettyRequestFactoryAcceptEncodingTest passes the availability flags to the helper and to the factory. Its newNettyRequest tests cover each combination of the flags with automatic decompression on, the header left untouched with it off, and the enforced-compression header. HttpUtilsTest covers position, separators, case, weights, zero-weight refusals and wildcards. On main 7 of the new assertions fail, for example gzip, deflate, br, zstd with neither library becomes gzip, deflate, br; the zero-weight and wildcard tests each failed before their fix.

mvnw.cmd clean verify passes locally on JDK 17 (no JDK 11 on the host): 1953 tests, 0 failures, Revapi green. CI covers JDK 11.

Not changed

  • A * still offers every coding the list does not refuse. gzip, * is left as is, and br, * with both libraries missing becomes br;q=0, *, where the * still covers zstd. Left for a separate change.
  • Over HTTP/2 the filter assumes too much: Http2ContentDecompressor decodes only gzip and deflate, so a br or zstd body reaches the handler compressed even with the library present. Needs its own issue.
  • As the review noted, Netty's HttpContentCompressor may still answer gzip, br;q=0, * with br. main gives the same result, and br;q=0 is the refusal RFC 9110 section 12.5.3 defines, so it is kept.

Second-agent review

Two agent reviewers, each a fresh read-only CLI run on git diff upstream/main...HEAD, reasoning effort medium; the model and effort below are as reported by each run.

Pass Commit Guided: Codex gpt-6-astra Independent: Claude claude-opus-5-5
1 f4c709d No actionable findings *, br;q=0 became *, letting a server pick br (fixed in 93f51c0); String.join per request for one field (fixed in 93f51c0)
2 93f51c0 No actionable findings *, zstd;q=0.5 became *, raising zstd's weight (fixed in d3a31c6)
3 d3a31c6 No actionable findings No actionable findings

Optional notes from pass 3 left as they are:

  • *;q=0, br or identity;q=0, br with Brotli missing becomes *;q=0 / identity;q=0, which accepts nothing, so a strict server may answer 406. The reviewer marked it optional and not a regression (main gave the same result). Not changed: rewriting the caller's explicit identity;q=0 would override their intent, and a 406 is easier to diagnose than a body that arrives still compressed.
  • headers.getAll still allocates a list per request when a library is missing. The reviewer marked it as needing no action: the path is taken only then, and the filters return early when the coding is absent.
  • Codex noted on each pass that the AGENTS.md gate asks for clean verify on JDK 11; locally it ran on JDK 17, and CI covers JDK 11.

Claude Code on behalf of Pavel Ptashyts

🤖 Generated with Claude Code

pavel-ptashyts and others added 5 commits November 5, 2026 19:49
With automatic decompression on, a caller-set Accept-Encoding is meant
to lose br and zstd when their native libraries are missing. Three bugs
defeated that:

- The zstd filter ran on the original value and overwrote the Brotli
  result, so with both libraries missing (both are optional
  dependencies) br was offered again. This dates from AsyncHttpClient#1950.
- Each filter only cut a trailing ", br" or ", zstd". The browser-style
  "gzip, deflate, br, zstd" kept br even with the filters chained, and
  so did a coding listed first, without a space, in another case or
  with a weight.
- Only the first Accept-Encoding field was read, and whenever either
  library was missing the header was replaced by that field alone,
  dropping the caller's other fields.

A server that picks br or zstd then sends a body AHC cannot decode:
the response decompressor leaves such a body as it is, so the handler
receives compressed bytes where it expects decoded ones.

Filter each list element by its coding token, chain the filters on the
running value, and filter all fields as one list, setting the header
only when something was dropped. The check moves into a package-private
helper so tests can drive it with both libraries missing, as both are
on the test classpath.

Claude Code on behalf of Pavel Ptashyts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve the HttpUtilsTest conflict by keeping both the cookie pair
test from upstream and the Accept-Encoding filter tests, and spell
the tab in one filter test as an escape instead of a raw tab.

Claude Code on behalf of Pavel Ptashyts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bring the branch up to date with main before opening the pull
request.

Claude Code on behalf of Pavel Ptashyts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The filters dropped every element naming br or zstd, including one
with weight 0. Such an element refuses the coding, so dropping it next
to a wildcard turned "*, br;q=0" into "*" and let a server pick the
coding AHC cannot decode, while main left that header alone. Elements
with q=0, in any spelling RFC 9110 allows, are now kept.

A single Accept-Encoding field, the common case, is also no longer
copied through String.join on every request when a library is
missing.

Found by the independent review of AsyncHttpClient#2381.

Claude Code on behalf of Pavel Ptashyts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dropping an element that offers br or zstd with a non-zero weight let
a "*" element offer the coding at the wildcard's weight instead, so
"*, zstd;q=0.5" became "*" and raised zstd from 0.5 to 1. When the
list has a "*" that offers codings, the dropped elements are now
replaced by one explicit "<coding>;q=0" refusal.

Found by the second independent review of AsyncHttpClient#2381.

Claude Code on behalf of Pavel Ptashyts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brotli and zstd are both on the test classpath, so nothing exercised
the filter call in newNettyRequest: removing it, swapping its
arguments or dropping the automatic-decompression check left every
test green, as review pointed out.

The factory now reads Brotli.isAvailable() and Zstd.isAvailable() once
in its constructor, and a package-private constructor takes both
flags. New tests build requests through newNettyRequest with each
combination of the flags, with automatic decompression on and off,
and with compression enforced.

Claude Code on behalf of Pavel Ptashyts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pavel-ptashyts

pavel-ptashyts commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor Author

The failing test (windows-latest, 17) check on cf07110 is not caused by this PR. It comes from multipart upload and retry code that this PR does not touch.

What happened in that job:

  1. In MultipartUploadTest.testSendFileInputStreamUnknownContentLength, Jetty aborted the connection mid-upload ("An established connection was aborted by the software in your host machine"), and AHC retried the POST.
  2. The retry re-sent the same InputStreamPart, whose stream InputStreamMultipartPart had already closed, so it failed with IOException: Stream closed.
  3. BodyChunkedInput.readChunk does not release its chunk when transferTo throws. That ByteBuf leaked, and the leak detector then failed about 15 later test classes with initializationError (PutByteBufTest, PutFileTest, TransferListenerTest, ...).

The other 12 checks passed. #2383 fixes this: the test servlet drains the request before answering, and readChunk releases the chunk when transferTo throws (with BodyChunkedInputTest). Replaying a consumed InputStreamPart on retry is not part of #2383; if needed, we can open a separate PR for that fix. The check here should pass once #2383 is merged or the job is rerun.

Claude Code on behalf of Pavel Ptashyts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants