Skip to content

Add rescore_factor to quantizer configs and rescore_limit to PQ - #2248

Merged
dirkkul merged 2 commits into
dev/1.41from
feat/rescore-factor
Nov 11, 2026
Merged

dirkkul merged 2 commits into
dev/1.41from
feat/rescore-factor

Conversation

@trengrj

@trengrj trengrj commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Motivation

weaviate/weaviate#13536 adds a rescoreFactor setting to every quantizer (PQ, BQ, SQ, RQ) and makes rescoreLimit apply to all of them, so the number of rescored candidates becomes min(ef, max(k, rescoreLimit, ceil(k × rescoreFactor))). The Python client had no way to set rescoreFactor on create or update, or to read it back from a collection config. PQ also had no rescore_limit in the client, so that is added here too to keep the four quantizers consistent.

Approach

rescore_factor is threaded through the three layers that already carry rescore_limit: the create models (Configure.VectorIndex.Quantizer.*), the update models (Reconfigure.VectorIndex.Quantizer.*), and the read-side dataclasses parsed from the schema JSON.

  • Appended, not grouped: the new keyword is added at the end of every existing factory signature rather than next to rescore_limit, so existing positional callers keep working. That is why Reconfigure...bq(rescore_limit, enabled, rescore_factor) looks oddly ordered.
  • Omitted when unset: the value is Optional[float] and defaults to None, so it is dropped from the payload. The server then applies its automatic value, and requests against servers that don't know the field are unchanged.
  • No client-side validation or version gate: the value is passed through as-is and the server decides what is valid (-1 automatic, or >= 1; fractional values allowed). The read side returns whatever the server sends, including -1 for automatic, rather than normalising it to None.

Key areas for review

  • weaviate/collections/classes/config_vector_index.py: the sq factory has two @overloads plus the implementation; all three signatures gain the argument.
  • weaviate/collections/classes/config.py: _PQConfig, _BQConfig, _SQConfig, _RQConfig gain new fields without defaults (see breaking changes).
  • weaviate/collections/classes/config_methods.py: __get_quantizer_config uses .get(), so configs from older servers parse with rescore_factor=None.

Risks

  • Older servers: nothing is sent unless the user sets it. If they do set it against a server without the feature, behaviour is whatever that server does with an unknown field. Worth deciding whether a version check is wanted.
  • First 1.41 entry in the CI matrix: WEAVIATE_141 is new on this branch, so unrelated tests may surface 1.41 differences. Add API key expiration for DB users on Weaviate 1.41+ #2249 adds the same variable with an older tag and will conflict on that line.

Testing

Unit tests plus one integration test. CI now also runs the integration suite against 1.41.0-dev-483b145, the image built from the merge commit of weaviate/weaviate#13536.

  • Create: all four quantizers serialise rescoreLimit and rescoreFactor under HNSW, and both keys are absent from the PQ payload when not set.
  • Read: an hfresh + RQ schema parses with rescoreFactor absent (None), -1, and 2.5.
  • Integration (test_hnsw_quantizer_rescore_factor, skipped below 1.41): for each of PQ, BQ, SQ and RQ on HNSW, create with rescore_limit and rescore_factor, read both back, update rescore_factor and check rescore_limit is preserved.
  • Update: reconfiguring RQ rescore_factor sets the new value and preserves the existing rescoreLimit.

Breaking changes

None for the public API. The read-side dataclasses gain required fields, so code constructing PQConfig/BQConfig/SQConfig/RQConfig directly needs the new arguments.

🤖 Generated with Claude Code

Exposes the rescoreFactor setting from weaviate/weaviate#13536 on the
pq, bq, sq and rq quantizers in Configure, Reconfigure and the config
returned by the server. PQ also gains rescore_limit, which the server
now honours for every quantizer.

Co-Authored-By: Claude <noreply@anthropic.com>

@orca-security-eu orca-security-eu Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca

@codecov-commenter

codecov-commenter commented Nov 11, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 81.25000% with 12 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (dev/1.41@6c129b2). Learn more about missing BASE report.

Files with missing lines Patch % Lines
integration/test_collection_config.py 33.33% 12 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@             Coverage Diff             @@
##             dev/1.41    #2248   +/-   ##
===========================================
  Coverage            ?   89.24%           
===========================================
  Files               ?      312           
  Lines               ?    24820           
  Branches            ?        0           
===========================================
  Hits                ?    22151           
  Misses              ?     2669           
  Partials            ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Co-Authored-By: Claude <noreply@anthropic.com>
@trengrj
trengrj requested a review from a team as a code owner November 11, 2026 05:56
@dirkkul
dirkkul merged commit 4700e6b into dev/1.41 Nov 11, 2026
127 checks passed
@dirkkul
dirkkul deleted the feat/rescore-factor branch November 11, 2026 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants