Repository navigation
fix: partition high-cardinality metric descriptors - #227
Conversation
|
@atheurer This PR has conflicts that need to be resolved. |
1 similar comment
|
@atheurer This PR has conflicts that need to be resolved. |
731e2a0 to
e73e082
Compare
k-rister
left a comment
There was a problem hiding this comment.
PR Review: CommonDataModel#227 — fix: partition high-cardinality metric descriptors
Summary: One confirmed regression affects large comparisons without breakouts.
Scope: ad391507bf5fceaaf1c61c424a644838f2925a9e → e73e082538c209b2353b0f086317885faabc3187
Changed files: 5
Findings
- [P2] Exclude ordinary queries from the partition budget —
queries/cdmq/cdm.js:3422–3426(inline comment). Initial queries consume the shared 128-query partition budget, causing ordinary comparisons with 129 sample sets to fail with HTTP 413 even withbreakout: []. Preserve ordinary batched queries and apply the budget to additional partition work.
Validation
npm test: 60 passed.- JavaScript syntax checks passed for
cdm.jsandserver.js. - Mocked before/after reproduction: 128 sets succeed on both revisions; 129 sets succeed on the base but return
METRIC_DESCRIPTOR_LIMITon the head before contacting OpenSearch. - Live OpenSearch validation was not repeated; Crucible integration CI remained in progress at inspection.
File coverage
-
queries/cdmq/README.md— no primary findings -
queries/cdmq/cdm.js— 1 finding -
queries/cdmq/package.json— no primary findings -
queries/cdmq/server.js— no primary findings -
queries/cdmq/test/metric-descriptor-partition.test.js— no primary findings
Verdict
Request changes — fix the budget regression to preserve existing large-comparison behavior.
— AI-signed: Codex | model: gpt-6.1-sol | effort: high | contribution: code review, test execution, and before/after reproduction.
k-rister
left a comment
There was a problem hiding this comment.
PR Review: CommonDataModel#227 — fix: partition high-cardinality metric descriptors
Summary: No actionable defects found. The previous query-budget regression is fixed.
Scope: ad391507bf5fceaaf1c61c424a644838f2925a9e → 2b914a15bb3d95c55e5322d2f37ed6d742e25c0c
Changed files: 5
Validation
- Ran all six test files used by
npm testthroughcrucible wrapper: 61 passed, including the 129-set regression test. - JavaScript syntax checks and
git diff --checkpassed. - Reported CI checks passed.
- Live OpenSearch validation was not repeated.
File coverage
-
queries/cdmq/README.md— no findings -
queries/cdmq/cdm.js— no findings -
queries/cdmq/package.json— no findings -
queries/cdmq/server.js— no findings -
queries/cdmq/test/metric-descriptor-partition.test.js— no findings
Verdict
Approve — the earlier finding is resolved, with no new primary findings.
— AI-signed: Codex | model: gpt-6.1-sol | effort: high | contribution: code review, test execution, and verification of the previous finding's resolution.
Fixes #225.
Problem
Metric descriptor discovery builds one nested terms aggregation per breakout. Five-breakout
perf-recordqueries can exceed OpenSearch'ssearch.max_bucketslimit. OpenSearch wraps the limit in asearch_phase_execution_exceptioninside the HTTP 200_msearchresponse. The failed subresponse was then treated as a successful aggregation, producing an opaquereading 'key'failure and an HTTP 500.Change
too_many_buckets_exception, enumerate breakout cardinalities, partition on the smallest safe dimension, recurse on partitions that still exceed the limit, and merge terms by label without duplicates.search_phase_execution_exceptionresponse is recognized as a bucket-limit failure and partitioned.CDM_METRIC_DESCRIPTOR_MAX_QUERIES(default 128) andCDM_METRIC_DESCRIPTOR_MAX_DEPTH(default 8)._msearchresponses before reading aggregations and returnMETRIC_DESCRIPTOR_LIMITwith HTTP 413, the OpenSearch cause, and a recovery hint when partitioning cannot fit the budget.Validation
npm testpasses all 60 tests inqueries/cdmq.git diff --checkpass for the changed files.Live OpenSearch validation
Validated against OpenSearch 3.6.0 with the default
search.max_buckets=65535and an existing representativeperf-record/cpu-cyclesdataset:too_many_buckets_exceptionat 65,536 buckets, nested undersearch_phase_execution_exception.