Skip to content

fix: partition high-cardinality metric descriptors - #227

Merged
k-rister merged 4 commits into
masterfrom
fix/cdm-issue-225-high-cardinality-breakouts
Nov 17, 2026
Merged

k-rister merged 4 commits into
masterfrom
fix/cdm-issue-225-high-cardinality-breakouts

Conversation

@atheurer

@atheurer atheurer commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #225.

Problem

Metric descriptor discovery builds one nested terms aggregation per breakout. Five-breakout perf-record queries can exceed OpenSearch's search.max_buckets limit. OpenSearch wraps the limit in a search_phase_execution_exception inside the HTTP 200 _msearch response. The failed subresponse was then treated as a successful aggregation, producing an opaque reading 'key' failure and an HTTP 500.

Change

  • Keep the existing single descriptor query when it fits.
  • On too_many_buckets_exception, enumerate breakout cardinalities, partition on the smallest safe dimension, recurse on partitions that still exceed the limit, and merge terms by label without duplicates.
  • Unwrap nested OpenSearch causes so the live search_phase_execution_exception response is recognized as a bucket-limit failure and partitioned.
  • Preserve value and regex filters, aggregate breakout labels, label order, multiple query sets, and the existing follow-up metric-data queries.
  • Bound partition work with CDM_METRIC_DESCRIPTOR_MAX_QUERIES (default 128) and CDM_METRIC_DESCRIPTOR_MAX_DEPTH (default 8).
  • Validate _msearch responses before reading aggregations and return METRIC_DESCRIPTOR_LIMIT with HTTP 413, the OpenSearch cause, and a recovery hint when partitioning cannot fit the budget.
  • Return structured transport and OpenSearch errors instead of dereferencing a null response.

Validation

  • npm test passes all 60 tests in queries/cdmq.
  • Prettier and git diff --check pass for the changed files.
  • New tests cover the exact nested bucket-limit response shape, recursive partitioning, low-cardinality selection, value and regex breakouts, aggregate label order, duplicate-free reassembly, multiple sets, fan-out and depth limits, malformed responses, and transport failures.

Live OpenSearch validation

Validated against OpenSearch 3.6.0 with the default search.max_buckets=65535 and an existing representative perf-record/cpu-cycles dataset:

  • The original five-breakout aggregation failed with too_many_buckets_exception at 65,536 buckets, nested under search_phase_execution_exception.
  • The patched planner partitioned by hostname and completed in 3 descriptor requests with 8 total subqueries.
  • Its 53,159 labels exactly matched an independent composite-aggregation enumeration of all breakout combinations: zero missing and zero extra labels.
  • A separate live metric-ID lookup using two requested sets returned 53,159 and 48,483 unique IDs with no empty labels, duplicates, or cross-set ID overlap.

@project-crucible-tracking project-crucible-tracking Bot moved this to In Progress in Crucible Tracking Nov 12, 2026
@atheurer
atheurer marked this pull request as ready for review November 12, 2026 15:29
@atheurer
atheurer requested a review from k-rister November 12, 2026 18:31
@k-rister

Copy link
Copy Markdown
Contributor

@atheurer This PR has conflicts that need to be resolved.

1 similar comment
@k-rister

Copy link
Copy Markdown
Contributor

@atheurer This PR has conflicts that need to be resolved.

@atheurer
atheurer force-pushed the fix/cdm-issue-225-high-cardinality-breakouts branch from 731e2a0 to e73e082 Compare November 12, 2026 20:36

@k-rister k-rister left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review: CommonDataModel#227 — fix: partition high-cardinality metric descriptors

Summary: One confirmed regression affects large comparisons without breakouts.
Scope: ad391507bf5fceaaf1c61c424a644838f2925a9e → e73e082538c209b2353b0f086317885faabc3187
Changed files: 5

Findings

  • [P2] Exclude ordinary queries from the partition budget — queries/cdmq/cdm.js:3422–3426 (inline comment). Initial queries consume the shared 128-query partition budget, causing ordinary comparisons with 129 sample sets to fail with HTTP 413 even with breakout: []. Preserve ordinary batched queries and apply the budget to additional partition work.

Validation

  • npm test: 60 passed.
  • JavaScript syntax checks passed for cdm.js and server.js.
  • Mocked before/after reproduction: 128 sets succeed on both revisions; 129 sets succeed on the base but return METRIC_DESCRIPTOR_LIMIT on the head before contacting OpenSearch.
  • Live OpenSearch validation was not repeated; Crucible integration CI remained in progress at inspection.

File coverage

  • queries/cdmq/README.md — no primary findings
  • queries/cdmq/cdm.js — 1 finding
  • queries/cdmq/package.json — no primary findings
  • queries/cdmq/server.js — no primary findings
  • queries/cdmq/test/metric-descriptor-partition.test.js — no primary findings

Verdict

Request changes — fix the budget regression to preserve existing large-comparison behavior.

— AI-signed: Codex | model: gpt-6.1-sol | effort: high | contribution: code review, test execution, and before/after reproduction.

Comment thread queries/cdmq/cdm.js Resolved
@atheurer
atheurer requested review from k-rister and removed request for k-rister November 13, 2026 01:11

@k-rister k-rister left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review: CommonDataModel#227 — fix: partition high-cardinality metric descriptors

Summary: No actionable defects found. The previous query-budget regression is fixed.
Scope: ad391507bf5fceaaf1c61c424a644838f2925a9e → 2b914a15bb3d95c55e5322d2f37ed6d742e25c0c
Changed files: 5

Validation

  • Ran all six test files used by npm test through crucible wrapper: 61 passed, including the 129-set regression test.
  • JavaScript syntax checks and git diff --check passed.
  • Reported CI checks passed.
  • Live OpenSearch validation was not repeated.

File coverage

  • queries/cdmq/README.md — no findings
  • queries/cdmq/cdm.js — no findings
  • queries/cdmq/package.json — no findings
  • queries/cdmq/server.js — no findings
  • queries/cdmq/test/metric-descriptor-partition.test.js — no findings

Verdict

Approve — the earlier finding is resolved, with no new primary findings.

— AI-signed: Codex | model: gpt-6.1-sol | effort: high | contribution: code review, test execution, and verification of the previous finding's resolution.

@k-rister
k-rister merged commit 29fe59b into master Nov 17, 2026
45 checks passed
@k-rister
k-rister deleted the fix/cdm-issue-225-high-cardinality-breakouts branch November 17, 2026 14:15
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Crucible Tracking Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Partition high-cardinality breakout queries to avoid nested aggregation bucket limits

2 participants