Skip to content

feat: add SQL_QUERY dashboard filter type to the schema - #3672

Open
karl-power wants to merge 3 commits into
mainfrom
sql-dashboard-variables
Open

karl-power wants to merge 3 commits into
mainfrom
sql-dashboard-variables

Conversation

@karl-power

Copy link
Copy Markdown
Contributor

Summary

Adds the SQL_QUERY dashboard filter type to the schema: a variable-only filter whose dropdown values come from the first column of a free-form ClickHouse query. This is the first of four PRs; it adds the data model and validation only, so the filter can't be created or used in the UI yet.

  • SqlQueryDashboardFilterSchema stores a connection ID and a sqlTemplate, with isBroadcastEnabled: false and isVariableEnabled: true fixed, since there's no column to broadcast.
  • isSqlQueryFilter type guard.
  • validateDashboardFilterQueries expands the template with empty variable selections and flags $__sourceTable / $__filters (the filter has no source) and any macro or variable expansion error. The SQL itself isn't parsed; ClickHouse reports syntax errors when it runs.
  • Dashboard template export replaces the filter's connection ID with the connection name, as it does for raw-SQL tiles.
  • The external API and MCP treat the type as internal-only, like ADHOC, until they expose it.
  • The filter settings list handles the new type in its exhaustive switch, labelling it "SQL query". It's only visible for a filter saved through the API or a JSON import, since the UI can't create one yet.

How to test on Vercel preview

N/A — non-UI change

References

  • Linear Issue: HDX-5524
  • Related PRs: follow-ups will add API connection validation and MCP / external API support, the dropdown values lookup, and the filter editor form.

@karl-power
karl-power requested a review from a team as a code owner November 16, 2026 10:56
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 16, 2026 11:32am UTC
hyperdx-storybook Ready Ready Preview Nov 16, 2026 11:32am UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 96a5282

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-2 Low risk — AI review + quick human skim label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Cross-layer change: touches frontend (packages/app) + backend (packages/api) + shared utils (packages/common-utils)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 6
  • Production lines changed: 248 (+ 202 in test files, excluded from tier calculation)
  • Branch: sql-dashboard-variables
  • Author: karl-power

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

Add a variable-only dashboard filter whose dropdown values come from the
first column of a ClickHouse query run on a connection.

- Add SqlQueryDashboardFilterSchema and the isSqlQueryFilter guard
- Validate the template's macros and variable references, rejecting
  $__sourceTable and $__filters
- Export the filter's connection as a name in dashboard templates
- Keep the type out of the external API and MCP for now

Refs HDX-5524
@karl-power
karl-power force-pushed the sql-dashboard-variables branch from 989bc97 to f94e9ab Compare November 16, 2026 10:57
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 466 passed • 1 skipped • 1091s

Status Count
✅ Passed 466
❌ Failed 0
⚠️ Flaky 2
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

1 new clone introduced by this PR.

  • packages/common-utils/src/core/utils.ts:988-1008 ↔ packages/common-utils/src/core/utils.ts:1036-1056 — 21 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.18%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

if (isStaticListFilter(filter)) return filter;

if (isSqlQueryFilter(filter)) {
filter.connection =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — Export swaps the SQL_QUERY connection ID for a name, but import never maps it back

In packages/app/src/DBDashboardImportPage.tsx, the Mapping effect (~L634) and onSubmit (~L1044-1064) only resolve connection names for raw SQL tiles. An exported SQL_QUERY filter is saved with connection: 'Production DB' (a name) where an ID is expected. Also, a connection missing at export time is written as '', which fails connection: z.string().min(1), so DashboardTemplateSchema rejects the whole file. Fix: add a per-filter connection mapping (match name to ID, require a selection, write connection: id on submit) in this PR, or keep the ID on export until import support lands.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 93a610b: the import page now maps SQL query filter connections by name, like raw SQL tiles, and blocks the import until each one is mapped.

return (filters ?? []).filter(
(filter): filter is ExternalCompatibleDashboardFilter =>
!isAdhocFilter(filter),
!isAdhocFilter(filter) && !isSqlQueryFilter(filter),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — External API/MCP updates silently delete SQL_QUERY filters they hide

GET hides SQL_QUERY filters through getExternalCompatibleFilters. But PUT (routers/external-api/v2/dashboards.ts:3632) and MCP saveDashboard.ts:386-415 replace filters with only the request's filters, so a read-modify-write round trip deletes a SQL_QUERY filter and breaks any tile that uses its $variable. Fix: carry existing internal-only filters whose IDs are not in the request into the saved list, as MCP already does for hidden PromQL filters, including the variable-name clash check.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

existing behaviour for hidden filters. a follow up PR will expose SQL filters anyway

filterName: filter.name,
language: 'sql',
where: filter.sqlTemplate,
detail,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The new SQL_QUERY validation message is never shown to the user

The only caller, DBDashboardImportPage.tsx:339-343, renders {filterName}: invalid {language} query: {where} and ignores detail. The user sees the whole template marked as an invalid SQL query (even though the SQL is never parsed) and never sees the reason, such as "$__sourceTable cannot be used". Fix: render warning.detail when it is set.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 96a5282: the import warning now shows detail under each filter line, for all filter types.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

4 finding(s): 🔴 0 critical · 🟠 1 major · 🔵 3 minor

4 posted as inline comment(s) on the changed lines.


Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. This is a clean, well-tested schema-only change. The findings below are consistency gaps and latent risks that become live when follow-up PRs wire up SQL_QUERY creation and value lookup — worth tracking now or filing as follow-ups.

🟡 P2 — recommended

  • packages/api/src/utils/externalApi.ts:264 — getExternalCompatibleFilterValues strips only ADHOC values, so a SQL_QUERY filter's variable-keyed savedFilterValues entry is returned by the external API/MCP even though the filter definition is now excluded, leaving an orphan value that leaks the variable name and selections and contradicts the "excluded like ADHOC" intent.
    • Fix: Exclude variable-keyed savedFilterValues whose filter id/variable belongs to an internal-only (ADHOC or SQL_QUERY) filter, mirroring the definition-level exclusion.
    • correctness, api-contract
  • packages/common-utils/src/core/utils.ts:955 — template export remaps a SQL_QUERY filter's connection from id to name, but the import path (DBDashboardImportPage.tsx) has no inverse mapping, so an exported-then-reimported SQL_QUERY filter keeps a connection name in a field expected to hold a connection id.
    • Fix: Defer the export-side connection remap until the import side can resolve names back to ids, or add a connection-mapping path on import analogous to the per-tile connection mappings.
    • correctness, maintainability
  • packages/api/src/utils/externalApi.ts:260 — the new getExternalCompatibleFilters exclusion of SQL_QUERY (and pre-existing ADHOC) has no test; a regression re-exposing the filter through the external API/MCP would go uncaught.
    • Fix: Add a test feeding a SQL_QUERY (and ADHOC) filter through getExternalCompatibleFilters / the external v2 dashboards path asserting it is dropped while other types pass through.
    • testing, api-contract, kieran-typescript
🔵 P3 nitpicks (4)
  • packages/api/src/routers/external-api/v2/dashboards.ts:3632 — adding SQL_QUERY to the internal-only class extends the existing silent-drop-on-write pattern (the REST $set and MCP saveDashboard preserve only PromQL-hidden filters), so a GET→PUT round-trip will drop internal-only SQL_QUERY filters once they can be persisted. Pre-existing pattern.
    • Fix: Re-merge all internal-only filters (ADHOC and SQL_QUERY) absent from the request body on the write path, not just PromQL-hidden ones.
  • packages/api/src/utils/externalApi.ts:244 — InternalOnlyDashboardFilter is derived via Extract<…, { type: 'ADHOC' | 'SQL_QUERY' }> while the runtime narrowing is a separate hand-written predicate, so a future internal-only type added to the union without a matching runtime guard would silently leak.
    • Fix: Drive both the type union and the runtime predicate from one INTERNAL_ONLY_TYPES constant, or add a satisfies never exhaustiveness assertion linking them.
  • packages/app/src/components/DashboardFiltersModal/filterFormState.ts:64 — the editor's filter.type if-chains default to QUERY_EXPRESSION with no exhaustiveness guard, so an existing SQL_QUERY filter opened in the editor would render QUERY_EXPRESSION fields. Pre-existing pattern, acknowledged as not-yet-implemented.
    • Fix: Add a satisfies never guard so a future author is forced to handle SQL_QUERY in the editor.
  • packages/common-utils/src/core/utils.ts:957 — the ?? '' fallback when a SQL_QUERY connection id is not found produces a value that would fail the schema's connection: z.string().min(1) on re-parse, and is not covered by any test.
    • Fix: Add a test for the not-found-connection export branch and decide whether an empty connection should be representable.

Reviewers (7): correctness, testing, maintainability, project-standards, api-contract, kieran-typescript, previous-comments.

Testing gaps:

  • No test asserts the external API/MCP excludes SQL_QUERY filter definitions or their saved values.
  • No round-trip test for exporting then re-importing a SQL_QUERY template (where the connection id→name asymmetry surfaces).
  • No test for the convertToDashboardTemplate connection-not-found ('') fallback branch.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions github-actions Bot added review/tier-3 Standard — full human review required and removed review/tier-2 Low risk — AI review + quick human skim labels Nov 16, 2026
tooltip: 'Sources the condition keys and values come from',
label: getSourceNames(filter.sources, sources),
};
case 'SQL_QUERY':

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — Editing a SQL_QUERY filter opens a broken form, and saving it always fails

The list shows every filter with an edit pencil (line 184). For a SQL_QUERY filter, toFormValues in filterFormState.ts drops connection and sqlTemplate, DashboardFilterEditForm.tsx falls through to QueryExpressionFilterEditForm, and the type picker has no SQL_QUERY option. toSavedFilter then reaches the final DashboardFilterSchema.parse, which throws, and the user only sees "This filter is incomplete and could not be saved." Until the editor PR lands, hide or disable the edit action for isSqlQueryFilter(filter), or carry the SQL fields through toFormValues/toSavedFilter.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond


if (isSqlQueryFilter(filter)) {
filter.connection =
connections.find(conn => conn.id === filter.connection)?.name ?? '';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Exporting a filter whose connection was deleted produces a template that cannot be imported

When the connection can't be found, the code falls back to ''. But SqlQueryDashboardFilterSchema.connection is z.string().min(1), so DashboardTemplateSchema rejects the exported file and the whole import fails. A raw-SQL tile's connection is plain z.string(), so tiles don't have this problem. Fall back to a non-empty placeholder (for example the original ID) so the import's unmapped-connection row shows up, or relax the template schema.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

]);

// Propagate connection mapping changes to other RawSQL tiles with the same input connection
// Propagate connection mapping changes to other RawSQL tiles and SQL query

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The rewritten connection-propagation effect has no test for the new tile↔filter propagation

This effect now copies a mapping between raw-SQL tiles and SQL query filters that share an input connection, and it tracks two prev-refs. No test in DBDashboardImportPage.test.tsx changes one mapping and checks that the other updates. The new tests only cover auto-mapping and the missing-connection error. Add a test with a template that has a raw-SQL tile and a SQL_QUERY filter on the same unknown connection: select a connection on one, then assert the other gets it and a dirty field isn't overwritten.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

{warning.filterName}: invalid {warning.language} query:{' '}
{warning.where}
</Text>
{warning.detail && (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — New warning detail text uses the raw palette color c="yellow"

The Repository conventions section 'Semantic component variants' says to use <Text variant="warning"> for status text instead of raw palette colors such as c="yellow", and to migrate nearby usages you touch. Use variant="warning" on the new detail Text (and on the sibling Text this diff rewraps).

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 96a5282c Deployed Nov 16, 2026 by vercel[bot]
Preview – hyperdx-storybook — 96a5282c Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant