Skip to content

feat(gooddata-eval): keep workflowStatus parts out of scored answers - #1926

Open
tychtjan wants to merge 1 commit into
masterfrom
jtyc/gdai-2733
Open

tychtjan wants to merge 1 commit into
masterfrom
jtyc/gdai-2733

Conversation

@tychtjan

@tychtjan tychtjan commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

gen-ai is adding a stored workflowStatus multipart part (GDAI-2733) that tells the browser a dashboard workflow ended with nothing left open, e.g. after the user declines a proposed chart. It rides on the turn's last visible answer, next to the text:

{"type": "multipart", "parts": [
  {"type": "text", "text": "Got it, leaving the dashboard as it is."},
  {"type": "workflowStatus", "skills": ["dashboard_editor"], "status": "ended"}
]}

Without this change the SSE client puts the part into unhandled_parts, and render_answer_text appends it to the text evaluators score:

Scored answer text
master Got it, leaving the dashboard as it is.\n\n[workflowStatus]\n{"skills": ["dashboard_editor"], "status": "ended"} plus an "unknown multipart part type" warning
this branch Got it, leaving the dashboard as it is.

_handle_multipart now skips the part, and _KNOWN_PART_TYPES lists it.

The gen-ai side is gooddata/gdc-nas#29907. This one is safe to merge first: until gen-ai sends the part, the new branch never runs. tests/tavern-e2e in gdc-nas locks gooddata-eval in its uv.lock, so that lock needs to move to a release with this change before or together with the gen-ai change.

Test plan

  • test_workflow_status_part_stays_out_of_the_answer: a status part followed by a text part scores as the text alone. Fails when the branch is removed, when continue becomes return, and when the part goes to search_results.
  • test_known_part_types_matches_the_documented_gen_ai_union includes workflowStatus.
  • make -C packages/gooddata-eval format lint type-check: clean.
  • gooddata-eval suite on py314: 2650 passed, 1 failed. The failure (test_only_the_first_questionnaire_gets_the_structured_answers) fails the same way on master and is unrelated.
  • py310 to py313 run in CI only.

Risk

low: an eval-only parser change that drops one part type the server does not send yet.

jira: GDAI-2733
risk: low

Summary by CodeRabbit

  • Bug Fixes
    • Workflow status updates are no longer treated as unhandled content or shown in rendered answers.

gen-ai is adding a stored workflowStatus multipart part that tells the
browser a dashboard workflow ended with nothing left open. It rides on
the turn's last visible answer, next to the text.

The SSE client put every non-text part it did not handle into
unhandled_parts, and render_answer_text appends those to the text the
evaluators score, so dashboard evals would grade a JSON blob as part of
the answer and log an unknown-part warning on every decline turn.

Register the type and skip it in _handle_multipart: it carries nothing
an evaluator scores.

jira: GDAI-2733
risk: low
@coderabbitai

coderabbitai Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The multipart client now recognizes workflowStatus and skips it instead of reporting it as unhandled. Tests check that a response with workflowStatus and text renders the text without unhandled or search-result parts.

Changes: workflowStatus multipart handling

Layer / File(s) Summary
Recognize and skip workflowStatus parts
packages/gooddata-eval/src/gooddata_eval/core/chat/sse_client.py, packages/gooddata-eval/tests/test_chat_render.py
The client recognizes and skips workflowStatus parts. Tests verify that the text "Done." renders without unhandled or search-result parts, and that workflowStatus is in the expected part-type set.

Estimated code review effort: 2 (Simple) | ~8 minutes

Merge Risk: Minimal · up to e6228

Workflow status is excluded from scored answers. No merge-blocking issue is evident.

Pre-merge checks: Passed 5

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly describes the main change: excluding workflowStatus parts from scored answers.
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 2 files.

Finishing Touches

Generate docstrings

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

A rabbit reads the stream with care
workflowStatus hops by there
The answer keeps its words in view
No search results sneak through
The rabbit thumps: “The tests pass too!”

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: 1012b57d-49f3-4dc7-92f0-b5755b5390e9
Commits

Reviewing files that changed from the base of the PR and between c226287 and e62283f.

Files selected for processing: 2

  • packages/gooddata-eval/src/gooddata_eval/core/chat/sse_client.py
  • packages/gooddata-eval/tests/test_chat_render.py

Included review availability: This review used your included allowance. 2 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour.

@codecov

codecov Bot commented Nov 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.15%. Comparing base (c226287) to head (e62283f).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1926   +/-   ##
=======================================
  Coverage   86.15%   86.15%           
=======================================
  Files         344      344           
  Lines       25874    25876    +2     
=======================================
+ Hits        22291    22293    +2     
  Misses       3583     3583           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant