[Lanre Shittu (Shizoqua)]

Summary

Fixes #374.

SearchQA's sub_em gave full credit to an empty answer. A prediction that normalizes to empty (an empty <answer></answer>, or an answer such as The.) scored sub_em = 1.0 against any gold answer, and a gold answer that normalizes to empty matched every prediction. An empty normalized string now only matches another empty string.

User impact: without this fix, SearchQA results report empty or article-only answers as substring matches, which inflates the reported sub_em rate.

Proof: the regression tests fail on upstream eb959c3 and pass with this change.

Root cause

sub_em() in skillopt/envs/searchqa/evaluator.py returns 1.0 when norm_gold in norm_pred or norm_pred in norm_gold. After normalize_answer() strips punctuation and articles, either side can be "", and "" in s is true for every s. exact_match() and f1_score() do not have this problem: an empty prediction only matches an empty gold there.

Changes

• sub_em() skips the substring test when either normalized side is empty, and returns 1.0 for that pair only when both are empty. This matches exact_match() and f1_score().
• Non-empty substring matches in both directions are unchanged.
• New tests/test_searchqa_evaluator.py, testing through evaluate(), which is what the SearchQA rollout calls.
• CHANGELOG entry under Unreleased / Fixed.

Reproduction

from skillopt.envs.searchqa.evaluator import evaluate, sub_em

print(evaluate("<answer></answer>", ["Paris"])["sub_em"])
print(evaluate("<answer>The.</answer>", ["Paris"])["sub_em"])
print(sub_em("London", ["The"]))
print(sub_em("Paris", ["Paris, France"]))

Before: 1.0, 1.0, 1.0, 1.0

After: 0.0, 0.0, 0.0, 1.0

Testing

• python -m pytest tests/test_searchqa_evaluator.py -q: on upstream eb959c3, 4 failed (assert 1.0 == 0.0) and 2 passed; with the fix, 6 passed.
• python -m pytest tests/test_searchqa_evaluator.py tests/test_searchqa_rollout_failfast.py tests/test_materialize_searchqa.py tests/test_qwen_backend.py -q: 40 passed.
• python -m pytest -q: 2039 passed, 12 skipped, 4 xfailed.
• python -m ruff check on the changed files: all checks passed.
• python -m mkdocs build --strict: passed.

Risk and compatibility

sub_em is recorded in each SearchQA rollout result but does not feed hard or soft (those come from EM and F1), so training and the validation gate are unchanged. The only visible change is that reported sub_em no longer counts empty answers as matches. No backend, config key, file format or CLI change.

[Yifan Yang (Yif-Yang)]

Merged, thank you. sub_em now matches an empty normalized string only against another empty one, consistent with exact_match and f1_score. I confirmed that the three cases from #374 now score 0, and that ordinary substring matches still score 1. The new tests fail on main and pass here, and the full suite passes. As noted in #374, sub_em is a reported metric only and does not feed hard/soft. Closes #374.