[RohithPariki]

Problem

In SkillOpt-Sleep, offline task replay (replay_one) previously measured latency and token consumption strictly around backend.attempt(). When tasks use semantic evaluators (rubric scoring or LLM-as-a-judge), backend.judge() is invoked afterwards to assess response quality. The computational latency, token spend, and model attribution of these judge evaluations were unmeasured and omitted from consolidation summaries and nightly reports (report.json and report.md). Consequently, nightly runs had no visibility into evaluation costs vs task generation attempts, and dual-backend setups could not distinguish optimizer/judge model identity from target execution.

In #283, maintainer Yifan Yang (@Yif-Yang) noted:

"A small first PR that records judge-side latency, request count and model identity separately in the night's report would be welcome."

Solution

1. Types (skillopt_sleep/types.py):
   • Added judge_latency_ms: float = 0.0, judge_tokens: int = 0, and judge_model: str = "" to ReplayResult.
   • Added judge_requests: int = 0, judge_latency_ms: float = 0.0, judge_tokens: int = 0, and judge_model: str = "" to SleepReport.
2. Backend Attribution (skillopt_sleep/backend.py):
   • Added judge_model_name property to Backend (resolving model or name) and DualBackend (delegating to optimizer.judge_model_name).
3. Replay Accounting (skillopt_sleep/replay.py):
   • Instrumented evaluator execution with time.time() and backend token accounting (token_delta / delta of tokens_used()).
   • Categorized deterministic rule-based checks as judge_model="rule" with 0 tokens.
   • For non-rule evaluation, attributed model identity to backend.judge_model_name and recorded latency and token deltas.
   • Preserved backwards compatibility and call-local token contracts across replay batching.
4. Consolidation & Cycle Aggregation (skillopt_sleep/consolidate.py, skillopt_sleep/cycle.py):
   • Aggregated judge requests, cumulative latency, and tokens across baseline, training, trial, and final validation replay passes.
   • Aggregated held-out test evaluation and multi-skill fanout group outcomes into the night's final SleepReport.
   • Rendered judge statistics in report.md (- judge: <requests> request(s), <latency>ms, <tokens> tokens (model: <model>)) and structured report.json.
   • Logged judge metrics in evidence.log("cycle", "end", ...).
5. Changelog:
   • Documented under ## [Unreleased] / ### Added.

Testing

• Added unit tests in tests/test_sleep_judge_observability.py:
  • test_replay_one_rule_judge_records_rule_model_and_zero_tokens: Verifies rule checks record zero tokens and "rule" model.
  • test_replay_one_rubric_judge_measures_latency_and_tokens: Verifies rubric evaluation tracks latency, tokens, and model attribution.
  • test_dual_backend_judge_model_attribution: Verifies DualBackend routes judge_model_name to the optimizer.
  • test_consolidate_aggregates_judge_metrics: Verifies consolidation passes accumulate judge metrics into ConsolidationResult.
  • test_render_report_md_includes_judge_observability: Verifies markdown report formatting for judge metrics.
  • test_run_sleep_cycle_records_judge_metrics: Verifies end-to-end run_sleep_cycle populates report judge metrics.
• Ran test suite: pytest tests/test_sleep_judge_observability.py (6/6 passed in 0.48s).
• Ran regression tests: pytest tests/test_sleep_dream_metric.py tests/test_sleep_adopt_skill_subset.py (102 passed, 4 skipped).
• Verified linter: python -m ruff check passed cleanly.

Risk

Low. All additions are additive and preserve backwards compatibility with existing backends and report schemas. Existing non-instrumented mock backends report 0 tokens and negligible latency without breaking existing assertions.

Fixes #283