Skip to content

feat(optimize-skill): prove pointers and estimate what a run reads - #2000

Merged
potiuk merged 1 commit into
apache:mainfrom
potiuk:perf/optimize-skill-pointer-checks
Nov 5, 2026
Merged

potiuk merged 1 commit into
apache:mainfrom
potiuk:perf/optimize-skill-pointer-checks

Conversation

@potiuk

@potiuk potiuk commented Nov 5, 2026

Copy link
Copy Markdown
Member

Summary

New optimize-skill guidance:

  • Name every trigger in a pointer, leave the heading in place, and keep the sibling in the same directory.
  • Add a pointer-only eval whose output spec names no file, and run it against a copy with the pointers deleted.
  • Before cutting prose because a tool "already does this", compare the two line by line; where they disagree, fix the tool first, with a failing test.
  • Estimate the run cost for the routine and heaviest paths, keeping tool output separate and labelling it a guess.

The guidance sits inside the existing Split section, so no heading is added and no surface_hash moves.

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below

Test plan

  • prek run --all-files --skip workspace-pytest passes, including lychee
  • workspace-pytest passes for the changed files
  • Validator exits 0; no surface_hash changed
  • optimize-skill evals: 7/7 before and after
  • New pointer-routing steps, 8 suites: 19/19 on the real skills; with the pointers deleted, all 11 cases that need a file fail and all 8 that need none pass

security-model-verify case-1 failed once on the real skill, returning two files. Step 6 uses helper.md and templates.md, so the fixture now says they are already open, with the same expected answer. It then passed 3/3 and failed on the copy without pointers. The runner does not show the extra file, so the cause is likely rather than confirmed.

Linked issues

Follows #1996, #1997, #1998.

Notes for reviewers

  • The optimize-skill suite cannot prove the guidance works. It only shows that diagnosis did not change.
  • Sweep ranking is unchanged. Run cost is reported and preferred when choosing a pass, but Step 1 still ranks by cap overflow and smells, which is what step-diagnose reads.
  • pointer-routing records a stated decision. The agent says which files it would open; it does not run with file tools.
  • Single model. No cross-model pass and no adversarial review were run.

🤖 Generated with Claude Code

Eval suites that paste moved text into the prompt with also_include show
a rule still works, never that an agent reaches it. Tell optimize-skill
to prove each pointer with a pointer-only eval and a copy with the
pointers deleted, to compare prose with a tool line by line before
cutting it, and to estimate a run cost per path, since the SKILL.md body
alone understates what an invocation reads.

Add pointer-routing steps for the pointers still untested: the
issue-stale-sweep failure-modes pointer and seven pre-PR review sites.

Generated-by: Claude Opus 5
@potiuk potiuk added the family:utilities utilities skills label Nov 5, 2026
@potiuk
potiuk enabled auto-merge (squash) November 5, 2026 01:43
@github-actions github-actions Bot added family:security security-* skills family:setup setup-* skills family:tools tools/* family:docs Docs, MISSION.md, READMEs family:issue issue-* skills capability:triage Sweep + classify + propose disposition capability:review Deep per-item code review or contributor mentoring capability:intake Import external signal + sync tracker state capability:platform Framework / agent substrate skills (install, verify, doctor, override, status, setup bootstrap) capability:authoring Skills that author or maintain other skills: write-skill, optimize-skill substrate:framework-dev Tool substrate: build / validate / eval the framework itself labels Nov 5, 2026
@potiuk
potiuk disabled auto-merge November 5, 2026 01:47
@potiuk
potiuk enabled auto-merge (squash) November 5, 2026 01:47
@potiuk
potiuk merged commit 4ce94a7 into apache:main Nov 5, 2026
10 checks passed
potiuk added a commit that referenced this pull request Nov 5, 2026
Describe the ASF network scoping from #1978, including the claim it made
false that containers share the sandbox's host allow-list. Record the run
cost and pointer proof from #2000, the verify-rc pane from #2007, and the
nine-skill pr-management family.

Recheck every Known gaps bullet in eleven specs: delete those now closed,
rewrite the stale ones, and annotate the rest with gap-closes-when.

Generated-by: Claude Opus 5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capability:authoring Skills that author or maintain other skills: write-skill, optimize-skill capability:intake Import external signal + sync tracker state capability:platform Framework / agent substrate skills (install, verify, doctor, override, status, setup bootstrap) capability:review Deep per-item code review or contributor mentoring capability:triage Sweep + classify + propose disposition family:docs Docs, MISSION.md, READMEs family:issue issue-* skills family:security security-* skills family:setup setup-* skills family:tools tools/* family:utilities utilities skills substrate:framework-dev Tool substrate: build / validate / eval the framework itself

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant