Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions RESEARCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ This document records the empirical findings that inform the design of `create-d
- **When proposing a change** to `create-dev-loop.md`, the generated skill template, or a phase definition: cite the relevant finding(s) below in the PR description. If no finding applies, say so explicitly rather than acting on intuition.
- **When new research surfaces** that affects the project's design, add a finding here. Include the citation, key numbers, confidence level, and the specific implication for this project.
- **When a finding is superseded** by stronger evidence or contradicted by replication, update it in place — don't silently leave stale claims. Strike-through or "Superseded by:" notes are preferred over deletion when the change in understanding is itself informative.
- **When a finding gets implemented**, add an entry under that finding's **Implementations** subsection recording the PR number, the date shipped, and an observed-effect placeholder. The format is:
- **When a finding gets implemented**, add an entry under that finding's **Implementations** subsection recording the PR number, the date shipped (the PR's merge date in UTC, as GitHub records it in `mergedAt`), and an observed-effect placeholder. The format is:
```
**Implementations.**
- PR #N (one-line summary): shipped YYYY-MM-DD. Observed effect: pending — needs N cycles of data.
Expand Down Expand Up @@ -47,10 +47,10 @@ Last reviewed: 2026-08-04.
- PR #34 (Structure Phase 9 self-audit prompts and label gap issues): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. Extends the rubric-over-free-form pattern from Phase 4 (self-review) to Phase 9 (self-audit), with a 6-item structured checklist and a 5-label taxonomy for filed gap issues.
- PR #54 (Apply consolidated template-rule backlog, #44/#46/#47): shipped 2026-06-14. Observed effect: pending — needs N cycles of data. Sharpens the external-anchor definition (#44: UNVERIFIED handling when the anchor can't run; #46: green CI is not verification when CI's scope excludes the changed files) and grounds the Tests-fix rubric item empirically (#47: stash-and-run FAIL→PASS, not scored from reasoning alone) — directly targeting the self-preference failure mode this finding's evidence documents for judgment-based rubric scoring.
- PR #57 (Fix documentation drift across the three sources of truth): shipped 2026-07-25. Observed effect: pending — needs N cycles of data. Restores `{{EXTERNAL_SIGNAL_LABEL}}` in the Phase 1 carried-over-PR bullet, where PR #54 had reintroduced a hardcoded "CI must be green first" after PR #28 generalized the anchor. An anchor named `CI` in a repo that has no CI is an anchor the loop cannot check, which degrades the self-review to exactly the unanchored critique this finding shows is neutral-to-harmful.
- PR #67 (Add Tests-fix CI-revert fallback ladder): shipped 2026-07-26. Observed effect: pending — needs N cycles of data. Closes a gap left by PR #54: the Tests-fix rubric item required a local `git stash` experiment with no substitute when the local anchor can't run, even though the same PR's UNVERIFIED handling fully anticipates that case. Without a defined fallback, Tests-fix silently degraded back to unanchored judgment on exactly the repos this finding is about — confirmed twice, in two separate PRs opened by a generated child skill running against a repo whose local test anchor could not be run.
- PR #67 (Add Tests-fix CI-revert fallback ladder): shipped 2026-07-27. Observed effect: pending — needs N cycles of data. Closes a gap left by PR #54: the Tests-fix rubric item required a local `git stash` experiment with no substitute when the local anchor can't run, even though the same PR's UNVERIFIED handling fully anticipates that case. Without a defined fallback, Tests-fix silently degraded back to unanchored judgment on exactly the repos this finding is about — confirmed twice, in two separate PRs opened by a generated child skill running against a repo whose local test anchor could not be run.
- PR #119 (Harden Phase 8's rebase fence and state the merge-authorization precedence rule): shipped 2026-08-22. Observed effect: pending — needs N cycles of data. Two more ways the anchor silently weakens at the merge phase: the post-rebase verification ran the test command alone, so a required check covering a linter or static analysis was never part of the signal a force-push committed to (#115); and nothing stated whether a run-level merge pre-authorization could stand in for a Phase 4 UNVERIFIED result (#118). Both are the same failure this finding's PR #54 entry records for CI scope — an anchor that does not cover the changed surface is not an anchor — applied to permission rather than coverage in the second case.
- PR #121 (Define the structurally-red-anchor path and require the anchor to have executed tests): shipped 2026-08-16. Observed effect: pending — needs N cycles of data. Extends the same argument as PR #54's "green CI is not verification when CI's scope excludes the changed files" to two further ways an anchor can carry no information: a required job that has *never* succeeded on the base branch (observed on a repo whose CI jobs have failed on empty credentials since February 2026, so every cycle terminated in a hand-off regardless of quality), and a build tool that reports success having executed zero tests (`:test NO-SOURCE` still prints `BUILD SUCCESSFUL`). Both were previously read as ordinary signals — the first as a blocking red, the second as a green — which is unanchored critique wearing an anchor's clothes.
- PR #136 (Make the checkout form Phase 4's default Tests-fix revert and gate ladder rung 1): shipped 2026-08-25. Observed effect: pending — needs N cycles of data. Repairs the mechanism PR #54's entry above introduced. The prescribed `git stash push` revert ran against a tree Phase 3 had already committed and Phase 4 had already pushed, so it saved nothing, the check then passed against the still-fixed tree, and the no-op was naturally scored as a false-negative in the *test* — an anchor reporting a verdict it never measured, which is precisely the failure this finding warns about. The checkout form becomes the default because it is correct in the state Phase 4 actually finds, and the revert must now be confirmed to have changed the tree before either half is trusted. Separately, the ladder rung PR #67 added is gated against headless dispatch, where an interrupted session leaves the branch head undoing its own fix and the substitute anchor damages the thing it substitutes for.
- PR #136 (Make the checkout form Phase 4's default Tests-fix revert and gate ladder rung 1): shipped 2026-08-27. Observed effect: pending — needs N cycles of data. Repairs the mechanism PR #54's entry above introduced. The prescribed `git stash push` revert ran against a tree Phase 3 had already committed and Phase 4 had already pushed, so it saved nothing, the check then passed against the still-fixed tree, and the no-op was naturally scored as a false-negative in the *test* — an anchor reporting a verdict it never measured, which is precisely the failure this finding warns about. The checkout form becomes the default because it is correct in the state Phase 4 actually finds, and the revert must now be confirmed to have changed the tree before either half is trusted. Separately, the ladder rung PR #67 added is gated against headless dispatch, where an interrupted session leaves the branch head undoing its own fix and the substitute anchor damages the thing it substitutes for.
- PR #138 (Ground Phase 4's Docs score in a real pass): shipped 2026-08-29. Observed effect: pending — needs N cycles of data. The **Docs** rubric item named a check that Phase 7 did not perform until three phases later, so the only way to score it on time was from judgment — an anchor-shaped item with no anchor behind it, the same defect as PR #121's zero-executed-tests green, relocated from the signal to the schedule. The table pass now runs at Phase 4, where the score is taken, and Phase 7 is reduced to a re-check of whatever the review phases changed. Phase 7 is also required to record that it ran, since a clean sweep that leaves no artifact is indistinguishable from a skipped one — which is the auditability half of the same problem.

---
Expand Down Expand Up @@ -99,8 +99,8 @@ Last reviewed: 2026-08-04.
- PR #31 (Localization + regression gates): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. Adds a Phase 3 "Localization verification" step (enumerate files, `test -f`, grep for the named surface area) and a Phase 8 regression gate (bug-fix `Closes #N` requires a new/modified test or validation step). Generalized to projects whose external anchor is manual validation rather than tests.
- PR #52 (Add documentation-accuracy and unit-test-expansion work-mode stages to Phase 2): shipped 2026-06-08. Observed effect: pending — needs N cycles of data. Adds Stage B, a characterization-test work-mode that locks in current behavior and builds the regression guards this finding argues are the only way to distinguish a real fix from a coincidental patch — and forbids changing production code under a test-only cycle, so an apparent bug becomes a filed issue rather than an unverified patch.
- PR #108 (Broaden Step 2 to find the PR template under every accepted spelling): shipped 2026-08-09. Observed effect: pending — needs N cycles of data. Widens Step 2's PR-template bullet from a single lowercase path to every spelling and location GitHub accepts. Applies the finding one level earlier than PR #31 does: before a generated skill can localize within its repo, Step 2 must localize the repo's own convention files, and a silently under-matching file list loses that input with no trace.
- PR #136 (Make the checkout form Phase 4's default Tests-fix revert and gate ladder rung 1): shipped 2026-08-25. Observed effect: pending — needs N cycles of data. Protects the regression gate this finding motivated in PR #31. That gate is satisfied by the Phase 4 Tests-fix experiment, which prescribed a `git stash push` revert that saves nothing on the clean tree Phase 4 always finds; the resulting evidence was manufactured rather than collected, so a coincidental patch and a real fix were indistinguishable exactly where this finding says they must be told apart. The checkout form becomes the default, and the revert must be confirmed to have changed the tree before the FAIL→PASS pair is scored.
- PR #162 (Require Stage B characterization tests to be shown failing against a mutation): shipped 2026-10-19. Observed effect: pending — needs N cycles of data. Extends the regression evidence this finding calls for to Stage B. Tests-fix only runs for bug fixes, so a characterization test that could never fail was indistinguishable from a real guard. Stage B now requires at least one new test per target to fail against a temporary, plausible mutation of the code it characterizes, and a Phase 4 **Tests-mutation** rubric item scores it.
- PR #136 (Make the checkout form Phase 4's default Tests-fix revert and gate ladder rung 1): shipped 2026-08-27. Observed effect: pending — needs N cycles of data. Protects the regression gate this finding motivated in PR #31. That gate is satisfied by the Phase 4 Tests-fix experiment, which prescribed a `git stash push` revert that saves nothing on the clean tree Phase 4 always finds; the resulting evidence was manufactured rather than collected, so a coincidental patch and a real fix were indistinguishable exactly where this finding says they must be told apart. The checkout form becomes the default, and the revert must be confirmed to have changed the tree before the FAIL→PASS pair is scored.
- PR #162 (Require Stage B characterization tests to be shown failing against a mutation): shipped 2026-10-20. Observed effect: pending — needs N cycles of data. Extends the regression evidence this finding calls for to Stage B. Tests-fix only runs for bug fixes, so a characterization test that could never fail was indistinguishable from a real guard. Stage B now requires at least one new test per target to fail against a temporary, plausible mutation of the code it characterizes, and a Phase 4 **Tests-mutation** rubric item scores it.

---

Expand Down
Loading