Catalogue the Incubator skills and check curated sources weekly

.github/workflows/skill-source-drift.yml

@@ -0,0 +1,116 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Checks the curated skill sources in `organizations/*/skill-sources.md`
+# against what their repositories ship today — see
+# `tools/dev/check-skill-source-drift.py`. A source adding a skill, re-laying
+# one out as a directory, tagging one for the first time, or tagging a newer
+# version of a pinned one changes nothing in this repository, so without this
+# the catalogue and the pins go stale unnoticed.
+#
+# Weekly (Monday), and from the Actions tab (*Run workflow*).
+#
+# ONE ISSUE, UPDATED IN PLACE. On drift, the open issue with the fixed title
+# below gets the new report as its body, or one is opened. When everything is
+# back in sync, that issue is closed with a comment. Nothing here edits the
+# catalogue or moves a pin: both need a human to read the source first, and a
+# moved tag in particular is a reason to look, not to re-pin.
+---
+name: skill-source drift
+
+on:  # yamllint disable-line rule:truthy
+  schedule:
+    - cron: "41 5 * * 1"
+  workflow_dispatch:
+
+permissions: {}
+
+env:
+  TITLE: "Skill-source drift: curated sources disagree with upstream"
+
+jobs:
+  check:
+    name: Check curated skill sources
+    runs-on: ubuntu-slim
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
+        with:
+          persist-credentials: false
+          fetch-depth: 1
+
+      - uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e  # v10.3.2
+        with:
+          enable-cache: true
+          cache-suffix: "skill-source-drift"
+
+      - name: Compare the catalogues and pins with their sources
+        id: drift
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
+        run: |
+          set -uo pipefail
+          uv run --project tools/dev python tools/dev/check-skill-source-drift.py \
+            > "$RUNNER_TEMP/report.md"
+          status=$?
+          cat "$RUNNER_TEMP/report.md" >> "$GITHUB_STEP_SUMMARY"
+          # 0 in sync, 1 drift, anything else: the check itself failed.
+          if [ "$status" -gt 1 ]; then
+            exit "$status"
+          fi
+          echo "drift=$status" >> "$GITHUB_OUTPUT"
+
+      - name: Open, update or close the drift issue
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+          DRIFT: ${{ steps.drift.outputs.drift }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+        run: |
+          set -euo pipefail
+          existing=$(gh issue list --repo "$REPO" --state open --search "in:title \"$TITLE\"" \
+            --json number,title --jq ".[] | select(.title == env.TITLE) | .number" | head -n1)
+
+          if [ "$DRIFT" = "0" ]; then
+            if [ -n "$existing" ]; then
+              gh issue close "$existing" --repo "$REPO" \
+                --comment "Back in sync as of $RUN_URL."
+            fi
+            exit 0
+          fi
+
+          {
+            cat "$RUNNER_TEMP/report.md"
+            echo
+            echo "---"
+            echo
+            echo "Found by the weekly [skill-source drift]($RUN_URL) check."
+            echo "To resolve: read the change in the source repository, then update the catalogue"
+            echo "and, where a skill is ready, the descriptor's \`ref\`, \`commit\` and \`provides\`"
+            echo "in \`organizations/<org>/skill-sources.md\` (and \`docs/skill-sources/registry.md\`)."
+            echo "This issue closes itself once the next run finds everything in sync."
+          } > "$RUNNER_TEMP/body.md"
+
+          if [ -n "$existing" ]; then
+            gh issue edit "$existing" --repo "$REPO" --body-file "$RUNNER_TEMP/body.md"
+          else
+            gh issue create --repo "$REPO" --title "$TITLE" \
+              --label "family:setup" --label "family:ci" --body-file "$RUNNER_TEMP/body.md"
+          fi

docs/skill-sources/registry.md

@@ -38,9 +38,15 @@ Sources an organization vouches for, declared in
 `organization: <org>`. Curation is still not installation — the adopter
 opts each one in.
 
+An org file may also carry a *catalogue* of everything a source ships, pinned or not,
+between `<!-- BEGIN SKILL-SOURCE CATALOGUE: <id> -->` markers.
+The [`skill-source-drift`](../../.github/workflows/skill-source-drift.yml) workflow
+checks every catalogue and pin against its source each week
+and opens an issue when they disagree, so a new or retagged skill upstream does not go unnoticed.
+
 | Organization | Curated sources | File |
 |---|---|---|
-| Apache Software Foundation | `apache-incubator`: [`apache/incubator`](https://github.com/apache/incubator) (`releasecheck`) | [`organizations/ASF/skill-sources.md`](../../organizations/ASF/skill-sources.md) |
+| Apache Software Foundation | `apache-incubator`: [`apache/incubator`](https://github.com/apache/incubator) — pinned: `releasecheck`; also catalogued: the IPMC, mentor and podling skills in `tools/skills/` (graduation, retirement, report review, release-vote review, maturity model, …), with how each installs today ([catalogue](../../organizations/ASF/skill-sources.md#what-the-incubator-ships)) | [`organizations/ASF/skill-sources.md`](../../organizations/ASF/skill-sources.md) |
 | Independent (no formal governing body) | *(none listed yet)* | [`organizations/independent/skill-sources.md`](../../organizations/independent/skill-sources.md) |
 
 ## Community / external sources

organizations/ASF/skill-sources.md

@@ -28,15 +28,93 @@ table in [`docs/skill-sources/registry.md`](../../docs/skill-sources/registry.md
 ### Apache Incubator
 
 Skills the Incubator PMC maintains in
-[`apache/incubator`](https://github.com/apache/incubator) for podlings,
-mentors and the IPMC. The first is `releasecheck`, which checks a podling
-release candidate before or during its vote. The same repo is also a plugin
-marketplace, so the skills install without Magpie too:
-`claude plugin marketplace add apache/incubator`.
-
-The source is pinned to a tag, because the rest of the Incubator repo
-changes often. A new skill version gets a new `releasecheck-<version>` tag,
-and this pin moves only when the tag is updated here.
+[`apache/incubator`](https://github.com/apache/incubator) under
+[`tools/skills/`](https://github.com/apache/incubator/tree/HEAD/tools/skills)
+for podlings, mentors and the IPMC.
+There are two ways to install them, and they do not conflict:
+
+- **Through Magpie, for any agent.**
+  A project that trusts the source gets each pinned skill with
+  `/magpie-setup:setup skill-sources`, wired into `.agents/skills/` and relayed
+  to every agent's skills directory like a framework skill.
+  This is the path for an adopted repo, and the only one for agents with no
+  marketplace (OpenCode, Pi, Goose).
+- **From the Incubator's own marketplaces, without Magpie.**
+  The repo carries a plugin catalog per agent; the *Marketplaces* column below
+  says which ones list a skill:
+
+  | Marketplace | Add it | Catalog in `apache/incubator` |
+  |---|---|---|
+  | `claude-code` | `/plugin marketplace add apache/incubator` | `.claude-plugin/marketplace.json` |
+  | `codex` | `codex plugin marketplace add apache/incubator` | `.agents/plugins/marketplace.json` |
+  | `copilot` | add `apache/incubator` as a VS Code / GitHub Copilot plugin marketplace | `marketplace.json` |
+
+  Updates then arrive through the agent's own plugin update.
+
+**Pinning and updates.**
+The Incubator versions each skill on its own, tagging it `<skill>-<version>`
+(for example `releasecheck-0.5.1`), so each skill is curated as its own
+descriptor pinned to its own tag: `apache-incubator` for `releasecheck`, and
+`apache-incubator-<skill>` for each skill pinned after it.
+A skill kept as a plugin directory (`tools/skills/<skill>/skills/<skill>/SKILL.md`,
+the layout Codex needs to find it) is pinned with
+`skills_root: tools/skills/<skill>/skills`; the drift check still scans
+`tools/skills/` for its siblings.
+A pin moves only when it is updated here, and an adopter's committed pin moves
+only when the adopter bumps it; `/magpie-setup:setup skill-sources` then
+re-fetches and re-verifies the new tag.
+
+#### What the Incubator ships
+
+Every skill in `tools/skills/`, and how it can be installed today.
+Most of these skills read the Incubator's own MCP servers (reports, podlings,
+health, mail archives); each skill names the ones it needs, and the
+[Incubator skills README](https://github.com/apache/incubator/blob/HEAD/tools/skills/README.md)
+says how to set them up.
+
+<!-- BEGIN SKILL-SOURCE CATALOGUE: apache-incubator -->
+| Skill | What it does | Install through Magpie | Marketplaces |
+|---|---|---|---|
+| `releasecheck` | Checks a podling release candidate before or during its vote; findings for a person to review, never a vote. | `pinned` | `claude-code`, `codex` |
+| `maturity-model-assessment` | Builds an evidence pack against the 38 items of the Apache Project Maturity Model, marking each evidenced, proxy or human; never a score. | `untagged` | — |
+| `graduation-self-assessment` | Compares the `ipmc` MCP's graduation-readiness verdict with what the podling says in its own reports, checked against the mail archives. | `single-file` | — |
+| `graduation-packet-builder` | Assembles a podling's graduation case and drafts the DISCUSS email and the board resolution. | `single-file` | — |
+| `retirement-shepherd` | Assesses whether a stalled podling should retire and drafts the discussion, the announcement and the shutdown checklist. | `single-file` | — |
+| `monthly-report-review` | The monthly IPMC report review: who has not filed, narratives against health data, draft shepherd comments. | `single-file` | — |
+| `release-vote-reviewer` | Walks a release vote thread on general@ and drafts the +1 or the -1 with reasons. | `single-file` | — |
+| `board-report` | Drafts the VP Incubator's quarterly report to the ASF board. | `single-file` | — |
+| `branding-sweep` | Trademark and branding check across all current podlings, as a ranked fix list. | `single-file` | — |
+| `podling-onboarding` | Reviews a new podling's proposal against the checklist and bootstraps its tracking. | `single-file` | — |
+| `podling-name-clearance` | Runs the podling name search end to end and writes it up in the required format. | `single-file` | — |
+| `asf-list-archaeology` | Not yet published (kept in `tools/skills/draft/`). | `draft` | — |
+<!-- END SKILL-SOURCE CATALOGUE: apache-incubator -->
+
+What *Install through Magpie* means:
+
+- **`pinned`** — in a descriptor's `provides` below.
+  An adopter that trusts that descriptor gets it with
+  `/magpie-setup:setup skill-sources`.
+- **`untagged`** — already a `<skill>/SKILL.md` directory, which is the layout
+  Magpie fetches, but there is no release tag to pin yet.
+  It moves to `pinned` once the Incubator tags it.
+- **`single-file`** — a single `<skill>.md` with its supporting files kept in
+  shared `scripts/`, `templates/` and `references/` directories.
+  Magpie wires a skill directory, and a marketplace installs one, so neither
+  can install these as they are; moving each into its own directory, as
+  `releasecheck` already is, is proposed to the Incubator.
+  Until then, build one with the Incubator's own packager and unpack it:
+  `python3 tools/skills/package-skills.py <skill>` in an `apache/incubator`
+  checkout, then `unzip dist/<skill>.skill -d ~/.claude/skills/`.
+- **`draft`** — not published by the Incubator; listed so it is not missed
+  when it is.
+
+The table is checked against `apache/incubator` every week by the
+[`skill-source-drift`](../../.github/workflows/skill-source-drift.yml)
+workflow, which runs
+[`tools/dev/check-skill-source-drift.py`](../../tools/dev/check-skill-source-drift.py)
+and opens an issue when a skill was added, removed, re-laid-out or tagged,
+when a marketplace catalog gained or lost it, or when a newer tag exists for a
+pin — which is when a maintainer moves the pin here.
 
 **Podlings: prefer `releasecheck` over `release-verify-rc`.** Magpie's
 `release-verify-rc` does not distinguish a podling release from a TLP one, so

tools/dev/README.md

@@ -64,6 +64,7 @@ installable for other members to depend on it.
 | [`check-family-plugins.py`](check-family-plugins.py) | Validates the marketplace plugins against the skills' `family:` frontmatter — version parity across every ecosystem manifest, Agent Plugins 1.0 conformance, and one well-formed per-family plugin whose `skills/` symlinks match the family exactly. `--fix` regenerates them, which is how the prek hook runs it. |
 | [`check-claude-mods.sh`](check-claude-mods.sh) | Runs `claude plugin validate --strict` and `claude plugin test` on every plugin whose `hooks/hooks.json` declares `modules` — the Claude Code mods ([`docs/when-to-use-mods.md`](../../docs/when-to-use-mods.md)). Claude Code skips a hooks module that fails its static analysis, so without this a mod that cannot load passes every other check and does nothing. A mod with no `*.test.ts` fails. Runs as the `check-claude-mods` prek hook, whose `language: node` env carries a pinned Claude Code. The hook is `stages: [manual]`, so commits never fetch the ~200 MB binary: the `prek` workflow runs it in a step of its own, and locally it is `prek run check-claude-mods --hook-stage manual --all-files`. A no-op when the tree has no mods. |
 | [`bump-hook-npm-pins.py`](bump-hook-npm-pins.py) | Moves the npm packages pinned in `.pre-commit-config.yaml` hooks' `additional_dependencies` — today the Claude Code CLI behind `check-claude-mods` — to the newest stable release past the package's cooldown: 12 hours for Claude Code, which ships several times a day and replaces a bad release within hours, 7 days otherwise, matching `.github/dependabot.yml`. Claude Code is a dev-only tool here — CI runs it to check the mods, and nothing from it ships. Dependabot's `pre-commit` ecosystem bumps a hook repo's `rev` but never these pins. Never moves a pin backwards. `--check` reports without writing. Run three times a week by [`bump-hook-npm-pins.yml`](../../.github/workflows/bump-hook-npm-pins.yml), which keeps a single draft PR open and updates it in place until it is merged. |
+| [`check-skill-source-drift.py`](check-skill-source-drift.py) | Checks every curated skill source in `organizations/*/skill-sources.md` against what its repository ships today, through the GitHub API: the pinned tag still resolves to the recorded `commit`, every skill in `provides` is a `<skill>/SKILL.md` directory at that tag, no newer tag exists for the pin, and the source's *catalogue* (the table between `BEGIN`/`END SKILL-SOURCE CATALOGUE` markers) lists every skill the source ships with the status its layout gives it (`pinned`, `untagged`, `single-file`, `draft`) and the right marketplace column. A source adding, re-laying out or tagging a skill changes nothing here, so without it the catalogue goes stale unnoticed. Network-bound, so not a prek hook: the weekly [`skill-source-drift`](../../.github/workflows/skill-source-drift.yml) workflow runs it and keeps one issue open while there is drift. Exit 0 in sync, 1 drift, 2 the check could not run. Read-only. |
 | [`bump-dev-version.py`](bump-dev-version.py) | Moves the `.dev<YYYYMMDDHHMM>` stamp on `project.version` in the root `pyproject.toml` — the single authority every manifest mirrors. Only the edit: `check-family-plugins.py --fix` and `uv lock` still follow, so the three steps read the same whether a human or CI runs them. The stamp is UTC at minute resolution and has to *move* for adopters to pick anything up (`claude plugin update` compares version strings, so a frozen suffix is a silent no-op). Refuses a version with no `.dev` suffix rather than stamping a release. Called by [`bump-dev-version.yml`](../../.github/workflows/bump-dev-version.yml). |
 | [`gh-signed-commit.py`](gh-signed-commit.py) | Commits the working tree through GitHub's `createCommitOnBranch` mutation instead of `git commit` + `git push`, so the commit is signed by GitHub and shows as **Verified** with no key material in CI. Collects changed and deleted paths from `git status --porcelain -z`, decomposes renames (the mutation has no rename concept), and pins `expectedHeadOid` so a concurrent push fails the call rather than being overwritten. Used by [`bump-dev-version.yml`](../../.github/workflows/bump-dev-version.yml). |
 | [`check-placeholders.sh`](check-placeholders.sh) | Fails the build on hardcoded project references in skill and tool docs, which must use `<PROJECT>` / `<project>` / `<tracker>` / `<upstream>` instead. Carries both casings and matches spaced variants. |