diff --git a/README.md b/README.md index 744bae4..3f8a54e 100644 --- a/README.md +++ b/README.md @@ -114,6 +114,16 @@ Set each one up in this order: `km doctor` notes a missing `mounted-pins` environment in a brain with mounts, and either environment when it is open to more than the default branch. +### Running on your own runners + +Every job of the team workflows runs on `${{ vars.KM_RUNNER || 'ubuntu-latest' }}`: set the repository or organisation variable `KM_RUNNER` to one runner label (`my-runner`, say) and they run there; unset, they run on GitHub's runners. bump-mounted-pins reads its own variable `MOUNTED_PINS_RUNNER` first. A brain written by an earlier km gets the change in the shipped workflows through `km upgrade`; `ci.yml` and `staleness.yml` are the brain's own, so change their `runs-on` lines by hand to exactly the template's `runs-on: ${{ vars.KM_RUNNER || 'ubuntu-latest' }}`, and below it, if you like, exactly the template's `timeout-minutes: 15` (km then still counts that `ci.yml` as its own; with any other line `km upgrade` takes it for the brain's own and does not add approve-fork-ci or automerge). + +A plain Ubuntu 24.04 VM with git, curl and tar is enough for every workflow but bump-mounted-pins: its jobs run in a container, so its runner needs docker, usable by the runner's user; without docker set `MOUNTED_PINS_RUNNER` to `ubuntu-latest`, and bump-mounted-pins stays on GitHub's runners. Each job that runs `gh`, `jq` or `yq` first fetches the ones the runner lacks (mikefarah's yq v4: Ubuntu's `yq` package is another program), pinned by version and sha256, into the job's temp folder, without sudo; that step runs only while `KM_RUNNER` is set. A runner image that has gh, jq and mikefarah's yq saves those downloads. Where the step fetches gh, that gh runs in jobs that hold a write `GITHUB_TOKEN` (automerge's act without the App, main-guard's issue), as a pinned action would. automerge's act job never fetches them where it merges with the App (`KM_AUTOMERGE_APP`): its token is not handed to a downloaded tool, so there the runner's image must have gh and jq, or nothing is merged and the run fails. The km action installs its Python with `actions/setup-python`, which needs a writable tool cache and network access to GitHub and PyPI. Every job has a `timeout-minutes` (15 at most), so a stuck job does not hold a runner for GitHub's default 6 hours. + +Security: approve-fork-ci, review-request-guide and automerge run on `pull_request_target` and never check out the pull request's code, but ci does run on a fork's pull request. If GitHub passes the repository's variables to that run, a fork author's pull request runs on your hardware, in the network the runner sits in, and approve-fork-ci starts that run without a person when the pull request changes no control file (ci then runs km on the fork's docs, nothing from the checkout); otherwise fork ci stays on GitHub's runner. So use `KM_RUNNER` with ephemeral, isolated runners only: one job per fresh VM, with a network that gives the job no more than it needs. On a persistent or non-isolated runner, do not set `KM_RUNNER` while approve-fork-ci auto-approves. + +Known limit: whether GitHub passes repository and organisation variables to a fork's or Dependabot's `pull_request` run is not verified yet; until it is, read the security note as if it does. `km doctor` names the runner `KM_RUNNER` sets, and notes it when `ci.yml` does not read it. + ## Layout ``` diff --git a/skills/km/reference/setup.md b/skills/km/reference/setup.md index d687b0d..ea6ea81 100644 --- a/skills/km/reference/setup.md +++ b/skills/km/reference/setup.md @@ -40,7 +40,7 @@ A **team brain** is a shared, served knowledge base: one maintainer merges, the 2. In the empty repo (or an empty clone of a fresh remote): `km init . --team --name --desc "" --initials --folder process="" --folder ... [--rule "" ...]`. It writes the docs, indexes, `schema.local.yaml`, the ci, staleness, main-guard, approve-fork-ci, review-request-guide and bump-mounted-pins workflows (the last with its script `.github/scripts/bump-mounted-pins.py`), the pre-commit hook, a Dependabot config and a `.gitattributes` for LF, all pinned by commit to this km's release (or `--pin `), and refuses to run if any file it would write already exists, such as a README the host created with the repo: delete that first. 3. `git add -A`, then `km gen-index` (it reads tracked files only, so add first; it fills the root and folder lists), `git add -A` and `km validate`: expect **0 errors / 0 warnings**. 4. Commit: `chore: initialize (team served knowledge base)`. -5. Tell the user what km does not do: create the repo (private), push `main` (after `pre-commit install` the pre-push hook refuses a push to `main`: the first push of the fresh brain takes `--no-verify`, once; main-guard lets that first push into the empty repository through), give the team **read** access and the maintainer write, and ask members to add the brain to their personal brain with `/km brain add `. Branch protection on a private repo needs a paid GitHub plan; without it the one-merger rule is held by the permission model alone. Members with read access contribute by fork PR, and GitHub does not run Actions on fork PRs of a private repo until "Run workflows from fork pull requests" is enabled (repository or organization settings, Actions): without it those PRs get no CI. With it, the approve-fork-ci workflow approves the waiting ci run of a fork PR that touches no control file (the rule is in the workflow's header); any other fork PR waits for "Approve and run". Dependabot then proposes a PR whenever a newer km or another action is released; mounted brains move through bump-mounted-pins ("Mounted pins on a team brain"). Set the repository variable `KM_MAINTAINERS` to the maintainers' GitHub logins: review-request-guide then comments on a PR whose author asks a maintainer for review before it is `km:ready`, with what `/km pr` and the sign-off still need. +5. Tell the user what km does not do: create the repo (private), push `main` (after `pre-commit install` the pre-push hook refuses a push to `main`: the first push of the fresh brain takes `--no-verify`, once; main-guard lets that first push into the empty repository through), give the team **read** access and the maintainer write, and ask members to add the brain to their personal brain with `/km brain add `. Branch protection on a private repo needs a paid GitHub plan; without it the one-merger rule is held by the permission model alone. Members with read access contribute by fork PR, and GitHub does not run Actions on fork PRs of a private repo until "Run workflows from fork pull requests" is enabled (repository or organization settings, Actions): without it those PRs get no CI. With it, the approve-fork-ci workflow approves the waiting ci run of a fork PR that touches no control file (the rule is in the workflow's header); any other fork PR waits for "Approve and run". Dependabot then proposes a PR whenever a newer km or another action is released; mounted brains move through bump-mounted-pins ("Mounted pins on a team brain"). Set the repository variable `KM_MAINTAINERS` to the maintainers' GitHub logins: review-request-guide then comments on a PR whose author asks a maintainer for review before it is `km:ready`, with what `/km pr` and the sign-off still need. To run the team workflows on the team's own runners, set the repository or organization variable `KM_RUNNER` to a runner label; a fork's pull request then runs ci on that hardware, so only ephemeral runners fit (km's README, "Running on your own runners", lists what a runner needs). ## Mounted pins on a team brain (`bump-mounted-pins`) diff --git a/src/km/doctor.py b/src/km/doctor.py index a00ee86..7e574be 100644 --- a/src/km/doctor.py +++ b/src/km/doctor.py @@ -552,6 +552,41 @@ def check_environment(repo: str, name: str, if_missing: tuple[str, str] | None, f"Settings, Environments, {name}: deployment branches, selected branches, `{default}` only") +RUNS_ON_RUNNER = re.compile(r"(?m)^[ \t]*runs-on:[^#\n]*vars\.KM_RUNNER") + + +def check_runner(root: Path, repo: str) -> None: + """The variable KM_RUNNER of the repository, else of its organisation, as far as gh may read them, + beside km's team ci.yml (one that installs km through pasrom/km@): a note naming the runner when the + runs-on lines of ci.yml and staleness.yml read it, or which of them do not. Unset or unreadable, + nothing is said; a repository variable gh may not read is not stood in for by the organisation's.""" + files = [p for p in (root / ".github/workflows/ci.yml", root / ".github/workflows/staleness.yml") if p.is_file()] + texts = {p.name: p.read_text(encoding="utf-8", errors="replace") for p in files} + if "pasrom/km@" not in texts.get("ci.yml", ""): + return + for args in ((f"repos/{repo}/actions/variables/KM_RUNNER", "--jq", ".value"), + ("--paginate", f"repos/{repo}/actions/organization-variables", + "--jq", '.variables[] | select(.name == "KM_RUNNER") | .value')): + if (res := ask(f"for the variable KM_RUNNER of {repo}", "api", "--hostname", "github.com", *args, + status="note")) is None: + return + if res[0] == 0 and (label := printable(res[1].strip())): + break + if res[0] != 0 and "Not Found" not in res[1]: + return # not readable (a 403): the organisation's would mislead + else: + return + if ignoring := [name for name, text in texts.items() if not RUNS_ON_RUNNER.search(text)]: + report("note", f"the variable KM_RUNNER names the runner {label}, but the runs-on line of {' and '.join(ignoring)} " + f"does not read it: that workflow stays where the line says", + "set their runs-on to `${{ vars.KM_RUNNER || 'ubuntu-latest' }}` (km's README, \"Running on your own runners\")") + else: + report("note", f"the team workflows run on the runner {label} (the variable KM_RUNNER; bump-mounted-pins reads " + f"MOUNTED_PINS_RUNNER first): if GitHub passes the brain's variables to a fork's pull request run, " + f"that ci runs on it too, otherwise on GitHub's runner", + "use ephemeral, isolated runners only, one job per fresh machine (km's README, \"Running on your own runners\")") + + def main() -> int: ap = argparse.ArgumentParser(prog="km doctor", description="check that this computer is set up to work on a brain") ap.add_argument("--offline", action="store_true", help="skip the checks that need the network") @@ -571,6 +606,7 @@ def main() -> int: check_line_endings(root) if repo := check_github(origin, args.offline): check_environments(root, repo) + check_runner(root, repo) check_git_login(root, origin) check_km_version(root) check_plugins(root) diff --git a/src/km/templates/team/approve-fork-ci.yml b/src/km/templates/team/approve-fork-ci.yml index 10484bc..c67d087 100644 --- a/src/km/templates/team/approve-fork-ci.yml +++ b/src/km/templates/team/approve-fork-ci.yml @@ -42,9 +42,43 @@ jobs: if: >- github.event.pull_request.head.repo.full_name != github.repository && github.event.pull_request.base.ref == github.event.repository.default_branch - runs-on: ubuntu-latest + runs-on: ${{ vars.KM_RUNNER || 'ubuntu-latest' }} timeout-minutes: 10 steps: + # GitHub's runner image has gh, jq and yq; a self-hosted runner (KM_RUNNER) may not. A missing one is + # fetched, pinned by version and sha256, into this job's temp folder; the same step in every km workflow + - name: the tools this job runs, where the runner lacks them + if: vars.KM_RUNNER != '' + env: + TOOLS: gh jq + run: | + set -euo pipefail + bin="$RUNNER_TEMP/km-tools"; mkdir -p "$bin"; echo "$bin" >> "$GITHUB_PATH" + case "$(uname -s)-$(uname -m)" in Linux-x86_64) arch=amd64;; Linux-aarch64|Linux-arm64) arch=arm64;; *) arch=;; esac + for tool in $TOOLS; do + if [ "$tool" = yq ]; then # mikefarah's yq v4: Ubuntu's yq package is another program + yq --version 2>/dev/null | grep -q 'mikefarah/yq.* v4\.' && continue + else + command -v "$tool" >/dev/null && continue + fi + gh=https://github.com/cli/cli/releases/download/v2.103.3/gh_2.103.3_linux + jq=https://github.com/jqlang/jq/releases/download/jq-1.8.2/jq-linux + yq=https://github.com/mikefarah/yq/releases/download/v4.54.1/yq_linux + case "$tool-$arch" in + gh-amd64) url="${gh}_amd64.tar.gz" sum=4c806978ffce1cec70853302a56d70ac6da4f19bae7adca4e1a8ab00633dd91d;; + gh-arm64) url="${gh}_arm64.tar.gz" sum=ee32df2323c8847f5fac6cb6b4b69ca75828e9329a8f22f677021f7b3306ce63;; + jq-amd64) url="$jq-amd64" sum=b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f;; + jq-arm64) url="$jq-arm64" sum=8b85c817833814ddca00a144c33705546355afccf0cf39b188f3cdb48b852309;; + yq-amd64) url="${yq}_amd64" sum=8e34fc298390875de416e6a4afcb8cabeceb25d9aa8506c1a2f9353cf702ea5f;; + yq-arm64) url="${yq}_arm64" sum=189088da0c6429ec5178dfaab1a114805f6cab0b61b165ab236efedf1d57a71b;; + *) echo "::error::$tool is missing and km pins no build of it for $(uname -s) $(uname -m): install it on the runner"; exit 1;; + esac + curl -fsSL --retry 3 -o "$bin/$tool.get" "$url" + echo "$sum $bin/$tool.get" | sha256sum -c --quiet - + if [ "$tool" = gh ]; then tar -xzOf "$bin/gh.get" --wildcards '*/bin/gh' > "$bin/gh"; rm "$bin/gh.get" + else mv "$bin/$tool.get" "$bin/$tool"; fi + chmod +x "$bin/$tool"; echo "installed $tool from $url" + done - name: approve the CI run when the pull request changes nothing CI executes env: GH_TOKEN: ${{ github.token }} diff --git a/src/km/templates/team/automerge.yml b/src/km/templates/team/automerge.yml index 9a21e9c..5fc476f 100644 --- a/src/km/templates/team/automerge.yml +++ b/src/km/templates/team/automerge.yml @@ -79,13 +79,47 @@ jobs: || (github.event_name == 'pull_request_target' && (github.event.action == 'ready_for_review' || github.event.label.name == 'km:ready') && contains(github.event.pull_request.labels.*.name, 'km:ready'))) - runs-on: ubuntu-latest + runs-on: ${{ vars.KM_RUNNER || 'ubuntu-latest' }} timeout-minutes: 5 permissions: pull-requests: read # list the open pull requests outputs: pr: ${{ steps.pr.outputs.pr }} steps: + # GitHub's runner image has gh, jq and yq; a self-hosted runner (KM_RUNNER) may not. A missing one is + # fetched, pinned by version and sha256, into this job's temp folder; the same step in every km workflow + - name: the tools this job runs, where the runner lacks them + if: vars.KM_RUNNER != '' + env: + TOOLS: gh jq + run: | + set -euo pipefail + bin="$RUNNER_TEMP/km-tools"; mkdir -p "$bin"; echo "$bin" >> "$GITHUB_PATH" + case "$(uname -s)-$(uname -m)" in Linux-x86_64) arch=amd64;; Linux-aarch64|Linux-arm64) arch=arm64;; *) arch=;; esac + for tool in $TOOLS; do + if [ "$tool" = yq ]; then # mikefarah's yq v4: Ubuntu's yq package is another program + yq --version 2>/dev/null | grep -q 'mikefarah/yq.* v4\.' && continue + else + command -v "$tool" >/dev/null && continue + fi + gh=https://github.com/cli/cli/releases/download/v2.103.3/gh_2.103.3_linux + jq=https://github.com/jqlang/jq/releases/download/jq-1.8.2/jq-linux + yq=https://github.com/mikefarah/yq/releases/download/v4.54.1/yq_linux + case "$tool-$arch" in + gh-amd64) url="${gh}_amd64.tar.gz" sum=4c806978ffce1cec70853302a56d70ac6da4f19bae7adca4e1a8ab00633dd91d;; + gh-arm64) url="${gh}_arm64.tar.gz" sum=ee32df2323c8847f5fac6cb6b4b69ca75828e9329a8f22f677021f7b3306ce63;; + jq-amd64) url="$jq-amd64" sum=b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f;; + jq-arm64) url="$jq-arm64" sum=8b85c817833814ddca00a144c33705546355afccf0cf39b188f3cdb48b852309;; + yq-amd64) url="${yq}_amd64" sum=8e34fc298390875de416e6a4afcb8cabeceb25d9aa8506c1a2f9353cf702ea5f;; + yq-arm64) url="${yq}_arm64" sum=189088da0c6429ec5178dfaab1a114805f6cab0b61b165ab236efedf1d57a71b;; + *) echo "::error::$tool is missing and km pins no build of it for $(uname -s) $(uname -m): install it on the runner"; exit 1;; + esac + curl -fsSL --retry 3 -o "$bin/$tool.get" "$url" + echo "$sum $bin/$tool.get" | sha256sum -c --quiet - + if [ "$tool" = gh ]; then tar -xzOf "$bin/gh.get" --wildcards '*/bin/gh' > "$bin/gh"; rm "$bin/gh.get" + else mv "$bin/$tool.get" "$bin/$tool"; fi + chmod +x "$bin/$tool"; echo "installed $tool from $url" + done - name: the pull request id: pr env: @@ -120,7 +154,7 @@ jobs: decide: needs: resolve if: needs.resolve.outputs.pr != '' - runs-on: ubuntu-latest + runs-on: ${{ vars.KM_RUNNER || 'ubuntu-latest' }} timeout-minutes: 15 # km automerge waits up to 5 minutes for pending checks permissions: contents: read # ci.yml, the docs and the schema km reads @@ -138,6 +172,40 @@ jobs: # the App (see the top) merges: KM_AUTOMERGE_APP=1 and a merge app: ${{ vars.KM_AUTOMERGE_APP == '1' && steps.decide.outputs.merge == '1' && '1' || '' }} steps: + # GitHub's runner image has gh, jq and yq; a self-hosted runner (KM_RUNNER) may not. A missing one is + # fetched, pinned by version and sha256, into this job's temp folder; the same step in every km workflow + - name: the tools this job runs, where the runner lacks them + if: vars.KM_RUNNER != '' + env: + TOOLS: gh + run: | + set -euo pipefail + bin="$RUNNER_TEMP/km-tools"; mkdir -p "$bin"; echo "$bin" >> "$GITHUB_PATH" + case "$(uname -s)-$(uname -m)" in Linux-x86_64) arch=amd64;; Linux-aarch64|Linux-arm64) arch=arm64;; *) arch=;; esac + for tool in $TOOLS; do + if [ "$tool" = yq ]; then # mikefarah's yq v4: Ubuntu's yq package is another program + yq --version 2>/dev/null | grep -q 'mikefarah/yq.* v4\.' && continue + else + command -v "$tool" >/dev/null && continue + fi + gh=https://github.com/cli/cli/releases/download/v2.103.3/gh_2.103.3_linux + jq=https://github.com/jqlang/jq/releases/download/jq-1.8.2/jq-linux + yq=https://github.com/mikefarah/yq/releases/download/v4.54.1/yq_linux + case "$tool-$arch" in + gh-amd64) url="${gh}_amd64.tar.gz" sum=4c806978ffce1cec70853302a56d70ac6da4f19bae7adca4e1a8ab00633dd91d;; + gh-arm64) url="${gh}_arm64.tar.gz" sum=ee32df2323c8847f5fac6cb6b4b69ca75828e9329a8f22f677021f7b3306ce63;; + jq-amd64) url="$jq-amd64" sum=b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f;; + jq-arm64) url="$jq-arm64" sum=8b85c817833814ddca00a144c33705546355afccf0cf39b188f3cdb48b852309;; + yq-amd64) url="${yq}_amd64" sum=8e34fc298390875de416e6a4afcb8cabeceb25d9aa8506c1a2f9353cf702ea5f;; + yq-arm64) url="${yq}_arm64" sum=189088da0c6429ec5178dfaab1a114805f6cab0b61b165ab236efedf1d57a71b;; + *) echo "::error::$tool is missing and km pins no build of it for $(uname -s) $(uname -m): install it on the runner"; exit 1;; + esac + curl -fsSL --retry 3 -o "$bin/$tool.get" "$url" + echo "$sum $bin/$tool.get" | sha256sum -c --quiet - + if [ "$tool" = gh ]; then tar -xzOf "$bin/gh.get" --wildcards '*/bin/gh' > "$bin/gh"; rm "$bin/gh.get" + else mv "$bin/$tool.get" "$bin/$tool"; fi + chmod +x "$bin/$tool"; echo "installed $tool from $url" + done - name: the km the default branch's ci.yml pins id: pin env: @@ -197,7 +265,7 @@ jobs: act: needs: [resolve, decide] if: needs.decide.outputs.labelled == '1' - runs-on: ubuntu-latest + runs-on: ${{ vars.KM_RUNNER || 'ubuntu-latest' }} timeout-minutes: 5 permissions: contents: write # merge @@ -210,6 +278,47 @@ jobs: # decide's app: the App merges; an empty name is none environment: ${{ needs.decide.outputs.app == '1' && 'automerge' || '' }} steps: + # GitHub's runner image has gh, jq and yq; a self-hosted runner (KM_RUNNER) may not. A missing one is + # fetched, pinned by version and sha256, into this job's temp folder; the same step in every km workflow + - name: the tools this job runs, where the runner lacks them + if: vars.KM_RUNNER != '' && needs.decide.outputs.app != '1' + env: + TOOLS: gh jq + run: | + set -euo pipefail + bin="$RUNNER_TEMP/km-tools"; mkdir -p "$bin"; echo "$bin" >> "$GITHUB_PATH" + case "$(uname -s)-$(uname -m)" in Linux-x86_64) arch=amd64;; Linux-aarch64|Linux-arm64) arch=arm64;; *) arch=;; esac + for tool in $TOOLS; do + if [ "$tool" = yq ]; then # mikefarah's yq v4: Ubuntu's yq package is another program + yq --version 2>/dev/null | grep -q 'mikefarah/yq.* v4\.' && continue + else + command -v "$tool" >/dev/null && continue + fi + gh=https://github.com/cli/cli/releases/download/v2.103.3/gh_2.103.3_linux + jq=https://github.com/jqlang/jq/releases/download/jq-1.8.2/jq-linux + yq=https://github.com/mikefarah/yq/releases/download/v4.54.1/yq_linux + case "$tool-$arch" in + gh-amd64) url="${gh}_amd64.tar.gz" sum=4c806978ffce1cec70853302a56d70ac6da4f19bae7adca4e1a8ab00633dd91d;; + gh-arm64) url="${gh}_arm64.tar.gz" sum=ee32df2323c8847f5fac6cb6b4b69ca75828e9329a8f22f677021f7b3306ce63;; + jq-amd64) url="$jq-amd64" sum=b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f;; + jq-arm64) url="$jq-arm64" sum=8b85c817833814ddca00a144c33705546355afccf0cf39b188f3cdb48b852309;; + yq-amd64) url="${yq}_amd64" sum=8e34fc298390875de416e6a4afcb8cabeceb25d9aa8506c1a2f9353cf702ea5f;; + yq-arm64) url="${yq}_arm64" sum=189088da0c6429ec5178dfaab1a114805f6cab0b61b165ab236efedf1d57a71b;; + *) echo "::error::$tool is missing and km pins no build of it for $(uname -s) $(uname -m): install it on the runner"; exit 1;; + esac + curl -fsSL --retry 3 -o "$bin/$tool.get" "$url" + echo "$sum $bin/$tool.get" | sha256sum -c --quiet - + if [ "$tool" = gh ]; then tar -xzOf "$bin/gh.get" --wildcards '*/bin/gh' > "$bin/gh"; rm "$bin/gh.get" + else mv "$bin/$tool.get" "$bin/$tool"; fi + chmod +x "$bin/$tool"; echo "installed $tool from $url" + done + # The App's token is never handed to a downloaded tool: with the App, gh and jq are the runner's own + - name: with the App, gh and jq come with the runner + if: needs.decide.outputs.app == '1' + run: | + command -v gh >/dev/null && command -v jq >/dev/null || { + echo "::error::KM_AUTOMERGE_APP is 1 and the runner lacks gh or jq: nothing merged. Install both in the runner's image; the App's token is not handed to a tool this job downloads" + exit 1; } - name: the App's token, for this repository's contents id: token if: needs.decide.outputs.app == '1' diff --git a/src/km/templates/team/bump-mounted-pins.yml b/src/km/templates/team/bump-mounted-pins.yml index 805eeb9..bcc7aec 100644 --- a/src/km/templates/team/bump-mounted-pins.yml +++ b/src/km/templates/team/bump-mounted-pins.yml @@ -29,7 +29,8 @@ name: bump-mounted-pins # brain's owner, a public one still resolves and a private one is reported and fails the run; a # brain whose mounts all belong to other owners is not handled (mounts finds none and stops). # Submodule paths must not contain spaces. The runner is ubuntu-latest unless the repository -# variable MOUNTED_PINS_RUNNER names another label. +# variable MOUNTED_PINS_RUNNER, else KM_RUNNER, names another label; the jobs run in a container, so +# a self-hosted runner needs docker. on: schedule: # Weekday mornings, 06:25 UTC. Off the hour on purpose: scheduled runs cluster there and are @@ -54,7 +55,8 @@ defaults: jobs: mounts: - runs-on: ${{ vars.MOUNTED_PINS_RUNNER || 'ubuntu-latest' }} + runs-on: ${{ vars.MOUNTED_PINS_RUNNER || vars.KM_RUNNER || 'ubuntu-latest' }} + timeout-minutes: 15 container: python:3.12-slim@sha256:4840d514c583a4d3d8cad3940e68dee6f0f6f3635c12fcd0a3d74c21a5b93a54 # 3.12-slim outputs: names: ${{ steps.mounted.outputs.names }} @@ -86,7 +88,8 @@ jobs: resolve: needs: mounts if: needs.mounts.outputs.names != '' - runs-on: ${{ vars.MOUNTED_PINS_RUNNER || 'ubuntu-latest' }} + runs-on: ${{ vars.MOUNTED_PINS_RUNNER || vars.KM_RUNNER || 'ubuntu-latest' }} + timeout-minutes: 15 # The app key lives in this environment, which only main may use. A workflow pushed on another # branch therefore cannot reach the key. environment: mounted-pins @@ -213,7 +216,8 @@ jobs: publish: needs: resolve if: ${{ !cancelled() && needs.resolve.outputs.changed == 'true' }} - runs-on: ${{ vars.MOUNTED_PINS_RUNNER || 'ubuntu-latest' }} + runs-on: ${{ vars.MOUNTED_PINS_RUNNER || vars.KM_RUNNER || 'ubuntu-latest' }} + timeout-minutes: 15 container: python:3.12-slim@sha256:4840d514c583a4d3d8cad3940e68dee6f0f6f3635c12fcd0a3d74c21a5b93a54 # 3.12-slim permissions: contents: write # push the bump branch diff --git a/src/km/templates/team/ci.yml b/src/km/templates/team/ci.yml index 9e1869a..5f30d07 100644 --- a/src/km/templates/team/ci.yml +++ b/src/km/templates/team/ci.yml @@ -19,7 +19,8 @@ permissions: jobs: validate: - runs-on: ubuntu-latest + runs-on: ${{ vars.KM_RUNNER || 'ubuntu-latest' }} + timeout-minutes: 15 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: diff --git a/src/km/templates/team/main-guard.yml b/src/km/templates/team/main-guard.yml index 5d1ea5b..8cf2d08 100644 --- a/src/km/templates/team/main-guard.yml +++ b/src/km/templates/team/main-guard.yml @@ -23,8 +23,43 @@ permissions: jobs: check: - runs-on: ubuntu-latest + runs-on: ${{ vars.KM_RUNNER || 'ubuntu-latest' }} + timeout-minutes: 10 # up to 3.5 minutes of waiting for the merged pull requests to show steps: + # GitHub's runner image has gh, jq and yq; a self-hosted runner (KM_RUNNER) may not. A missing one is + # fetched, pinned by version and sha256, into this job's temp folder; the same step in every km workflow + - name: the tools this job runs, where the runner lacks them + if: vars.KM_RUNNER != '' + env: + TOOLS: gh jq + run: | + set -euo pipefail + bin="$RUNNER_TEMP/km-tools"; mkdir -p "$bin"; echo "$bin" >> "$GITHUB_PATH" + case "$(uname -s)-$(uname -m)" in Linux-x86_64) arch=amd64;; Linux-aarch64|Linux-arm64) arch=arm64;; *) arch=;; esac + for tool in $TOOLS; do + if [ "$tool" = yq ]; then # mikefarah's yq v4: Ubuntu's yq package is another program + yq --version 2>/dev/null | grep -q 'mikefarah/yq.* v4\.' && continue + else + command -v "$tool" >/dev/null && continue + fi + gh=https://github.com/cli/cli/releases/download/v2.103.3/gh_2.103.3_linux + jq=https://github.com/jqlang/jq/releases/download/jq-1.8.2/jq-linux + yq=https://github.com/mikefarah/yq/releases/download/v4.54.1/yq_linux + case "$tool-$arch" in + gh-amd64) url="${gh}_amd64.tar.gz" sum=4c806978ffce1cec70853302a56d70ac6da4f19bae7adca4e1a8ab00633dd91d;; + gh-arm64) url="${gh}_arm64.tar.gz" sum=ee32df2323c8847f5fac6cb6b4b69ca75828e9329a8f22f677021f7b3306ce63;; + jq-amd64) url="$jq-amd64" sum=b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f;; + jq-arm64) url="$jq-arm64" sum=8b85c817833814ddca00a144c33705546355afccf0cf39b188f3cdb48b852309;; + yq-amd64) url="${yq}_amd64" sum=8e34fc298390875de416e6a4afcb8cabeceb25d9aa8506c1a2f9353cf702ea5f;; + yq-arm64) url="${yq}_arm64" sum=189088da0c6429ec5178dfaab1a114805f6cab0b61b165ab236efedf1d57a71b;; + *) echo "::error::$tool is missing and km pins no build of it for $(uname -s) $(uname -m): install it on the runner"; exit 1;; + esac + curl -fsSL --retry 3 -o "$bin/$tool.get" "$url" + echo "$sum $bin/$tool.get" | sha256sum -c --quiet - + if [ "$tool" = gh ]; then tar -xzOf "$bin/gh.get" --wildcards '*/bin/gh' > "$bin/gh"; rm "$bin/gh.get" + else mv "$bin/$tool.get" "$bin/$tool"; fi + chmod +x "$bin/$tool"; echo "installed $tool from $url" + done - name: every pushed commit comes from a merged pull request env: GH_TOKEN: ${{ github.token }} diff --git a/src/km/templates/team/review-request-guide.yml b/src/km/templates/team/review-request-guide.yml index 511b82f..7ea52f9 100644 --- a/src/km/templates/team/review-request-guide.yml +++ b/src/km/templates/team/review-request-guide.yml @@ -20,8 +20,9 @@ name: review-request-guide # # It never checks out or runs the pull request's code: pull_request_target runs this file from the # default branch, and the docs are read from the API as data, by the blob ids of the pull request's -# file list (no file name enters a URL). Only the YAML frontmatter is parsed (yq, from the runner image, -# bounded in time and memory; a block with anchors or merge keys, or over 64 KB, is left to ci), and +# file list (no file name enters a URL). Only the YAML frontmatter is parsed (mikefarah's yq, from the +# runner image or fetched by the first step, bounded in time and memory; a block with anchors or merge +# keys, or over 64 KB, is left to ci), and # only fixed words derived from it go into the comment; file names are quoted as JSON inside a code # block, so a name cannot pose as Markdown, a mention or a workflow command. on: @@ -47,9 +48,43 @@ jobs: concurrency: group: review-request-guide-${{ github.event.pull_request.number }} cancel-in-progress: false # one request after another: the second finds the first one's comment - runs-on: ubuntu-latest + runs-on: ${{ vars.KM_RUNNER || 'ubuntu-latest' }} timeout-minutes: 10 # one read per changed doc, at most 300 steps: + # GitHub's runner image has gh, jq and yq; a self-hosted runner (KM_RUNNER) may not. A missing one is + # fetched, pinned by version and sha256, into this job's temp folder; the same step in every km workflow + - name: the tools this job runs, where the runner lacks them + if: vars.KM_RUNNER != '' + env: + TOOLS: gh jq yq + run: | + set -euo pipefail + bin="$RUNNER_TEMP/km-tools"; mkdir -p "$bin"; echo "$bin" >> "$GITHUB_PATH" + case "$(uname -s)-$(uname -m)" in Linux-x86_64) arch=amd64;; Linux-aarch64|Linux-arm64) arch=arm64;; *) arch=;; esac + for tool in $TOOLS; do + if [ "$tool" = yq ]; then # mikefarah's yq v4: Ubuntu's yq package is another program + yq --version 2>/dev/null | grep -q 'mikefarah/yq.* v4\.' && continue + else + command -v "$tool" >/dev/null && continue + fi + gh=https://github.com/cli/cli/releases/download/v2.103.3/gh_2.103.3_linux + jq=https://github.com/jqlang/jq/releases/download/jq-1.8.2/jq-linux + yq=https://github.com/mikefarah/yq/releases/download/v4.54.1/yq_linux + case "$tool-$arch" in + gh-amd64) url="${gh}_amd64.tar.gz" sum=4c806978ffce1cec70853302a56d70ac6da4f19bae7adca4e1a8ab00633dd91d;; + gh-arm64) url="${gh}_arm64.tar.gz" sum=ee32df2323c8847f5fac6cb6b4b69ca75828e9329a8f22f677021f7b3306ce63;; + jq-amd64) url="$jq-amd64" sum=b1c22172dd303f3be49e935aa56aa48a8b7a46e0bc838b4997d3bb451495870f;; + jq-arm64) url="$jq-arm64" sum=8b85c817833814ddca00a144c33705546355afccf0cf39b188f3cdb48b852309;; + yq-amd64) url="${yq}_amd64" sum=8e34fc298390875de416e6a4afcb8cabeceb25d9aa8506c1a2f9353cf702ea5f;; + yq-arm64) url="${yq}_arm64" sum=189088da0c6429ec5178dfaab1a114805f6cab0b61b165ab236efedf1d57a71b;; + *) echo "::error::$tool is missing and km pins no build of it for $(uname -s) $(uname -m): install it on the runner"; exit 1;; + esac + curl -fsSL --retry 3 -o "$bin/$tool.get" "$url" + echo "$sum $bin/$tool.get" | sha256sum -c --quiet - + if [ "$tool" = gh ]; then tar -xzOf "$bin/gh.get" --wildcards '*/bin/gh' > "$bin/gh"; rm "$bin/gh.get" + else mv "$bin/$tool.get" "$bin/$tool"; fi + chmod +x "$bin/$tool"; echo "installed $tool from $url" + done - name: tell the contributor what comes before the maintainer env: GH_TOKEN: ${{ github.token }} diff --git a/src/km/templates/team/staleness.yml b/src/km/templates/team/staleness.yml index 9e90335..b0b9636 100644 --- a/src/km/templates/team/staleness.yml +++ b/src/km/templates/team/staleness.yml @@ -14,7 +14,8 @@ permissions: jobs: demote: - runs-on: ubuntu-latest + runs-on: ${{ vars.KM_RUNNER || 'ubuntu-latest' }} + timeout-minutes: 15 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: diff --git a/src/km/upgrade.py b/src/km/upgrade.py index 8e58ccc..24dd253 100644 --- a/src/km/upgrade.py +++ b/src/km/upgrade.py @@ -93,19 +93,19 @@ # shipped_hash of every version km shipped, the current one included (a test checks it): a copy that # is still one of them is replaced by the current template, an edited one is left with a note. SHIPPED = { - "team/bump-mounted-pins.yml": {"c72b295b1ba76e65", "bb30b4db64fb033e", "139a62edc4339833"}, + "team/bump-mounted-pins.yml": {"c72b295b1ba76e65", "bb30b4db64fb033e", "139a62edc4339833", "20994093d5d54451"}, "team/bump-mounted-pins.py": {"52f798d7ca2f7d32", "9dc066d86b74a44f"}, - "team/approve-fork-ci.yml": {"accf1785f4710b4e", "70ad9729fd426ebd", "e10d6138ae696efb", "5f87b231aff97388", "bac7f4477dadece9"}, - "team/review-request-guide.yml": {"e2a600682b732ddc", "1a0d3ea14f80d3e7"}, - "team/main-guard.yml": {"c2896be807a78e55", "80921af35464658e"}, - "team/automerge.yml": {"ecae0d6c1aa0ed2a", "5803e8303b7cf540", "5ad6d46ffa325bc7", "7af086f2b99c3493", "dde68b9c6272983a", "de0b3cc467824c05", "9ee47ce4374f2938", "c61b4233e5119f6d"}, + "team/approve-fork-ci.yml": {"accf1785f4710b4e", "70ad9729fd426ebd", "e10d6138ae696efb", "5f87b231aff97388", "bac7f4477dadece9", "b944e6b5358d92dd"}, + "team/review-request-guide.yml": {"e2a600682b732ddc", "1a0d3ea14f80d3e7", "bf78ce9ff5ddd127"}, + "team/main-guard.yml": {"c2896be807a78e55", "80921af35464658e", "75dee4df6004f801"}, + "team/automerge.yml": {"ecae0d6c1aa0ed2a", "5803e8303b7cf540", "5ad6d46ffa325bc7", "7af086f2b99c3493", "dde68b9c6272983a", "de0b3cc467824c05", "9ee47ce4374f2938", "c61b4233e5119f6d", "9c4ff32ba808ba4c"}, } # approve-fork-ci versions that approved data endings only: the later ones approve any file but control # files, safe only beside km's ci and a km from 1.11.0 on, so upgrade does not widen one of these otherwise ALLOWLIST_AFC = {"accf1785f4710b4e", "70ad9729fd426ebd"} # ci_hash of every km team ci version: upgrade adds approve-fork-ci only where ci.yml is one of these, since # a fork's approved run is safe only while ci runs nothing but km. A change to team/ci.yml adds its hash here. -KM_CI = {"4ab8b6e4f2ef1e22", "e97892ef9bb0d77f"} +KM_CI = {"4ab8b6e4f2ef1e22", "e97892ef9bb0d77f", "76dc97b57c9649e3", "0606bf55cf58761c", "a1f4566802ede1da", "2e8e021af1f8ada5"} DIGEST_AFTER = (1, 10, 1) # the last km release that does not record or check approved_digest INDEX_AFTER = (1, 11, 0) # the last km release whose gen-index writes a title as YAML reads it (1.10 as 1.1) diff --git a/tests/automerge_smoke.sh b/tests/automerge_smoke.sh index 72b9ab9..f3b32b6 100644 --- a/tests/automerge_smoke.sh +++ b/tests/automerge_smoke.sh @@ -216,9 +216,9 @@ out="$(cd "$T" && PATH="$T/bin:$PATH" kmpy -c 'import sys, km.automerge as a; a. # --self: the workflow's own pending check is left out, another name's is not setup 26; run --self other-name; grep -q "the check automerge/act is pending" <<<"$out" && ok "--self leaves out only the named workflow's own checks" || no "--self ($out)" # --- the workflow's act job: its own script, against the same stand-in --- -# step JOB [ID]: the script of JOB's step ID (the first step without one) +# step JOB [ID]: the script of JOB's step ID (without one, the first step but the one that fetches the tools) step(){ kmpy -c 'import sys, yaml; steps = yaml.safe_load(open(sys.argv[1]))["jobs"][sys.argv[2]]["steps"] -print(next(s for s in steps if s.get("id") == sys.argv[3])["run"] if len(sys.argv) > 3 else steps[0]["run"])' \ +print(next(s for s in steps if s.get("id") == sys.argv[3])["run"] if len(sys.argv) > 3 else next(s for s in steps if "TOOLS" not in s.get("env", {}))["run"])' \ "$REPO/src/km/templates/team/automerge.yml" "$@"; } # the merge step's script; MERGE_TOKEN and APP (the App's token, KM_AUTOMERGE_APP) from the environment, # GH_TOKEN as GitHub's: the stand-in logs the token of each write diff --git a/tests/doctor_smoke.sh b/tests/doctor_smoke.sh index 0620a0d..7782cbd 100644 --- a/tests/doctor_smoke.sh +++ b/tests/doctor_smoke.sh @@ -254,6 +254,9 @@ case "$*" in "api --hostname github.com repos/O/R --jq .default_branch") [ "$GH_DEFAULT" = fail ] && { echo '{"message":"x"}'; exit 1; }; echo "${GH_DEFAULT:-main}" ;; "api --hostname github.com repos/O/R/environments/automerge --jq .deployment_branch_policy"*) [ -n "$GH_AMENV" ] && echo "$GH_AMENV" || exit 1 ;; "api --hostname github.com repos/O/R/environments/automerge/deployment-branch-policies --jq"*) echo "${GH_AMBRANCHES:-main}" ;; + "api --hostname github.com repos/O/R/actions/variables/KM_RUNNER --jq .value") [ "$GH_RUNNER" = denied ] && { echo '{"message":"Resource not accessible","status":"403"}'; exit 1; } + [ -n "$GH_RUNNER" ] && echo "$GH_RUNNER" || { echo '{"message":"Not Found","status":"404"}'; exit 1; } ;; + "api --hostname github.com --paginate repos/O/R/actions/organization-variables --jq"*) [ "$GH_ORGRUNNER" = fail ] && exit 1; [ -n "$GH_ORGRUNNER" ] && echo "$GH_ORGRUNNER"; : ;; *) exit 2 ;; esac SH @@ -311,6 +314,30 @@ for s, w, f in d.results: print(s, w, "|", f)' "$1" 2>&1)"; } && ok "an unreadable default branch skips the environment checks with a note, never judging against gh's error" || no "automerge default unread ($out)" GH_AMENV=custom GH_DEFAULT=trunk envcheck "$A"; { out_has "open to main, not to the default branch trunk alone" && out_has '`trunk` only'; } \ && ok "restricted to main where the default branch is trunk gets a note naming trunk" || no "automerge main vs trunk ($out)" + # the runner the team workflows run on: a note where KM_RUNNER is set and readable, else nothing + runcheck(){ out="$(PATH="$G:$PATH" kmpy -c 'import sys, pathlib, km.doctor as d +d.check_runner(pathlib.Path(sys.argv[1]), "O/R") +for r in d.results: print(*r)' "$1" 2>&1)"; } + R="$T/runbrain"; mkdir -p "$R/.github/workflows" + ci(){ printf 'jobs:\n validate:\n runs-on: %s\n steps:\n - uses: pasrom/km@abc # v1\n' "$1" > "$R/.github/workflows/ci.yml" + printf 'jobs:\n demote:\n runs-on: %s\n' "${2:-$1}" > "$R/.github/workflows/staleness.yml"; } + V="\${{ vars.KM_RUNNER || 'ubuntu-latest' }}"; ci "$V" + GH_RUNNER=my-runner runcheck "$R"; out_has "note the team workflows run on the runner my-runner (the variable KM_RUNNER; bump-mounted-pins reads MOUNTED_PINS_RUNNER first): if GitHub passes" \ + && ok "a repository's KM_RUNNER gets a note: a fork's pull request may run ci on that runner" || no "repo KM_RUNNER ($out)" + GH_RUNNER= GH_ORGRUNNER=org-runner runcheck "$R"; out_has "run on the runner org-runner" && ok "an organisation's KM_RUNNER gets the note too" || no "org KM_RUNNER ($out)" + GH_RUNNER=denied GH_ORGRUNNER=org-runner runcheck "$R"; [ -z "$out" ] && ok "a repository variable gh may not read is not stood in for by the organisation's" || no "repo KM_RUNNER 403 ($out)" + GH_RUNNER= GH_ORGRUNNER= runcheck "$R"; [ -z "$out" ] && ok "no KM_RUNNER: no note" || no "no KM_RUNNER ($out)" + GH_RUNNER= GH_ORGRUNNER=fail runcheck "$R"; [ -z "$out" ] && ok "a KM_RUNNER gh may not read: no note, no failure" || no "unreadable KM_RUNNER ($out)" + GH_RUNNER=my-runner runcheck "$T/ambrain"; [ -z "$out" ] && ok "a brain without the team ci workflow is not asked about KM_RUNNER" || no "KM_RUNNER without ci ($out)" + ci "$V" ubuntu-latest + GH_RUNNER=my-runner runcheck "$R"; out_has "note the variable KM_RUNNER names the runner my-runner, but the runs-on line of staleness.yml does not read it" \ + && ok "KM_RUNNER set beside a staleness.yml that ignores it gets a note naming that file" || no "KM_RUNNER staleness ignores ($out)" + ci "ubuntu-latest # not $V" + GH_RUNNER= GH_ORGRUNNER=org-runner runcheck "$R"; out_has "names the runner org-runner, but the runs-on line of ci.yml and staleness.yml does not read it" \ + && ok "an organisation's KM_RUNNER beside a ci.yml that ignores it (a comment naming it does not count) gets a note" || no "KM_RUNNER with an older ci ($out)" + GH_RUNNER= GH_ORGRUNNER= runcheck "$R"; [ -z "$out" ] && ok "a ci.yml that ignores an unset KM_RUNNER gets no note" || no "older ci, no KM_RUNNER ($out)" + sed -i.bak 's/pasrom/some/' "$R/.github/workflows/ci.yml" + GH_RUNNER=my-runner runcheck "$R"; [ -z "$out" ] && ok "a ci.yml of the brain's own (not km's) is not told to read KM_RUNNER" || no "own ci, KM_RUNNER ($out)" C="$T/cred"; git init -q "$C" login(){ out="$(PATH="$G:$PATH" kmpy -c 'import sys, pathlib, km.doctor as d d.check_git_login(pathlib.Path(sys.argv[1]), sys.argv[2]) diff --git a/tests/mounted_pins_smoke.sh b/tests/mounted_pins_smoke.sh index 3155885..ee69902 100644 --- a/tests/mounted_pins_smoke.sh +++ b/tests/mounted_pins_smoke.sh @@ -23,7 +23,7 @@ def need(cond, what): need(wf["defaults"]["run"]["shell"] == "bash", "defaults.run.shell is bash (a container job runs sh otherwise)") need(wf["permissions"] == {"contents": "read"}, "read-only by default") for name, job in jobs.items(): - need(job["runs-on"] == "${{ vars.MOUNTED_PINS_RUNNER || 'ubuntu-latest' }}", f"{name}: ubuntu-latest unless the brain names a runner") + need(job["runs-on"] == "${{ vars.MOUNTED_PINS_RUNNER || vars.KM_RUNNER || 'ubuntu-latest' }}", f"{name}: ubuntu-latest unless the brain names a runner") need("@sha256:" in job.get("container", ""), f"{name}: the image is pinned by digest") uses = [s["uses"].split("@")[0] for s in job["steps"] if "uses" in s] allowed = ["actions/checkout", "actions/create-github-app-token"] if name == "resolve" else ["actions/checkout"] diff --git a/tests/team_smoke.sh b/tests/team_smoke.sh index 25fa33e..39eb4b4 100644 --- a/tests/team_smoke.sh +++ b/tests/team_smoke.sh @@ -903,7 +903,7 @@ case "$1" in esac SH printf '#!/bin/sh\n' > "$SB/sleep"; chmod +x "$SB/gh" "$SB/sleep" -kmpy -c 'import sys, yaml; print(yaml.safe_load(open(sys.argv[1]))["jobs"]["check"]["steps"][0]["run"])' "$T/.github/workflows/main-guard.yml" > "$F/guard.sh" +kmpy -c 'import sys, yaml; print(next(s["run"] for s in yaml.safe_load(open(sys.argv[1]))["jobs"]["check"]["steps"] if "TOOLS" not in s.get("env", {})))' "$T/.github/workflows/main-guard.yml" > "$F/guard.sh" # guard FORCED COMMITS [CREATED [BRANCHES [PRS]]]: the repository's branches and its pull request count as gh answers them guard(){ GH_LOG="$F/gh.log" PATH="$SB:$PATH" REPO=o/r FORCED="$1" COMMITS="$2" CREATED="${3:-false}" \ BRANCHES="${4:-main}" PRS="${5:-0}" PUSHER=p RUN=u bash "$F/guard.sh" >/dev/null 2>&1; } @@ -925,12 +925,59 @@ unflagged true main down || bad+=" pulls-down" || no "main-guard first push:$bad" kmpy -c 'import km.upgrade as u; from km.init import template; raise SystemExit(u.ci_hash(template("team/ci.yml", {})) not in u.KM_CI)' \ && ok "KM_CI lists the current team ci template" || no "team/ci.yml changed: add its ci_hash to KM_CI in upgrade.py" -kmpy - <<'PY' && ok "a km ci.yml with the sign-off step turned on by upgrade still counts as km's" || no "with_signoff result not in KM_CI" +kmpy - <<'PY' && ok "km's earlier ci.yml copies (ubuntu-latest; runs-on changed by hand to KM_RUNNER) and the current one count as km's, with the sign-off step or without it" || no "a km ci.yml copy is not in KM_CI" import re, km.upgrade as u from km.init import template -t = template("team/ci.yml", {}) -old, n = re.subn(r"(?m)^( *)env:\n *BASE: .*\n( *)run: km validate .*\n", r"\2run: km validate\n", t) -raise SystemExit(n != 1 or u.ci_hash(u.with_signoff(old)) not in u.KM_CI) +new = template("team/ci.yml", {}) +hand = new.replace(" timeout-minutes: 15\n", "", 1) # runs-on edited by hand, no timeout +old = hand.replace("${{ vars.KM_RUNNER || 'ubuntu-latest' }}", "ubuntu-latest") # before KM_RUNNER +bad = len({new, hand, old}) != 3 +for t in (new, hand, old): + unsigned, n = re.subn(r"(?m)^( *)env:\n *BASE: .*\n( *)run: km validate .*\n", r"\2run: km validate\n", t) + bad |= n != 1 or u.ci_hash(unsigned) not in u.KM_CI or u.ci_hash(u.with_signoff(unsigned)) not in u.KM_CI or u.ci_hash(t) not in u.KM_CI +raise SystemExit(bad) +PY +# Every job of every team workflow runs on the runner the brain names (KM_RUNNER; bump-mounted-pins' +# own variable first), and a job that runs gh, jq or yq first fetches the ones the runner lacks, with +# the same step everywhere; a pull_request_target workflow checks out km at most, never the pull request +kmpy - "$REPO/src/km/templates/team" <<'PY' && ok "every team workflow job runs on KM_RUNNER or ubuntu-latest and fetches the gh, jq and yq it runs where they are missing; no pull_request_target job checks out the pull request" || no "team workflow runners" +import re, sys, yaml +from pathlib import Path +RUNNER = "${{ vars.KM_RUNNER || 'ubuntu-latest' }}" +PINS = "${{ vars.MOUNTED_PINS_RUNNER || vars.KM_RUNNER || 'ubuntu-latest' }}" +TOOL = "the tools this job runs, where the runner lacks them" +problems, scripts = [], set() +files = sorted(Path(sys.argv[1]).glob("*.yml")) +for f in files: + wf = yaml.safe_load(f.read_text(encoding="utf-8")) + on = wf.get(True, wf.get("on")) + if not isinstance(wf.get("jobs"), dict): + continue # dependabot.yml + for name, job in wf["jobs"].items(): + want = PINS if f.name == "bump-mounted-pins.yml" else RUNNER + job["runs-on"] == want or problems.append(f"{f.name} {name}: runs-on {job['runs-on']!r}") + if "timeout-minutes" not in job: + problems.append(f"{f.name} {name}: no timeout-minutes (the default holds a runner for 6 hours)") + steps = job["steps"] + tools = steps[0].get("env", {}).get("TOOLS", "").split() if steps[0].get("name") == TOOL else [] + if tools: + scripts.add(steps[0]["run"]) + steps[0].get("if", "").startswith("vars.KM_RUNNER != ''") or problems.append(f"{f.name} {name}: the tools step runs on GitHub's runners too") + if tools and any("create-github-app-token" in s.get("uses", "") for s in steps) and "app != '1'" not in steps[0]["if"]: + problems.append(f"{f.name} {name}: a downloaded tool would hold the App's token") + runs = "\n".join(s.get("run", "") for s in (steps[1:] if tools else steps)) + used = {t for t in ("gh", "jq", "yq") if re.search(rf"(? "$F/af.sh" +kmpy -c 'import sys, yaml; print(next(s["run"] for s in yaml.safe_load(open(sys.argv[1]))["jobs"]["approve"]["steps"] if "TOOLS" not in s.get("env", {})))' "$T/$AF" > "$F/af.sh" # afrun FILES-JSON [CHANGED]: 0 when it approved run 7; 1 when it left it to a person with a notice and # exit 0; 2 when it exited 0 with neither; 3 when it failed (a crash would turn every fork pull request # red). TREE, PRS, PRS2 (the pull requests from the second lookup on, the one right before approving), @@ -1169,7 +1216,7 @@ esac SH chmod +x "$RB/gh" command -v timeout >/dev/null || { printf '#!/usr/bin/env bash\nshift; exec "$@"\n' > "$RB/timeout"; chmod +x "$RB/timeout"; } # stock macOS has none -kmpy -c 'import sys, yaml; print(yaml.safe_load(open(sys.argv[1]))["jobs"]["guide"]["steps"][0]["run"])' "$T/$RG" > "$F/rg.sh" +kmpy -c 'import sys, yaml; print(next(s["run"] for s in yaml.safe_load(open(sys.argv[1]))["jobs"]["guide"]["steps"] if "TOOLS" not in s.get("env", {})))' "$T/$RG" > "$F/rg.sh" fmd(){ printf -- '---\n%s\n---\nbody\n' "$1"; } # a doc with this frontmatter EVIL='d/`'$'\n''```'$'\n''@everyone ::error::pwned.md' # a file name that would close a code block, mention and pose as a command # rgrun: run the script on the pull request of the docs in rg_docs (pathcontent, one per