Skip to content

feat(apt): build and publish the signed APT repository from mmt-dpi Releases (#2) - #5

Merged
luongnv89 merged 5 commits into
mainfrom
feat/2-build-apt-repo
Nov 11, 2026
Merged

luongnv89 merged 5 commits into
mainfrom
feat/2-build-apt-repo

Conversation

@luongnv89

@luongnv89 luongnv89 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Closes #2

Summary

  • .github/workflows/apt-repo.yml rebuilds the APT repository from scratch from the last 2 published Montimage/mmt-dpi Releases, signs it with subkey A in apt-signing, verifies it with gpgv against the committed keyring, and deploys it with site/ to Pages. After the deploy, it runs apt-get install mmt-dpi in ubuntu:24.04 and debian:13 against the live URL.
  • The signing key does not exist yet, so every run today is an unsigned dry run. The dry run builds the full tree, prints a summary, uploads the apt-repo-unsigned artifact, tests signing with a throwaway key, and installs from the tree in containers. It never deploys. Pull requests are always dry runs, so this PR's CI exercises the dry run.
  • All the logic is in scripts/*.sh (bash, set -euo pipefail, shellcheck-clean). The workflow only calls the scripts.
  • pages.yml is folded in and deleted (see the Decision Record below).

Decision Record

Folding pages.yml in. A Pages deploy replaces the whole site. With two deployers, every push to main that touched site/ would have deployed site/ alone and erased dists/ and pool/. A shared concurrency group serializes deploys, but it does not stop one deploy from overwriting another. So apt-repo.yml is now the only deployer, and it copies site/ into the tree. The push trigger therefore also covers site/**, in addition to scripts/** and the workflow file. One consequence: until the key exists, an edit to site/index.html is not published, because the dry run never deploys. The live page stays as #1 deployed it.

Concurrency. Runs on main (push, dispatch) use concurrency: { group: pages, cancel-in-progress: false } at workflow level, not job level. That way build, sign, deploy and smoke run one run at a time and in order. With job-level concurrency, a slower, older build could deploy after a newer one. GitHub keeps only the newest pending run in a group. That is fine here, because each run rebuilds from the latest releases. PR runs and branch workflow_dispatch runs get their own apt-repo-<ref> group, so they never displace a pending main run.

Signed or dry run. scripts/detect-signing.sh runs in apt-signing and receives only whether each secret is set, as secrets.X != '', never the value.

  • Signed: APT_SIGNING_KEY, APT_SIGNING_PASSPHRASE, the variable APT_SIGNING_FINGERPRINT and site/montimage-archive-keyring.gpg all exist.
  • Dry run: none of them exists.
  • Any partial setup fails loudly, so a half-configured key never turns silently into a dry run.

Job split.

  • build holds no secrets. It fetches, verifies, builds and tests.
  • sign (environment apt-signing, main only) receives only dists/ and never sees a .deb.
  • deploy (environment github-pages, pages: write and id-token: write) never sees a secret, and runs gpgv again on the full tree before deploying.
  • Every job sets persist-credentials: false and the least permissions it needs.

Signing (scripts/sign-repo.sh).

  • The secret is read from stdin, piped by the shell's built-in printf, so it never appears in a process's arguments. The passphrase goes through fd 3.
  • Both go into a mktemp -d GNUPGHOME under $RUNNER_TEMP, which is removed on exit, and the env vars are unset.
  • The committed keyring is imported first, so a renewed expiry is honoured.
  • It signs with --local-user "$APT_SIGNING_FINGERPRINT" (the value ends in !) and --digest-algo SHA512, and writes InRelease (clearsigned) and Release.gpg (detached, armoured).
  • scripts/verify-repo.sh then checks each file with gpgv --keyring site/montimage-archive-keyring.gpg, in an empty GNUPGHOME. It requires exactly one GOODSIG and VALIDSIG, made by the expected subkey. It fails on EXPKEYSIG, REVKEYSIG, EXPSIG, BADSIG, ERRSIG or NO_PUBKEY; gpgv itself exits 0 on an expired key, which apt would then reject. It also checks that InRelease carries the text of Release, the Release fields, and every Packages and pool size and SHA256.

Inputs (scripts/fetch-releases.sh).

  • It takes the last 2 releases by published_at, excluding drafts and prereleases.
  • It keeps only ^(mmt-dpi|mmt-dpi-security)_<distro>-<ver>_<arch>\.deb$.
  • SHA256SUMS itself must have an attestation.
  • The canonical names in the attested SHA256SUMS must equal the canonical assets, so a missing upload fails the run.
  • Each .deb must match its SHA256SUMS line and pass gh attestation verify <file> --repo Montimage/mmt-dpi --signer-workflow Montimage/mmt-dpi/.github/workflows/release-packages.yml --source-ref refs/tags/<tag> --deny-self-hosted-runners. These flags are stricter than --repo alone: a PR or branch build's attestation does not count.
  • Package, Version and Architecture come from dpkg-deb -f and must agree with the file name.
  • An unknown distribution or architecture fails the run, and so does a suite with no packages. Nothing is skipped.

Attestations in the 2 releases: both have them. v1.10.2 and v1.10.1 carry SLSA v1 provenance from release-packages.yml@refs/tags/<tag>. That covers every one of the 40 .deb files and SHA256SUMS; all were verified in the local run below. v1.10.0 has them too. v1.9.1 and older have no canonical .deb names. If such a release were ever in the window, the build would fail with "predates the canonical package names" rather than publish it unverified. There is no fallback path that skips attestations.

Ubuntu 26.04 codename: resolute (Resolute Raccoon). The package metadata does not name the codename, so I checked official sources:

The draft used plucky, which is 25.04 and wrong. The map is in scripts/common.sh.

Layout and reproducibility.

  • pool/<suite>/main/m/mmt-dpi/<Package>_<Version>_<Arch>.deb, with names taken from the control fields: the asset names repeat across releases.
  • apt-ftparchive packages runs once per file, in sorted order, and release runs with Origin/Label Montimage, Suite = Codename, amd64 arm64, main, and no Valid-Until.
  • Date and every mtime are the newest release's published_at (SOURCE_DATE_EPOCH), and gzip -n is used. The unsigned tree is byte-identical across runs. Signatures carry the signing time.

Rejected:

  • the draft build-repo.sh: it wrote Packages by hand, used the plucky codename, had a $GNUPG_SIGN typo, swallowed gpg errors with 2>/dev/null || true, and allowed a dry-run "skip";
  • reprepro/aptly: not needed for 5 suites × 2 architectures.

Known limit, documented rather than fixed: if the newest release is deleted or demoted, Date moves backwards, and apt silently ignores a Release with an older Date (measured in review: apt-get update exits 0, existing clients keep the stale indexes and get a 404 from the pool for the removed version). AGENTS.md ("Never delete or demote a published mmt-dpi release") and the README now say: publish a newer release instead. I did not add a workflow_dispatch input to override SOURCE_DATE_EPOCH: an override lasts one run, and the next regular run moves Date back to the newest published_at, so clients would ignore that one in turn. It would have to be sticky state, which is not small; publishing a newer release fixes it for good.

Test Results (local, real output)

Host: Ubuntu 24.04 arm64, gh 2.87.3, apt 2.8.3, GnuPG 2.4.4, Docker 29.2.1. Work dirs were in a scratch dir outside the repo. No real key was touched.

Lint:

$ shellcheck --version | sed -n 2p ; shellcheck scripts/*.sh && echo shellcheck clean
version: 0.11.0
shellcheck clean
$ sha256sum -c --ignore-missing actionlint_1.7.12_checksums.txt   # upstream checksums; amd64 line = ci.yml's pin 8aca8db9…a3d8
actionlint_1.7.12_linux_arm64.tar.gz: OK
$ ./actionlint -version | head -1 ; ./actionlint -shellcheck "$(command -v shellcheck)" && echo "actionlint clean"
1.7.12
actionlint clean

Dry-run build against the real last 2 releases:

$ scripts/fetch-releases.sh "$W/work"
==> v1.10.2 (published 2026-11-10T11:52:04Z)
    SHA256SUMS: attestation verified
    mmt-dpi-security_debian-12_amd64.deb: sha256 + attestation OK -> bookworm mmt-dpi-security 1.10.2-a84a6050 amd64
    … (40 lines like this, 20 per release) …
    mmt-dpi_ubuntu-26.04_arm64.deb: sha256 + attestation OK -> resolute mmt-dpi 1.10.1-97860a01 arm64
==> 40 packages from 2 releases verified
$ scripts/build-repo.sh "$W/work" "$W/tree" site
==> Building …/tree (Date: Tue, 10 Nov 2026 11:52:04 UTC)
    jammy: 8 packages
    noble: 8 packages
    resolute: 8 packages
    bookworm: 8 packages
    trixie: 8 packages
==> Built suites: jammy noble resolute bookworm trixie (390101853 bytes)
$ # second, independent fetch+build, compared file by file with the first
REPRODUCIBLE vs run1: 66 files identical
$ scripts/summarize-repo.sh "$W/tree" "$W/work" | tail -6
- Suites: 5
- Package files: 40
- Pool size: 372MiB
- Tree size: 373MiB
- Files in tree: 66
- Metadata: unsigned

Each suite has mmt-dpi and mmt-dpi-security, at 1.10.2-a84a6050 and 1.10.1-97860a01, for amd64 and arm64.

Signing path with a throwaway key, verified with gpgv:

$ scripts/test-signing.sh "$W/tree" "$W/signed"
# TEST ONLY: throwaway signing key, generated in this run.  #
TEST ONLY: subkey A A414B26F…ACBC (signs), subkey B A1E28618…0CEC (must not sign)
==> TEST ONLY: signing with scripts/sign-repo.sh (secret on stdin)
==> Importing the public keyring test-keyring.gpg
==> Importing the signing subkey from stdin
==> Signed: bookworm jammy noble resolute trixie
    bookworm: InRelease and Release.gpg verified with gpgv (key A414B26FF4CA0DB46235EFAC575E54F97DACACBC), indexes match
    … jammy, noble, resolute, trixie: same …
==> 5 suites verified against test-keyring.gpg
==> TEST ONLY: negative checks
    signatures from the wrong expected signer: rejected (expected FD8B8A515450AE1B3166F1F0061A2D9F9A528056)
    a keyring where the signing subkey has expired: rejected (gpgv reported EXPKEYSIG)
    a keyring where the signing subkey is revoked: rejected (gpgv reported REVKEYSIG)
    a tampered Release: rejected (InRelease does not carry the text of Release)
    stale export without the refreshed keyring: cannot sign (so importing the keyring first matters)
==> TEST ONLY: signing path verified with gpgv against the throwaway keyring
$ gpgv --keyring "$W/signed/test-keyring.gpg" "$W/signed/dists/noble/InRelease"
gpgv: Good signature from "TEST ONLY throwaway APT key <apt-test@invalid>"
$ scripts/verify-repo.sh "$W/combined" "$W/signed/test-keyring.gpg"   # signed dists + full pool: pool hashes too
==> 5 suites verified against test-keyring.gpg
$ scripts/verify-repo.sh --no-pool <tree signed by an expired subkey> <stale keyring>
ERROR: noble/InRelease: gpgv reported EXPKEYSIG

The throwaway subkey A is created already expired and then renewed, the same situation as the maintainer's expiry renewal. Signing succeeds only because the refreshed public keyring is imported first.

apt install from the built tree in containers (arm64 host, so these are the arm64 packages; PR CI runs amd64):

Image Mode Result
ubuntu:24.04 UNSIGNED, file:/repo, Trusted: yes ==> OK: mmt-dpi 1.10.2-a84a6050 installed on Ubuntu 24.04.5 LTS (noble)
debian:13 UNSIGNED, file:/repo, Trusted: yes ==> OK: mmt-dpi 1.10.2-a84a6050 installed on Debian GNU/Linux 13 (trixie) (trixie)
ubuntu:24.04 throwaway-signed, Signed-By Get:1 file:/repo noble InRelease [2267 B] … ==> OK: mmt-dpi 1.10.2-a84a6050 installed
debian:13 throwaway-signed, Signed-By Get:1 file:/repo trixie InRelease [2269 B] … ==> OK: mmt-dpi 1.10.2-a84a6050 installed

The unsigned legs print WARNING: UNSIGNED dry-run repository. apt is told to trust it (Trusted: yes). Every leg runs dpkg -s mmt-dpi, checks install ok installed, and requires the newest version in that suite.

Not testable here: the signed deploy and the live smoke job. Both need the real key, which does not exist yet. Their gating, the action pins and the artifact layout were reviewed. The cross-repo GITHUB_TOKEN reads of Montimage/mmt-dpi (release download, attestation API) are exercised by this PR's CI.

Actions pinned (looked up with gh api repos/actions/<name>/git/ref/tags/<tag>; all lightweight tags pointing at commits):

  • checkout v7.0.1 3d3c42e5…
  • upload-artifact v7.0.2 cf430e03…
  • download-artifact v8.0.2 9000827c…
  • upload-pages-artifact v5.0.0 fc324d35…
  • deploy-pages v5.0.1 368f8252…

Second review (WARN) — fixes

Finding 1, negative signing tests. The stale keyring is now exported without export-minimal, so it keeps the expired subkey A (the script asserts that subkey A shows validity e in it, else the test aborts). A new case revokes subkey A after signing (validity r asserted). Every negative check captures stderr and must fail with its own message: expected <B>, gpgv reported EXPKEYSIG, gpgv reported REVKEYSIG, InRelease does not carry the text of Release. A rejection for any other reason fails the test.

Mutation test (scratch copies of scripts/, real tree from the dry run below; the repository was not modified). Mutant a is the deletion the review asked for; b and c remove only the status names from the pattern, and c shows the revoked case is caught on its own:

=== mutant a: diff scripts/verify-repo.sh
@@ -58,8 +58,6 @@
   # gpgv exits 0 for a signature by an expired or revoked key (EXPKEYSIG,
   # REVKEYSIG), which apt rejects. Require exactly one good, valid signature
   # and nothing else.
-  bad=$(awk '$1 == "[GNUPG:]" && $2 ~ /^(EXPKEYSIG|REVKEYSIG|EXPSIG|BADSIG|ERRSIG|NO_PUBKEY)$/ { print $2 }' "$status")
-  [ -z "$bad" ] || die "$label: gpgv reported $(printf '%s' "$bad" | tr '\n' ' ')"
   [ "$(awk '$1 == "[GNUPG:]" && $2 == "GOODSIG"' "$status" | wc -l)" -eq 1 ] ||
     die "$label: expected exactly one good signature"
   [ "$(awk '$1 == "[GNUPG:]" && $2 == "VALIDSIG"' "$status" | wc -l)" -eq 1 ] ||
$ a/test-signing.sh "$W/tree" "$W/mut-a-out"
==> TEST ONLY: negative checks
    signatures from the wrong expected signer: rejected (expected 113019D227A8EB65B9159EF447CB17604398C787)
ERROR: verify-repo.sh rejected a keyring where the signing subkey has expired, but not with 'gpgv reported EXPKEYSIG': ERROR: bookworm/InRelease: expected exactly one good signature
exit=1
=== mutant b: diff scripts/verify-repo.sh
@@ -58,7 +58,7 @@
   # gpgv exits 0 for a signature by an expired or revoked key (EXPKEYSIG,
   # REVKEYSIG), which apt rejects. Require exactly one good, valid signature
   # and nothing else.
-  bad=$(awk '$1 == "[GNUPG:]" && $2 ~ /^(EXPKEYSIG|REVKEYSIG|EXPSIG|BADSIG|ERRSIG|NO_PUBKEY)$/ { print $2 }' "$status")
+  bad=$(awk '$1 == "[GNUPG:]" && $2 ~ /^(EXPSIG|BADSIG|ERRSIG|NO_PUBKEY)$/ { print $2 }' "$status")
   [ -z "$bad" ] || die "$label: gpgv reported $(printf '%s' "$bad" | tr '\n' ' ')"
   [ "$(awk '$1 == "[GNUPG:]" && $2 == "GOODSIG"' "$status" | wc -l)" -eq 1 ] ||
     die "$label: expected exactly one good signature"
$ b/test-signing.sh "$W/tree" "$W/mut-b-out"
==> TEST ONLY: negative checks
    signatures from the wrong expected signer: rejected (expected C458C488EE2B11F3994CD2D826E91AA1A5597609)
ERROR: verify-repo.sh rejected a keyring where the signing subkey has expired, but not with 'gpgv reported EXPKEYSIG': ERROR: bookworm/InRelease: expected exactly one good signature
exit=1
=== mutant c: diff scripts/verify-repo.sh
-  bad=$(awk '$1 == "[GNUPG:]" && $2 ~ /^(EXPKEYSIG|REVKEYSIG|EXPSIG|BADSIG|ERRSIG|NO_PUBKEY)$/ { print $2 }' "$status")
+  bad=$(awk '$1 == "[GNUPG:]" && $2 ~ /^(EXPKEYSIG|EXPSIG|BADSIG|ERRSIG|NO_PUBKEY)$/ { print $2 }' "$status")
$ c/test-signing.sh "$W/tree" "$W/mut-c-out"
==> TEST ONLY: negative checks
    signatures from the wrong expected signer: rejected (expected 8982724EEFCA94EFE3A75A67ECF128115CE77893)
    a keyring where the signing subkey has expired: rejected (gpgv reported EXPKEYSIG)
ERROR: verify-repo.sh rejected a keyring where the signing subkey is revoked, but not with 'gpgv reported REVKEYSIG': ERROR: bookworm/InRelease: expected exactly one good signature
exit=1

With the unmodified scripts, test-signing.sh exits 0 (output in the test results below).

Finding 2, required check: gate job, not dropping the filter. I chose the mmt-dpi packages-gate pattern (mmt-dpi #490):

  • pull_request no longer has paths:, so the workflow runs on every PR;
  • a cheap changes job (checkout with fetch-depth: 2, git diff --no-renames --name-only -z HEAD^1 HEAD on the merge commit) sets build=true when the PR touches scripts/**, site/** or .github/workflows/apt-repo.yml; every non-PR event builds;
  • build runs only when build == 'true';
  • apt-repo-gate runs if: always() and passes only when changes succeeded and build is success (relevant) or skipped (not relevant). A failed, cancelled or unexpectedly skipped build fails it.

Why not drop the filter: the build takes about 4 minutes, downloads ~390 MB of releases and pulls two container images for four apt installs. Spending that on every README or ci.yml PR buys nothing, and the gate costs two few-second jobs. The push trigger keeps its paths: filter (no required check there).

Local check of the filter and gate scripts (extracted from the workflow, run against a throwaway git repo with real --no-ff merge commits):

PR touching 'README.md' -> build=false
PR touching 'docs/x.md' -> build=false
PR touching 'site/é x.html' -> build=true
PR touching '.github/workflows/apt-repo.yml' -> build=true
PR touching '.github/workflows/ci.yml' -> build=false
PR touching 'scripts/a.sh' -> build=true
push -> build=true
gate changes=success relevant=false build=skipped: pass
gate changes=success relevant=true build=success: pass
gate changes=success relevant=true build=failure: FAIL
gate changes=success relevant=true build=cancelled: FAIL
gate changes=success relevant=true build=skipped: FAIL
gate changes=success relevant=false build=success: FAIL
gate changes=failure relevant='' build=skipped: FAIL

Check names to require on main: lint (workflow CI) and apt-repo-gate (workflow APT repository). Do not require Build unsigned tree and test it itself (skipped on unrelated PRs), nor Relevant paths changed?, Sign with subkey A, Deploy to GitHub Pages or Smoke test the live repository (…).

Finding 3, release deletion. Documented in AGENTS.md and the README (see "Known limit" above); no SOURCE_DATE_EPOCH input, for the reason given there.

Finding 4, live smoke freshness. The smoke job downloads this run's apt-repo-dists and apt-repo-signatures artifacts and passes them as --dists (now required with --url). smoke-install.sh reads the image's VERSION_CODENAME and architecture, then polls <url>/dists/<suite>/InRelease, main/binary-<arch>/Packages and Packages.gz every 15 s until each is byte-equal to the deployed one, for at most SMOKE_FRESH_TIMEOUT seconds (default 900, above Pages' max-age=600). On timeout it fails with the stale file named. The container then requires the newest mmt-dpi in that Packages, as the dry run does. Simulated locally: a python3 -m http.server on docker0 serving the real pool, with a stale dists/ (signed by another throwaway key) swapped for the fresh one after ~35 s:

### stale site, SMOKE_FRESH_TIMEOUT=20
==> Waiting up to 20 s for http://172.17.0.1:8765 to serve the noble files just deployed
    http://172.17.0.1:8765/dists/noble/InRelease is not the new one yet; retrying in 15 s
    http://172.17.0.1:8765/dists/noble/InRelease is not the new one yet; retrying in 15 s
ERROR: after 20 s, http://172.17.0.1:8765/dists/noble/InRelease is still not the one just deployed (stale Pages cache, or a newer deploy)
exit=1
### ubuntu:24.04: stale site, fresh deploy appears after ~35 s
==> Waiting up to 900 s for http://172.17.0.1:8765 to serve the noble files just deployed
    http://172.17.0.1:8765/dists/noble/InRelease is not the new one yet; retrying in 15 s   (x3)
    http://172.17.0.1:8765 serves the noble InRelease and Packages just deployed (45 s)
==> expecting mmt-dpi 1.10.2-a84a6050, the newest in /expect/dists/noble/main/binary-arm64/Packages
Get:1 http://172.17.0.1:8765 noble InRelease [2267 B]
==> OK: mmt-dpi 1.10.2-a84a6050 installed on Ubuntu 24.04.5 LTS (noble)
exit=0
### debian:13: same
    http://172.17.0.1:8765 serves the trixie InRelease and Packages just deployed (45 s)
==> OK: mmt-dpi 1.10.2-a84a6050 installed on Debian GNU/Linux 13 (trixie) (trixie)
exit=0

The real Pages path still runs only once the key exists.

Finding 5: trust model unchanged (see "Not in this PR").

Re-run after these fixes (same host; scratch dirs outside the repo; throwaway keys only): shellcheck scripts/*.sh clean; actionlint 1.7.12 (arm64 tarball verified with sha256sum -c against upstream actionlint_1.7.12_checksums.txt; its amd64 line matches ci.yml's pin 8aca8db9…a3d8) clean; a fresh fetch-releases.sh (40 .deb + 2 SHA256SUMS, sha256 and attestations verified), build-repo.sh (5 suites, 390101853 bytes), summarize-repo.sh, test-signing.sh (exit 0), and the four --tree smoke legs (unsigned and throwaway-signed, ubuntu:24.04 and debian:13): all ==> OK: mmt-dpi 1.10.2-a84a6050 installed.

Acceptance Criteria Verification

Criterion Status Evidence
Triggers: repository_dispatch mmt-dpi-release, workflow_dispatch, pushes to main that touch the build; signing only in apt-signing ✅ on: in apt-repo.yml; the sign job is the only one with environment: apt-signing, and it runs only when github.ref == 'refs/heads/main' and the event is not a PR
Fails before signing unless every file verifies against SHA256SUMS and its attestation ✅ fetch-releases.sh runs in build, before sign; 40/40 .deb + 2/2 SHA256SUMS verified above
dists/<codename>/{Release,InRelease,Release.gpg} and main/binary-{amd64,arm64}/Packages(.gz), each distribution's packages in its own pool ✅ build log + tree; signatures in the throwaway run
Secret from stdin into a throwaway GNUPGHOME under $RUNNER_TEMP, never printed; gpgv against the committed keyring alone before deploy ✅ sign-repo.sh, verify-repo.sh (empty GNUPGHOME), run again in deploy
After the deploy, apt-get update && apt-get install mmt-dpi in Ubuntu 24.04 and Debian 13 against the live URL ✅ (written; runs once the key exists) smoke job → smoke-install.sh --url … --keyring site/montimage-archive-keyring.gpg --dists <this run's dists> (deb822 + Signed-By; checks that the served keyring equals the committed one, waits until the served InRelease/Packages are the ones just signed, then requires the newest version; simulated locally above)
Unsigned dry run that never deploys until the key exists ✅ detect-signing.sh → mode=dry-run; deploy needs mode == 'signed'

Docs: README ("How the repository is built", dry-run status, resolute in the suite list) and AGENTS.md (single Pages deployer, script commands).

Not in this PR

  • Signed-By could be narrowed to subkey A (<keyring> <SUBA>!) as a follow-up.

…#2)

Fetch the last 2 mmt-dpi Releases and verify SHA256SUMS and every .deb
against SHA256SUMS and their build-provenance attestations; build one
pool per suite with apt-ftparchive, reproducibly; sign with subkey A
from stdin in a throwaway GNUPGHOME; verify with gpgv against the
committed keyring alone; test the signing path with a throwaway key;
smoke-install mmt-dpi in containers.
…#2)

One workflow deploys Pages, so a site-only deploy can no longer erase
dists/ and pool/. Runs on repository_dispatch, workflow_dispatch, pushes
to main and pull requests; without the signing key it is an unsigned
dry run that never deploys.
- verify-repo: fail on EXPKEYSIG/REVKEYSIG/EXPSIG/BADSIG/ERRSIG/NO_PUBKEY
  (gpgv exits 0 for an expired key), require one GOODSIG and VALIDSIG,
  and refuse an empty --expect-signer.
- fetch-releases: the canonical names in the attested SHA256SUMS must
  equal the release's canonical assets; fail clearly if listing fails.
- build-repo: a suite without packages fails the build.
- Concurrency: only main runs use the pages group.
- test-signing: primary never expires; add an expired-keyring negative.
- smoke: expect the newest mmt-dpi of the container's own suite.
- Pass only whether the secrets are set to the detect step; unset the
  key in sign-repo.
@luongnv89

Copy link
Copy Markdown
Contributor Author

Review of PR #5: build and publish the signed APT repository (#2)

Verdict: WARN

The PR is safe to merge. Nothing blocks it, every acceptance criterion of #2 is met, and every security property I tested holds. The WARN covers three things:

  • The regression test for the PR's most serious fix (gpgv exits 0 when the key has expired) does not test that fix.
  • The new build check cannot be made a required check as the workflow stands.
  • Two low-severity behaviours should be documented.

Reviewed head: 1d1fa8d8450f923a84446831e48b01d32abe33bf (4 commits). Review only: nothing was fixed, committed, pushed or approved.

What I ran (scratch dirs outside the repo, throwaway keys only)

Host: Ubuntu 24.04 arm64, gh 2.87.3, apt 2.8.3, GnuPG 2.4.4, Docker. Containers used apt 3.0.3 (debian:13) and apt 2.8.3 (ubuntu:24.04).

Check Result
Full dry run against the real releases: fetch-releases.sh, then build-repo.sh, then summarize-repo.sh ✅ v1.10.2 and v1.10.1: 40 .deb files and 2 SHA256SUMS, each matched against SHA256SUMS and its attestation; 5 suites with 8 packages each; 66 files, 373 MiB
Rebuild from the same inputs at a later time, then diff -r plus comparison of mtimes and sizes ✅ byte-identical, mtimes identical
Release fields (dists/resolute/Release) ✅ Origin and Label Montimage, Suite = Codename, amd64 arm64, main, Date = newest published_at, no Valid-Until
Pool layout ✅ pool/<suite>/main/m/mmt-dpi/<pkg>_<ver>_<arch>.deb; the same version sits in 5 pools without colliding; Filename: is per suite
Ubuntu 26.04 codename ✅ resolute: changelogs.ubuntu.com/meta-release (Dist: resolute, 26.04.1 LTS), Launchpad API (resolute 26.04 Supported), and VERSION_CODENAME=resolute in the ubuntu:26.04 image
Action pins, checked with gh api repos/actions/<a>/git/ref/tags/<t> ✅ all 5 match their tag comments (checkout v7.0.1, upload-artifact v7.0.2, download-artifact v8.0.2, upload-pages-artifact v5.0.0, deploy-pages v5.0.1). All are GitHub-owned, and the repo enforces sha_pinning_required and allows only GitHub-owned actions
AI trailers in the commits or PR body; .deb, key or binary files tracked ✅ none

Focus area 1: fail-closed inputs (fault injection)

I ran fetch-releases.sh through a gh wrapper that serves the cached verified assets and injects one fault per run:

Fault Result
none (control) exit 0
one .deb with a byte appended exit 1: SHA256 mismatch
SHA256SUMS altered exit 1: attestation check failed for SHA256SUMS
.deb replaced by a different, attested .deb exit 1: SHA256 mismatch
canonical asset missing from the release listing exit 1: names in SHA256SUMS and in the assets differ
asset listed but not downloaded exit 1: was not downloaded
attestation checked against the wrong tag (--source-ref) exit 1, on SHA256SUMS
the same, applied to .deb files only exit 1: attestation failed for mmt-dpi-security_debian-12_amd64.deb
attestation from a different signer workflow exit 1
no attestation (different --repo) exit 1

No failed run wrote inputs.tsv. Package, Version and Architecture come from dpkg-deb -f and are checked against the file name.

Focus area 3: signing (my own throwaway-key tests, independent of test-signing.sh)

Case Raw gpgv verify-repo.sh / sign-repo.sh
CI path: secret on stdin, --local-user A! — ✅ 5 InRelease and 5 armoured Release.gpg files; passphrase and key absent from all output; throwaway GNUPGHOME removed
expired subkey (signed while valid) exit 0, EXPKEYSIG ✅ rejected: gpgv reported EXPKEYSIG
revoked subkey (A revoked in the keyring) exit 0, REVKEYSIG ✅ rejected: gpgv reported REVKEYSIG
sign-repo.sh with the revoked keyring — ✅ refuses to sign
sign-repo.sh with an expired, unrenewed subkey — ✅ refuses to sign
signed by subkey B, --expect-signer A GOODSIG ✅ rejected
empty --expect-signer — ✅ rejected
InRelease body tampered (BADSIG) exit 1 ✅ rejected
Packages tampered after signing — ✅ rejected (size or hash)
keyring without the signer (NO_PUBKEY) exit 2 ✅ rejected
Release.gpg with an extra unknown-key signature exit 2 ✅ rejected
Release.gpg missing — ✅ rejected
detect-signing.sh, every partial combination — ✅ exit 1; all set gives signed, none set gives dry-run

The implementer's fix is correct: gpgv exits 0 for EXPKEYSIG and REVKEYSIG, and verify-repo.sh catches both.

Workflow review:

  • sign runs only when github.event_name != 'pull_request' && github.ref == 'refs/heads/main', in apt-signing, whose branch policy is main only.
  • The detect step receives only booleans (secrets.X != '').
  • The key reaches the script through the printf builtin on stdin, and the passphrase through fd 3.
  • set -x appears nowhere, and no step echoes its environment.
  • The committed keyring is imported before the secret.
  • deploy runs gpgv again on the full tree, with --expect-signer.

Focus area 4: deploy and concurrency

  • Folding pages.yml in is right. A second deployer would replace the whole site and erase dists/ and pool/. build-repo.sh copies site/ (so index.html now, and the keyring once committed) into every tree, and refuses a site/ that contains dists/ or pool/ (tested).
  • The live site after merge: pages.yml is deleted in the merge commit, so the merge push does not run it. apt-repo.yml runs on that push in dry-run mode, and deploy requires mode == 'signed'. The current placeholder (Not Yet Live, last modified 2026-11-11 17:47 GMT) stays up and is not wiped. Edits to site/ are not published until the key exists (stated in the README).
  • Concurrency: the pages group applies only to non-PR runs on refs/heads/main. It is set at workflow level, so build, sign, deploy and smoke run strictly one run at a time. Two main runs cannot race. A pending run can be replaced by a newer one, which is harmless because every run lists the releases again. PR runs and runs on other branches get apt-repo-<ref>, and PR runs cancel their older runs.
  • A PR cannot deploy: sign is skipped on PRs, deploy depends on sign's output, and both environments allow main only.

Focus area 5: triggers

repository_dispatch mmt-dpi-release, workflow_dispatch, a push to main with path filters, and pull_request with no secrets. There is no pull_request_target. client_payload.tag is passed only through env: and only echoed; the releases are always listed again. No ${{ }} appears inside any run:.

Focus area 6: reproducibility, and Date moving backwards (tested with apt over HTTP)

  • A tree with the same Date but changed indexes is accepted, and the new Packages are fetched.
  • A tree with an older Date is downloaded, and then silently discarded. apt-get update exits 0 with no warning, and the client keeps the newer cached indexes. Same result on apt 2.8.3 and apt 3.0.3.

So apt does not reject the older Date with an error, but it does not use the new tree either (see finding 3).

Focus area 7: smoke tests

  • Trusted: yes (the deb822 form of [trusted=yes]) is set only when there is no keyring. It is reachable only through --tree, from the build job's dry-run steps, and prints a banner.
  • --url requires --keyring. The live smoke uses deb822 with Signed-By, and checks first that the served keyring is byte-equal to the committed one.

Focus areas 8–10: supply chain, docs, PR body

  • Every job sets its own permissions; pages: write and id-token: write are set only in deploy. Every checkout sets persist-credentials: false. No tool is downloaded in apt-repo.yml, and ci.yml's checksums are unchanged.
  • README and AGENTS.md match the behaviour, and the dry-run status is stated at the top of the README.
  • The PR body is accurate, except for one claim (finding 3).

Findings

  1. [test-coverage, medium] The negative test for the expired subkey never reaches the EXPKEYSIG check. scripts/test-signing.sh:73

    • The test exports its stale keyring with --export-options export-minimal while subkey A is expired. GnuPG 2.4 then drops the expired subkey: the stale keyring holds only the primary key and subkey B.
    • As a result, verify-repo.sh fails with NO_PUBKEY (gpgv exit 2) instead of EXPKEYSIG.
    • Mutation test: with the EXPKEYSIG|REVKEYSIG|… check deleted from verify-repo.sh, test-signing.sh still exits 0 and still prints "keyring where the signing subkey has expired: rejected (apt would refuse it)". CI therefore does not protect the PR's most serious fix.
    • The production code is still safe twice over: when I removed the status check, the "exactly one GOODSIG" check still rejected my real expired-subkey tree.
    • There is no revoked-subkey case at all.
    • Fix:
      • export the stale keyring without export-minimal, and assert that EXPKEYSIG appears in the error;
      • add a case where subkey A is revoked and assert that REVKEYSIG appears;
      • stop discarding stderr (2>/dev/null), so each negative check asserts why it failed.
  2. [ci, medium] Build unsigned tree and test it cannot be required as the workflow stands. .github/workflows/apt-repo.yml:23-28

    • pull_request has a paths: filter. A PR that touches only README, docs or ci.yml never reports this check, so a ruleset that required it would leave such PRs waiting forever.
    • mmt-dpi met the same problem (#490) and solved it with an always-reporting packages-gate job.
    • Fix: drop the PR paths: filter, or add a changes job and an always-reporting gate job.
  3. [correctness and docs, low] When the newest release is deleted, existing clients keep stale indexes.

    • Measured above: apt silently ignores a Release with an older Date. Clients that ran apt-get update before the deletion keep the Packages entries for the deleted release. apt-get install of those entries then gets a 404 from the pool, until a release newer than the deleted one is published. New clients are unaffected.
    • Security is not at risk, since the client fails closed.
    • The PR body calls this "a documented edge case", but it is documented only in the PR body. README, AGENTS.md and the scripts say nothing about it.
    • Fix: add a line to AGENTS.md ("do not delete or demote a published mmt-dpi release; publish a newer one instead"). Optionally, add a workflow_dispatch input that overrides SOURCE_DATE_EPOCH.
  4. [correctness, low] The live smoke does not check that it sees the new deploy.

    • Pages serves with cache-control: max-age=600.
    • smoke-in-container.sh sets an expected version only for file:/repo.
    • Against the live URL, the smoke can therefore pass against the previous deploy's cached InRelease. It can also fail on a hash mismatch when the cached InRelease and Packages come from different deploys, which the 5 × 20 s retries might not outlast.
    • Fix: before running apt, poll until the served dists/<suite>/InRelease equals the one just signed, then require the newest version, as the dry run does.
  5. [note, out of scope] Signed-By trusts every key in the keyring.

    • Signed-By trusts the whole keyring, which also contains subkey B, the mmt-dpi tarball key. So a compromise of B would also let it sign APT metadata.
    • This follows docs/signing-key.md, so it is not a defect of this PR.
    • apt can narrow trust to a single subkey with Signed-By: <keyring> <SUBA>!, a possible follow-up for the user docs.
    • Related: export-minimal, as docs/signing-key.md prescribes, drops any subkey that has already expired. This is harmless while renewal happens before expiry.

Acceptance criteria (#2)

Criterion Status
Triggers; signing only in apt-signing ✅
Fails before signing unless every file verifies against SHA256SUMS and its attestation ✅ (9 injected faults, all fail closed)
dists/<codename>/{Release,InRelease,Release.gpg}, Packages(.gz), a pool per suite ✅
Secret on stdin into a throwaway GNUPGHOME, never printed; gpgv against the committed keyring alone ✅
apt-get install in Ubuntu 24.04 and Debian 13 against the live URL after deploy ⚠ written but unexercised until the key exists (finding 4)
Unsigned dry run that never deploys ✅

Required checks for the main ruleset after merge

  • Keep lint (workflow CI).
  • Add Build unsigned tree and test it (workflow APT repository), only after finding 2 is fixed. Until then, keep lint alone.
  • Never require Sign with subkey A, Deploy to GitHub Pages or Smoke test the live repository (…). They are skipped on every PR by design.

Uncertainty

  • The signed deploy and the live smoke could not run, because no key exists.
  • The cross-repo GITHUB_TOKEN reads succeeded in the PR's CI run 43103428814.
  • The tamper runs served cached assets through a wrapper; the attestation calls were real.
  • Findings 1–4 come from executed tests; finding 5 is a judgement.

- test-signing.sh: export the stale keyring without export-minimal, so it
  keeps the expired subkey A and verify-repo.sh sees EXPKEYSIG instead of
  NO_PUBKEY; add a revoked-subkey case (REVKEYSIG); every negative check
  now asserts its error message instead of discarding stderr.
- apt-repo.yml: run on every pull request; a `changes` job decides whether
  the build runs, and the always-reporting `apt-repo-gate` job is the check
  to require (the mmt-dpi packages-gate pattern).
- Live smoke: wait until Pages serves the InRelease and Packages just
  signed (bounded, default 900 s), then require the newest mmt-dpi.
- AGENTS.md, README: never delete or demote a published mmt-dpi release;
  apt silently ignores a Release with an older Date.
@luongnv89

Copy link
Copy Markdown
Contributor Author

Re-review at 8002524

Verdict: PASS

The delta (git diff 1d1fa8d..80025247, one commit, 6 files) resolves findings 1–4. Finding 5 was out of scope and is unchanged. The delta introduces nothing blocking, and two low-severity notes remain (below). CI is green at 80025247: lint, Relevant paths changed?, Build unsigned tree and test it and apt-repo-gate passed; sign, deploy and the live smoke were skipped by design. Review only: nothing was fixed, pushed, approved or merged.

Findings 1–5

# Status Evidence (all run here, scratch copies only)
1. Negative test for the expired subkey ✅ Resolved The unmodified test-signing.sh passes and rejects each case with its own message: EXPKEYSIG, REVKEYSIG, wrong signer, tampered Release. Mutation test: with the EXPKEYSIG|REVKEYSIG|… check removed from verify-repo.sh, test-signing.sh now exits 1 with rejected … but not with 'gpgv reported EXPKEYSIG'. With that check and the GOODSIG-count check both removed, it exits 1 with accepted a keyring where the signing subkey has expired. The new revoked-subkey case revokes A after signing and asserts that A has validity r in the exported keyring and that the message is gpgv reported REVKEYSIG. The stale keyring now keeps A and asserts that its validity is e. stderr is no longer discarded.
2. The build check could not be required ✅ Resolved pull_request has no paths:, so the workflow runs on every PR. Details below.
3. Date moving backwards ✅ Resolved (documented) AGENTS.md "Never delete or demote a published mmt-dpi release" and README step 2 now state the behaviour I measured, and the README link to that AGENTS.md heading resolves.
4. Live smoke freshness ✅ Resolved, with note A --url now needs --dists. The smoke waits until the served InRelease, Packages and Packages.gz equal the files just signed, byte for byte, then requires the newest mmt-dpi. Details below.
5. Signed-By trusts subkey B ○ Unchanged (out of scope, a design matter for docs/signing-key.md) —

changes and apt-repo-gate

Gate logic. I extracted the run: blocks from the workflow and ran them under every combination:

changes relevant build gate
success true success pass
success false skipped pass
success true failure / cancelled / skipped fail
success false success fail
failure / cancelled / skipped — skipped fail
  • The gate fails closed whenever the build fails or is cancelled.
  • It runs with if: always(), so it also runs in a cancelled run and fails there.
  • If the gate itself is cancelled, the check is not satisfied.
  • It has permissions: {} and a 5-minute timeout.

Path filter. I ran it against throwaway repositories with real --no-ff merge commits:

Change build
docs only false
ci.yml only false
scripts/ true
site/ true
apt-repo.yml true
a file moved out of scripts/ true (thanks to --no-renames)
any non-PR event true
no HEAD^1 job exits 128, so the gate fails (fail closed)

Reports on every PR, including docs-only ones: yes. The workflow has no PR path filter now. A docs-only PR runs only changes and apt-repo-gate, and the gate passes there because the build is skipped.

Unchanged: sign still needs a successful build, and push and dispatch runs always build.

Live-smoke wait loop

I tested it against a local HTTP server on the docker bridge, with the real tree and throwaway signatures:

Case Result
site always stale, SMOKE_FRESH_TIMEOUT=20 ERROR: after 20 s, …/InRelease is still not the one just deployed, exit 1 after 30 s
stale, then fresh after 20 s waits 30 s, then OK: mmt-dpi 1.10.2-a84a6050 installed on Debian 13, checked against the newest version in the deployed Packages
--url without --dists; a non-numeric timeout clear error
  • The loop is bounded: the deadline is checked after each attempt, so it can overshoot by at most one 15 s sleep plus one round of requests.
  • It is not an infinite loop.

Stray commit and branch

  • db118b9 ("c", author t <t@t>) appears only in the local reflog of feat/2-build-apt-repo, before a reset: moving to 1d1fa8d.
  • It is not an ancestor of 80025247, and the PR timeline never had it.
  • It held the same change plus a trailing x line in README.md. git diff db118b9 80025247 shows that the pushed commit drops the x. At head, README.md and AGENTS.md have no bare x line.
  • There is no branch or ref named pr, local or remote. The remote has only main and feat/2-build-apt-repo.
  • 80025247 is authored by Luong NGUYEN and has no AI trailers; neither does the PR body.
  • The PR body is updated and accurate for the delta.

Notes (low severity, not blocking)

A. A server that never responds is not bounded.

  • Neither curl in scripts/smoke-install.sh (the keyring download and the wait loop) has --max-time, and the smoke job has no timeout-minutes.
  • Against a server that accepts connections and never answers, smoke-install.sh with SMOKE_FRESH_TIMEOUT=5 hung until my outer timeout 60 killed it (exit 124). In CI it would run until the 6 h job default.
  • Suggested fix: curl --connect-timeout 10 --max-time 60 and timeout-minutes: 30 on smoke.

B. The gate passes with an empty relevant output. apt-repo-gate passes when changes succeeded but wrote no build output, because an empty value falls through to "skipped". The filter always writes the output when it succeeds, so this cannot happen today. Requiring RELEVANT to be true or false would make the gate robust against future edits of the filter.

Required checks for main after merge

  • lint (workflow CI)
  • apt-repo-gate (workflow APT repository)

Do not require Build unsigned tree and test it, Relevant paths changed?, Sign with subkey A, Deploy to GitHub Pages or Smoke test the live repository (…).

Uncertainty

  • The signed deploy and the real Pages smoke still cannot run without the key.
  • The docs-only path was verified by simulation, not by a real docs-only PR run.

@dev-montimage dev-montimage left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent Claude Code review: WARN at 1d1fa8d (2 medium, 2 low findings), all resolved in 8002524; re-review PASS at 8002524 (mutation test confirms the EXPKEYSIG/REVKEYSIG guard, gate fails closed in every combination). CI green: lint, apt-repo-gate, Build unsigned tree and test it.

@luongnv89
luongnv89 merged commit 5f04c02 into main Nov 11, 2026
7 checks passed
@luongnv89
luongnv89 deleted the feat/2-build-apt-repo branch November 11, 2026 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build and publish the signed APT repository from mmt-dpi Releases

2 participants