Skip to content

Fix the 261 CI job losing its runner - #67

Merged
Tsyklop merged 2 commits into
mainfrom
ci/fix-261-runner-loss
Nov 6, 2026
Merged

Tsyklop merged 2 commits into
mainfrom
ci/fix-261-runner-loss

Conversation

@Tsyklop

@Tsyklop Tsyklop commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

The "Check the plugin for 261" job has failed on every CI run since July 2026. It always ended 23–27 minutes in, with "The self-hosted runner lost communication with the server" and no log.

Cause

The runner pods run out of memory when two of these jobs share a node. It's not a failing test.

I measured this with a temporary monitor that posted memory, disk and process snapshots to a check run every 10 s, so the data survived the pod being killed.

  • Runner nodes: arc-runners-large pods run on nodes with 16 GB RAM and 8 CPUs. The runner container has no memory limit (memory.max = max), and several runner pods share a node.

  • Memory per job: one job holds 9–10 GB of process memory during its test phase:

    Process Memory
    Gradle daemon 3.1 GB
    Test JVM ~3 GB
    Kotlin compiler daemon, idle since compilation 2.3 GB
    buildSrc's Kotlin compiler daemon, idle 1.1 GB

    A node has about 14 GB available, so two jobs don't fit.

  • One kill observed directly: a re-run 261 job shared a node with this PR's 262 job.

    • The node's available memory fell from 4.4 GB to 0.6 GB while free disk stayed at 12 GB.
    • The 261 job's "Run tests" step ended at 19:50:58 and its runner died right after.
    • The node got its memory back at 19:51:04.
    • GitHub reported "lost communication" about 10 minutes later, at 20:00:36.
    • The 262 job carried on and passed.
  • Why it's always 261: the job that is further along gets killed. 261 configures and compiles a few minutes faster than 262, which provisions a JDK 25 toolchain and forks javac. So when both matrix jobs start together, 261 is the bigger pod when the node runs out. The lost jobs ended 23–27 min in, which is a kill at about 13–17 min plus GitHub's roughly 10-minute detection delay.

  • When 261 runs alone it passes, in about 18 minutes.

Fix

  • max-parallel: 1: the two matrix jobs of a run never overlap, so they can't share a node. A CI run now takes about 40 minutes instead of about 27.
  • Stop the build daemons before tests: compile in a separate step, then stop the Gradle daemon and the Kotlin compiler daemons. The test step starts a fresh Gradle daemon and finds every compile task UP-TO-DATE. A job's test phase drops from about 9 GB to about 3.5 GB, which leaves room for jobs of other runs that land on the same node, for example two PRs at once. The extra configuration pass costs about 45 s.

Verification

  • With the monitor and the daemon stop, both jobs passed. The test step compiled nothing (0 compile tasks executed, 32 UP-TO-DATE), and its peak memory was 3.5 GB for 261 and 3.7 GB for 262.
  • This exact commit has run on this PR too; see the checks below.

Follow-ups (not in this PR)

  • Runner pods: a memory request on the arc-runners-large pods (for example 12 Gi) would stop Kubernetes from putting two heavy jobs on one node. That is the real fix, and max-parallel could then go.
  • Kotlin daemon heap: kotlin.daemon.jvm.options=-Xmx1g in gradle.properties has no effect. KGP reads that name only as a JVM system property; the Gradle property is kotlin.daemon.jvmargs. The Kotlin daemon inherits the Gradle daemon's -Xmx2g.
  • Actions cache: the cache doesn't work for this repo (restore returns HTTP 400, and no caches exist). Every job downloads Gradle, the IDEs and the plugins from scratch, which is about 9 minutes and 40 GB of disk.
  • Slow test: HyperskillCheckEduTaskMessageTest.test solve all edu tasks in topic requests /api/steps, which its mock server rejects. It then waits out two 60 s client timeouts on every platform.

The arc-runners-large pods have no memory limit and share 16 GB nodes, so
two jobs of this workflow can run on one node. A job holds ~9-10 GB in its
test phase: the Gradle daemon (3.1 GB), the test JVM (~3 GB) and two idle
Kotlin compiler daemons (2.3 GB for the build, 1.1 GB for buildSrc). Two of
them don't fit, and the node kills the pod of the job that is further along.
That is the 261 job, which configures and compiles faster than 262, and
GitHub reports it as "The self-hosted runner lost communication with the
server".

- Run the matrix jobs one at a time, so they never share a node.
- Compile in a separate step and stop the Gradle and Kotlin daemons before
  running tests. The test phase drops to ~3.5 GB, which leaves room for jobs
  of other workflow runs on the same node.
@Tsyklop
Tsyklop force-pushed the ci/fix-261-runner-loss branch from 054f15a to 568d66f Compare November 2, 2026 20:31
@Tsyklop
Tsyklop marked this pull request as ready for review November 2, 2026 22:05
@Tsyklop
Tsyklop merged commit d5da829 into main Nov 6, 2026
0 of 2 checks passed
@Tsyklop
Tsyklop deleted the ci/fix-261-runner-loss branch November 6, 2026 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant