Repository navigation
Fix the 261 CI job losing its runner - #67
Merged
Merged
Conversation
The arc-runners-large pods have no memory limit and share 16 GB nodes, so two jobs of this workflow can run on one node. A job holds ~9-10 GB in its test phase: the Gradle daemon (3.1 GB), the test JVM (~3 GB) and two idle Kotlin compiler daemons (2.3 GB for the build, 1.1 GB for buildSrc). Two of them don't fit, and the node kills the pod of the job that is further along. That is the 261 job, which configures and compiles faster than 262, and GitHub reports it as "The self-hosted runner lost communication with the server". - Run the matrix jobs one at a time, so they never share a node. - Compile in a separate step and stop the Gradle and Kotlin daemons before running tests. The test phase drops to ~3.5 GB, which leaves room for jobs of other workflow runs on the same node.
Tsyklop
force-pushed
the
ci/fix-261-runner-loss
branch
from
November 2, 2026 20:31
054f15a to
568d66f
Compare
Tsyklop
marked this pull request as ready for review
November 2, 2026 22:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The "Check the plugin for 261" job has failed on every CI run since July 2026. It always ended 23–27 minutes in, with "The self-hosted runner lost communication with the server" and no log.
Cause
The runner pods run out of memory when two of these jobs share a node. It's not a failing test.
I measured this with a temporary monitor that posted memory, disk and process snapshots to a check run every 10 s, so the data survived the pod being killed.
Runner nodes:
arc-runners-largepods run on nodes with 16 GB RAM and 8 CPUs. The runner container has no memory limit (memory.max = max), and several runner pods share a node.Memory per job: one job holds 9–10 GB of process memory during its test phase:
buildSrc's Kotlin compiler daemon, idleA node has about 14 GB available, so two jobs don't fit.
One kill observed directly: a re-run 261 job shared a node with this PR's 262 job.
Why it's always 261: the job that is further along gets killed. 261 configures and compiles a few minutes faster than 262, which provisions a JDK 25 toolchain and forks
javac. So when both matrix jobs start together, 261 is the bigger pod when the node runs out. The lost jobs ended 23–27 min in, which is a kill at about 13–17 min plus GitHub's roughly 10-minute detection delay.When 261 runs alone it passes, in about 18 minutes.
Fix
max-parallel: 1: the two matrix jobs of a run never overlap, so they can't share a node. A CI run now takes about 40 minutes instead of about 27.UP-TO-DATE. A job's test phase drops from about 9 GB to about 3.5 GB, which leaves room for jobs of other runs that land on the same node, for example two PRs at once. The extra configuration pass costs about 45 s.Verification
UP-TO-DATE), and its peak memory was 3.5 GB for 261 and 3.7 GB for 262.Follow-ups (not in this PR)
arc-runners-largepods (for example 12 Gi) would stop Kubernetes from putting two heavy jobs on one node. That is the real fix, andmax-parallelcould then go.kotlin.daemon.jvm.options=-Xmx1gingradle.propertieshas no effect. KGP reads that name only as a JVM system property; the Gradle property iskotlin.daemon.jvmargs. The Kotlin daemon inherits the Gradle daemon's-Xmx2g.HyperskillCheckEduTaskMessageTest.test solve all edu tasks in topicrequests/api/steps, which its mock server rejects. It then waits out two 60 s client timeouts on every platform.