Improve interactive debugging

[casparvl]

Problem
We often have issues reproducing build failures interactively. Currently, we have about three options:

1. Run with EESSI-extend. This is the easiest, but also the most 'different' from the bot's build environment. It uses EESSI for dependencies, but it doesn't run in a build container.
2. Run the bot/inspect.sh from the job dir of the failed build job. This script was designed to mimic bot/build.sh and drop you into an interactive shell in the failed build environment. However, it is very much outdated, and keeping it in sync currently requires a lot of code duplication from bot/build.sh. An example of how out-of-sync it is: it does not have any GPU support (no NVIDIA GPU support either).
3. Try to set all relevant environment variables, extracted from the build log. Then, start a build container manually with the command reported in the build log. Then, try to build to reproduce the failure. This is potentially the most 'similar' in terms of environment to the build job, but very error prone, as it's easy to miss environment variables, or steps that are executed as part of bot/build.sh. It is also very time-consuming.

Proposed solution
The solution presented here is to tightly integrate an option for interactive debugging into the bot/build.sh, that will drop you into an interactive debugging shell if EESSI_INTERACTIVE_BUILD is set. To get an environment that is as similar as possible, one should call the job script that the bot submits interactively. I.e. if a bot is configured to use the default, that would be EESSI_INTERACTIVE_BUILD=<mode> eessi-bot-software-layer/scripts/bot-build.slurm.

mode supports three options:

• shell - start a plain shell in the build container instead of running the build
• before - run the build, but start a shell (inside the compat layer, EasyBuild loaded) right before EasyBuild is called
• onfail - run the build, and start a shell when EasyBuild fails (or the build script exits non-zero)

The idea is that this is ran from the job directory left behind by the bot, with the failing build. By default, it will resume from the latest tarball of the build step (this is selectable). If accelerators are used, it will run for the last accelerator in the list by default (this is also selectable).

Limitations

• If you're running this from a different user than the bot user, some things may fail on file permission - though typically this just means that things in the shared_fs_directory can not be updated (what's there can be used as long as your user has read permissions to that dir).
• Currently, the bot/build.sh from software-layer hard-clones and links the software-layer-scripts/bot/build.sh. That results in an exit, because the script runs with set -e. My current hack is just to replace the bot/build.sh in the job dir with a one-liner: software-layer-scripts/bot/build.sh. We should fix this more properly, and maybe make the whole clone-business conditional on EESSI_INTERACTIVE_BUILD not being set.

[Neves-P]

I tried this out using the RUG Zen5 bot that builds for RTX6000 acel builds and it worked fine. Note: I didn't easily find a failed build, so I tried the shell and before options which dropped me where I expected to be dropped.

One thing I noticed, after interrupting a build (^C) it seems to start again the build process again and I couldn't figure out why. Do you also see this @casparvl ?

[laraPPr]

Testing this now for LAMMPS. It works from the bot account like a charm. But if you try running it from another account you run into trouble.

[laraPPr]

Oh this is anoying when you exit before it continues and now I'm stuck until LAMMPS fails in an hour let me try killing the proccess.

[casparvl]

Is it a bug, or is it a feature? Yes, the build continues after you exit the debug shell. It is essentially a breakpoint that enables you to play around, then continue the build process. Maybe we can make that behaviour dependent on (another) environment variable, I don't know how EESSI that will be. For now, I think this is better than nothing - you might actually want to continue the build process, because you want to know if you encounter more bugs, or if you fixed anything by whatever you did interactively.

I agree though that it is extremely annoying that a ctrl+C doesn't work. I'm not sure why that is happening, and if it can easily be resolved. I'll check with my AI friend....

[casparvl]

Tried to fix it, it didn't work. The AI couldn't replicate it outside our container+startprefix environment anyway, so it was a bit of guess work in debugging. I'll look further next week.

[casparvl]

This seems to be really tricky. If I understand things correctly, the reason that the ctrl+C isn't working is because after existing the debug shell is that a ctrl+C kills process that are in the same process group as your tty. Once the debug shell takes control, that's a different process group than the one running the EasyBuild process (and the container, and startprefix, etc, etc). That's fine: this means you could run manual commands there, and press ctrl+C to kill your manual process, without killing your entire container along with it.

The issue is that once you exit the debug shell, you want it to hand control back to the process group runnin EasyBuild/container/startprefix. That seems to happen automatically in a regular shell - but doesn't happen here since we are in a container. Claude then tried to 'force' it with a small python snippet in ec9035c but that failed with:

  tcsetpgrp: fg=882 mine=0
  Traceback (most recent call last):
    File "<string>", line 5, in <module>
      os.tcsetpgrp(fd, os.getpgrp())
      ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  ProcessLookupError: [Errno 3] No such process
  tcsetpgrp snippet failed (python3: /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/usr/bin/python3)

This made Claude understand the issue: that process group doesn't exist IN the container, so we can't hand control back to it.

Then, it tried to add +m to run a shell without job control. The way I understand is that this would run the debug shell without job control "so that there is nothing to give back". Alas, it failed in the same way.

Claude is now churning on trying to actually reproduce the behavior itself using unsare, so it can do more fast trial-and-error. Seems it's getting somewhere, so I expect I will follow up on this message with another commit that attempts to fix things.