Fix LD_LIBRARY_PATH inside startprefix for native GPU builds so that nvidia-smi works there

[casparvl]

Make sure /.singularity.d/libs gets added to the LD_LIBRARY_PATH inside the startprefix environment, so that nvidia-smi also works there.

Edit: Just to clarify for future reference (in regards to @boegel 's review):

Why is this suddenly required, and it wasn't before?

nvidia-smi was always broken inside a startprefix environment. The problem is not the container, you can do eessi_container.sh --nvidia all and run nvidia-smi just fine. It only breaks if you run startprefix in the container (or, I guess, outside the container).

Or was nvidia-smi simply always broken in the build container, and this only surfaces now because EasyBuild's get_gpu_info function relies on nvidia-smi working correctly?

Yes, it was always broken, but we never used the get_gpu_info from EasyBuild before, so we never noticed. And our other scripts in EESSI/software-layer-scripts would set LD_LIBRARY_PATH in a 'just-in-time' fashion (see e.g. here, and only locally to a function. I guess we could actually get rid of those as a form of code cleanup... But wanted this fix in first and didn't dare to remove the other stuff yet - afraid to break things.

We only noticed because this LAMMPS build was using an EasyBuild hook that would invoke get_gpu_info. And in debugging a build issue there, I found out it unexpectedly returned an empty list (which was a blocker, as it would then try to cross-compile using stubs libraries, but it would still run unit tests -those would obviously fail with the stubs libraries).

More details in this comment of that LAMMPS PR.

[boegel]

@casparvl We is this suddenly required, and it wasn't before?

Or was nvidia-smi simply always broken in the build container, and this only surfaces now because EasyBuild's get_gpu_info function relies on nvidia-smi working correctly?

[laraPPr]

I believe I had also tested at UGent with the new easyblock but now I'm not sure anymore. It does look like nvidia-smi was already broken in the build container. In our utils we were already taking this into account. But doe not do the same mitagation.

software-layer-scripts/scripts/utils.sh

Lines 151 to 168 in 6327b3a

	function nvidia_gpu_available {
	if command_exists "nvidia-smi"; then
	# We are careful here in case we are running in a container and LD_LIBARY_PATH has been wiped
	LD_LIBRARY_PATH="/.singularity.d/libs:${LD_LIBRARY_PATH}" nvidia-smi --version
	ec=$?
	if [ ${ec} -eq 0 ]; then
	echo "Command 'nvidia-smi' found."
	return 0
	else
	echo "Warning: command 'nvidia-smi' found, but 'nvidia-smi --version' did not run succesfully."
	echo "This script now assumes this is NOT a GPU node."
	return 1
	fi
	else
	echo "No 'nvidia-smi' found, no available GPU."
	return 2
	fi
	}

[boegel]

lgtm