The problem with special mounts

The fuse and chroot setup works well for many tools, but not all.

Some tools rely on the /proc filesystem to operate. The best example is cargo and rustc that fails with the following message:

Thread 'main' panicked at 'failed to get current_exe: no /proc/self/exe available. Is /proc mounted?

The goal of this series is to solve this, so you never have to worry about the problem stated here.

There are also problems with go, node, javac that have been observed in the bb-deployments repo. Just like rustc, the go compiler tries to find itself through /proc/self/exe and find the GOROOT based on that path. We have used bb-deployments as the development repo when diagnosing this problem and developing patches, it can be built successfully when the filesystems are mounted.

Mounting the special filesystems in the worker?

The action is prepared in the worker, the core is the local build executor where Execute executes an action. It creates sandbox directory with all the input files, through a virtual directory abstraction, so the files can be loaded lazily through fuse. The worker also creates the special character devices. These are needed for chroot actions can use /dev/null, /dev/full et cetera.

Though chroot configuration is done in the runner, which prepares and executes the action process described by the worker. This indicates that we should keep the configuration of the filesystems in the runner. Combined with the worker's virtual filesystem, where the directories can be purely virtual we have nothing to pin mounts on in the worker.

Mounting the special filesystems in the runner

We want to mount the filesystems in the runner which has a local Directory, just with the limitation that we should not use filepaths. We want to use the at-family of syscalls. This is not technically a requirement, as we can construct absolute file paths here, but we value compartmentalization that the at-syscalls give us. We know that code for one action can not address paths outside its input root.

The search for "mountat"

This led to a search for the missing "mountat" syscall, which can be implemented with the new Linux mount API, and a lot of consternation in trying to implement unmountat with the same API.

The path to better mounting in bb-remote-execution is tracked here. This series aims to document the details and a solution. Patches that you can apply are listed here, as well as discussion about merging them into Buildbarn.

Note that the special character devices that are already in use are not actually mounted, and can be created in the virtual layer. Which is why they are not handled in the runner, and the chroot configuration is split between the worker and the runner,