Chroot in Buildbarn

Buildbarn workers and runners have functionality to chroot into the input root, this is valuable if the REAPI action input root contains all the required tools, the action will then be completely isolated from the host machine.

Buildbarn's remote build engine, RBE, is split in two programs, the worker and the runner. The worker is responsible for communicating with the scheduler and by extension the build client (bazel, buck2, ...). It accepts an action to perform, downloads the necessary files (if they are not already cached locally), prepares the action in its own sandbox directory and tells the runner to execute it. This allows for more process isolation and better security controls. The runner is simple and executes commands on input files fed by the worker.

Configuration

To enable chroot you just need a couple of settings, first enable chroot, then create the special character devices in the action root. The well-known family of /dev/null, /dev/random, and so on. They are not technically mounted, but created through mknodat.

Runner.jsonnet [runner proto documentation]: [runner proto documentation]: https://github.com/buildbarn/bb-remote-execution/blob/96c4fdce659fabfaba7ee2a60fd4e2ffab8352e2/pkg/proto/configuration/bb_runner/bb_runner.proto#L39

{
  chrootIntoInputRoot: true,
  ...
}

Worker.jsonnet [worker proto documentation]: [worker proto documentation]: https://github.com/buildbarn/bb-remote-execution/blob/96c4fdce659fabfaba7ee2a60fd4e2ffab8352e2/pkg/proto/configuration/bb_worker/bb_worker.proto#L258

{
  ...
  buildDirectories: [
    {
      runners: [
        {
        ...
          inputRootCharacterDeviceNodes: [
            "full",
            "null",
            "random",
            "tty",
            "urandom",
            "zero"
          ],
        }
      ] // runners
    }
  ] // build directories
}

Fuse workers

If your action roots have many files, you do not want to use the hardlinking workers. Especially if many of the tool files are never read, it is better to use fuse workers. There the input root is prepared in a fuse or NFSv4 filesystem and downloaded lazily.

Fuse setup is trickier, thankfully we have already demonstrated it in bb-deployments. It is important to set up the docker bind mounts correctly, so the worker and runner can communicate. Without this, the worker fails to setup the input root and fallback code in the fuse layers tries to launch programs that do not exist.

The worker needs privileges for setting up the fuse filesystem and preparing the chroot.

services:
  worker-fuse:
  ...
    privileged: true
    volumes:
      - ../.mounts/apps/worker-fuse:/app:ro
      - type: bind
        source: ../.mounts/data/worker-fuse
        target: /worker
        bind:
          # Bidirectional mount to expose the FUSE mount.
          propagation: shared

  runner-fuse:
    ...
    privileged: false
    volumes:
      - ../.mounts/apps/runner:/app:ro
      - type: bind
        source: ../.mounts/data/worker-fuse
        target: /worker
        bind:
          # HostToContainer mount to use the FUSE mount.
          propagation: slave