4 posts tagged with "release"

The example configuration for bb-deployments has been updated.

This post will give an overview of what has happened since the last update summary in November 2023, and will cover important changes since then up to 2026-06-12.

Additional Buildbarn changes can be found in the bb-deployments changelog.

Major changes​

Upgrade to Bazel 9​

The Buildbarn components have been upgraded to use Bazel 9.

Remove tini (Jan 29, 2026)​

Buildbarn binaries will now relaunch themselves in a child process, if the PID is 1. This removes the need to run bb_runner through tini because zombie processes in containers are no longer an issue.

Generic gRPC stream forwarding (Jan 19, 2026)​

The gRPC servers can now forward incoming streams of specified services. This enables passing Bazel's build event stream to the same DNS name, without having to add an extra L7 router in front of the bb_storage frontend.

grpcServers: [{
  ...
  relays: [
    {
      endpoint: {
        address: 'localhost:1234',
      },
      services: [
        'build.bazel.remote.execution.v2.Execution',
        'com.google.devtools.build.v1.PublishBuildEvent',
      ],
    },
  ],
}],

OAuth Client credentials for HTTP clients (Sep 26, 2025)​

Buildbarn's HTTP client now supports OAuth Client Credential flow.

WinFSP virtual file system implementation (Aug 20, 2025)​

An implementation of WinFSP VFS has been added, enabling bb_worker to succesfully execute Bazel builds with virtualised file inputs on Windows. See the commit message for more details.

Allow files to be read by JMESPath Expressions (Aug 11, 2025)​

Buildbarn has expanded the configuration of JMESPath expressions. Instead of only specifying a string, an optional set of files can be included, whose contents will be available for the expression using the files field. The contents of the file refreshes every 60 seconds. The struct also includes optional test vectors, each test consisting of an input and expected output. The program fails to start if the expression does not produce the expected output from the input.

addMetadataJmespathExpression: {
  expression: |||
    {
      "authorization": [std.format('bearer %s', files.token)]
    }
  |||,
  files: [
    {
      key: "token",
      path: "/tokens/buildbarn",
    },
  ],
},

ZSTD compression​

Support has been added for in-transit ZSTD compression in the gRPC ByteStream layer (Jul 15, 2025), used by the CAS. This feature is a pre-requisite for supporting compressed bb_clientd. A pool of decoders and encoders can be configured with an optional upper bound of instances (Mar 12, 2026). See this commit for the gRPC client side configuration (Mar 2, 2026).

Sharding algorithm changes (Apr 8, 2025)​

The previous sharding algorithm has been replaced with Rendezvous hashing, making resharding less disruptive. This change is not backwards compatible. See Buildbarn ADR #11 for more details.

Remote authentication and authorization (Feb 20, 2025)​

HTTP and gRPC servers can now forward authentication and authorization requests to a remote service. The results are cached for a short while to reduce the load on the remote service.

NFSv4 changes​

NFSv4.1 for bb_worker is now supported (Jul 9, 2024) in addition to NFSv4.0. The versions have some substantial protocol differences, as described in the commit message.

NFSv4 has been extended to support named attributes, see the related PR (Oct 5, 2025) for more details.

Permit predeclaring multiple size class queues (Jun 4, 2024)​

Predeclared platform queue size classes are now defined by an array of size classes, previously configured by setting the maximum size class. This allows all workers for all sizes to be autoscaled down to zero. When using multiple size classes, the lowest cannot be zero.

{
  ...
  predeclaredPlatformQueues: [
    {
      instanceNamePrefix: 'testingQueue',
      sizeClasses: [1, 2, 3, 4, 5, 6, 7],
    },
    {
      sizeClasses: [1, 2, 3, 4, 5, 6, 7],
    },
  ],
  ...
}

// If the user does not care about multiple size classes:
{
  ...
  predeclaredPlatformQueues: [
    {
      sizeClasses: [0],
    },
  ],
}

macOS mount configuration changes​

The support for FUSE mounts on macOS has been removed (Mar 3, 2024) due to OSXFUSE/macFUSE being unstable. macOS users are recommended to instead use NFSv4. However the NFSv4 support for macOS 14 and older has been dropped (May 7, 2026) as macOS 15 has been out for some time.

Add support for uploading output directories as Directory messages (Dec 22, 2023)​

REv2 has added the ability to upload output directories as multiple Directory messages instead of a single Tree object, if requested by the client. bb_worker has added support for this feature, and can also force upload Directory messages to the CAS in addition to uploading the Tree object by setting

forceUploadTreesAndDirectories: true;

in the worker configuration. Forcefully uploading Directory messages has several advantages as documented in the worker Proto file:

• bb_browser is capable of displaying listings of individual directories contained in an output directory without needing to load the full Tree object from the CAS.

• Root directories of output directories created through bb_clientd's Bazel Output Service feature load slightly faster, as they can be validated without processing full Tree objects.

• Even for clients that only support output directories in the form of Tree messages, having Directory messages present means that subsequent build actions need to upload fewer objects.

The disadvantage of enabling this option is that a larger number of objects are written into the CAS.

Minor changes​

Extended digest algorithm support​

The list of supported digest algorithms has been extended with blake3 (Dec 31, 2025) and GITSHA1 (Jan 27, 2026)

Remote Action Router (Dec 8, 2025)​

This allows the scheduler to delegate action routing decisions to a remote gRPC service.

gRPC server ALTS Authentication (Nov 7, 2025)​

The gRPC server component now supports basic ATLS authentication.

Additional OIDC support​

The authenticator has been extended to support PKCE (Oct 31, 2025), and user information claims can now be extracted from an ID token (May 28, 2025) instead of the user info endpoint.

Add support for execution timeout compensation via HTTP (Oct 31, 2025)​

A configuration option has been added which allows bb_worker to control of the execution timeout timer via HTTP. It is configured by specifying two URLs which are polled by the worker: a "suspend URL", whose response controls whether the timeout timer is suspended, and a "resume URL" whose response controls whether to resume the timer. This feature is useful for when the worker timeout needs to be compensated for file downloads from outside the input root.

cron job for saving JWKSes to Kubernetes ConfigMap (Aug 19, 2025)​

Buildbarn now includes simple program which saves JWKSes in a Kubernetes ConfigMap, which can then be made available to Buildbarn with a volume mount. The program is meant to be run periodically in the cluster.

Synchronize request authorization (Jul 27, 2025)​

The scheduler can now be configured with an additional authorizer, to authorize bb_worker synchronize requests.

Add the ability to set owner user/group IDs on directories (Jun 11, 2025)​

VFS and FUSE backends now support setting directory ownership. This is useful for when running repo rules as remote actions, as Bonanza will, many of which depend on having ownership.

Deadline enforcing blob access (Feb 17, 2025)​

A decorator, DeadlineEnforcingBlobAccess, has been added which sets a configurable timeout duration. This sets an upper limit on the durations for bb_storage's incoming RPCs. For example, this is useful for reject writes that are taking an excessive amouint of time to complete.

Enable custom gRPC connection load balancing method (Jan 23, 2025)​

The default gRPC connection load balancing policy is now configurable. This enables using the round-robin strategy. See gRPC service configuration for more information.

Double the number of recommended KeyLocationMap attempts (Sep 10, 2024)​

The recommended values from keyLocationMapMaximumGetAttempts and keyLocationMapMaximumPutAttempts have been doubled. For more information about the Key Location Map, see our blog post about the topic.

HTTP server TLS support (Jun 12, 2024)​

The HTTP server configuration now supports an optional TLS configuration. This is useful when running Buildbarn without an ingress controller, for example in a bare metal environment.

Allow TLS certificate authenticator to validate URI SANs (Apr 10, 2024)​

URI subject alternative names can now be matched with JMESPath expressions when validating TLS certificates.

Custom Kubernetes service endpoint connections (Feb 27, 2024)​

Custom Kubernetes resolvers can be configured in the global configuration. The URL schema and Kubernetes API server is specified, whose endpoints are expanded and registered. This is useful in a bare-metal cluster with Buildbarn pods using the host network, where it can be difficult to establish network connections between components.

Delay file uploads until output files are closed (Feb 22, 2024)​

Before uploading the output files, bb_worker will now wait for them to be closed, with a user-specified upper bound for the waiting duration. This prevents cases where the output files are still open for writing when they are uploaded, due to the kernel closing files asynchronously.

Add support for capturing server logs (Jan 26, 2024)​

Every action now gets its own "server_logs" directory, created by bb_worker. bb_runner doesn't do anything with the directory, but a custom runner can for example use the directory to store core dumps of failed actions, which bb_worker will then include in the upload. For more information on how to configure the directory, see the commit message.

Add option to set resource limits on startup (Jan 8, 2024)​

A global option has been added which calls setrlimit(2) with the configured values as arguments. The keys in the configuration correspond to the RLIMIT_* suffixes in the setrlimit resource parameter.

global: {
  ...
  setResourceLimits: {
    "NOFILE": {
      softLimit: 1024,
      hardLimit: 4096,
    },
    "CPU": {
      softLimit: 2,
      hardLimit: 4,
    },
  },
  ...
}

This feature was introduced because a new version of Go changed the logic for setting RLIMIT_NOFILE as part of child processes, risking build actions to run into file descriptor limits. This change can help avoid this by overriding the resource limits.

ReferenceExpandingBlobAccess CAS reference support (Dec 5, 2023)​

ReferenceExpandingBlobAccess has been extended to support request forwarding to a REv2 CAS. This is useful when designing asset storage services that provide direct exposition over bytestream://.

Additional JWT support​

JWKs can now optionally be read from a file (Oct 30, 2023), rather than than providing it inline in the configuration.

JWTs can now be signed with the Ed25519 algorithm (Jun 19, 2024), and JWTs signed with RSA-PSS signatures can be validated (May 7, 2026).

This is a continuation of the previous update article and is a high level summary of what has happened in Buildbarn from 2023-02-16 to 2023-11-14.

Added support for JWTs signed with RSA​

Support for JWTs signed with RSA has been added. The following JWT signing algorithms are now supported:

• HS256
• HS384
• HS512
• RS256
• RS384
• RS512
• EdDSA
• ES256
• ES384
• ES512

Generalized tuneables for Linux BDI options​

Linux 6.2 added a sysfs attribute for toggling BDI_CAP_STRICTLIMIT on FUSE mounts. If using the FUSE backed virtual file system on Linux 6.2 adding { "strict_limit": "0" } to linux_backing_dev_info_tunables will remove the BDI_CAP_STRICTLIMIT flag from the FUSE mount.

This may improve fileystem performance especially when running build actions which uses mmap'ed files extensively.

Add support for injecting Xcode environment variables​

Remote build with macOS may call into locally installed copies of Xcode. The path to the local copy of Xcode may vary and Bazel assumes that the remote execution service is capable of processing Xcode specific environment variables.

See the proto files for details.

Add a minimum timestamp to ActionResultExpiringBlobAccess​

A misbehaving worker may polluted the action cache, after fixing the misbehaving worker we would rather not throw away the entire action cache.

A minimum timestamp in ActionResultExpiringBlobAccess allows us to mark a timestamp in the past before which the action should be considered invalid.

Add authentication to HTTP servers​

Much like the gRPC servers are capable of authenticated configuration the http servers can now also require authentication.

This allows the bb_browser and bb_scheduler UI to authenticate access using OAuth2 without involving any other middleware.

This also allows us to add authorization configuration for administrative tasks such as draining workers or killing of jobs.

Authentication using a JSON Web Key Set​

JSON Web Key Sets (JWKS) is a standard format which allows us to specify multiple different encryption keys that may have been used to sign our JWT authentication.

Buildbarn can load the JWKS specification, either inline or as a file, when specifying trusted encryption keys.

This allows us to have rotation with overlap of encryption keys.

The example configuration project for buildbarn bb-deployments has gotten updates.

This is a continuation of the updates from last year article and is a high level summary of what has happened since April 2022 up to 2023-02-16.

Let ReferenceExpandingBlobAccess support GCS​

ReferenceExpandingBlobAccess already supports S3 so support was extended to Google Cloud Storage buckets.

Support for prefetching Virtual Filesystems​

Running workers with Fuse allows inputs for an action to be downloaded on demand. This significantly reduces the amount of data that gets sent in order to run overspecified actions. This however leads to poor performance for actions which reads a lot of their inputs synchronously.

With the prefetcher most of these actions can be recognized and data which is likely to be needed can be downloaded ahead of time.

Support for sha256tree​

Buildbarn has added support for sha256tree which uses sha256 hashing over a tree structure similar to blake3.

This algorithm will allow large CAS objects to be chunked and decompositioned with guaranteed data integrity while still using sha256 hardware instructions.

Completeness checking now streams REv2 Tree objects​

This change introduces a small change to the configuration schema. If you previous had this:

backend: { completenessChecking: ... },

You will now need to write something along these lines:

backend: {
    completenessChecking: {
        backend: ...,
        maximumTotalTreeSizeBytes: 64 * 1024 * 1024,
    },
},

See also the bb-storage commit 1b84fa8.

Postponed healthy service status​

The healthy and serving status, i.e. HTTP /-/healthy and grpc_health_v1.HealthCheckResponse_SERVING, are now postponed until the whole service is up and running. Before, the healthy status was potentially reported before starting to listen to the gRPC ports. Kubernetes will now wait until the service is up before forwarding connections to it.

Server keepalive parameter options​

The option buildbarn.configuration.grpc.ServerConfiguration.keepalive_parameters can be used for L4 load balancing, to control when to ask clients to reconnect. For default values, see keepalive.ServerParameters.

Graceful termination of LocalBlobAccess​

When SIGTERM or SIGINT is received, the LocalBlobAccess now synchronize data to disk before shutting down. Deployments using persistent storage will no longer observe loss of data when restarting the bb_storage services.

Non-sector Aligned Writes to Block Device​

Using sector aligned storage is wasteful for the action cache where the messages are typically very small. Buildbarn can now fill all the gaps when writing, making storage more efficient.

DAG Shaped BlobAccess Configuration​

Instead of a tree shaped BlobAccess configuration, the with_labels notation allows a directed acyclic graph. See also the bb-storage commit cc295ad.

NFSv4 as worker filesystem​

The bb_worker can now supply the working directory for bb_runner using NFSv4. Previously, FUSE and hard linking files from the worker cache were the only two options. This addition was mainly done to overcome the poor FUSE support on macOS.

The NFSv4 server in bb_worker only supports macOS at the moment. No effort has been spent to write custom mount logic for other systems yet.

Specify forwardMetadata with a JMESPath​

Metadata forwarding is now more flexible, the JMESPath expressions can for example add authorization result data. The format is described in grpc.proto.

A common use case is to replace

{
    forwardMetadata: ["build.bazel.remote.execution.v2.requestmetadata-bin"],
}

with

{
    addMetadataJmespathExpression: '{
        "build.bazel.remote.execution.v2.requestmetadata-bin":
            incomingGRPCMetadata."build.bazel.remote.execution.v2.requestmetadata-bin"
    }',
}

Tracing: Deprecate the Jaeger collector span exporter​

This option is deprecated, as Jaeger 1.35 and later provide native support for the OpenTelemetry protocol.

bb-deployments Ubuntu 22.04 Example Runner Image​

The rbe_autoconfig in bazel-toolchains has been deprecated. In bb-deployments it has been replaced by the Act image ghcr.io/catthehacker/ubuntu:act-22.04, distributed by catthehacker, used for running GitHub Actions locally under Ubuntu 22.04.

bb-deployments Integration Tests​

The bare deployment and Docker Compose deployment have now got tests scripts that builds and tests @abseil-hello//:hello_test remotely, shuts down and then checks for 100% cache hit after restart. Another CI test is checking for minimal differences between the Docker Compose deployment and the Kubernetes deployment.

If there are any other changes you feel deserve a mention feel free to submit a pull request at github using the link below.

The sample configuration project for Buildbarn was recently updated after a long hiatus. As an aid for people to understand which changes have been done see the following high level summary.

April 2022 Updates​

This includes updates to Buildbarn since December 2020.

Authorizer Overhaul​

Authorizers have been rehauled to be more flexible it is now part of each individual cache and execution configuration.

Using a JWT authorization bearer token has been added as an authorization method.

Hierarchical Blob Access​

Using hierarchical blob access allows blobs in instance name foo/bar to be accessed from instance foo/bar/baz but not instance foo or foo/qux.

Action Result Expiration​

An expiry can be added to action result which lets the action cache purge the result of an exection that was performed too far in the past. This can be used to ensure that all targets are rebuilt periodically even if they are accessed frequently enough to not normally be purged from the cache.

Read Only Cache Replicas​

Cache read traffic can now be sent to a read-only replica which is periodically probed for availability.

Concurrency Limiting Blob Replication​

Limit the number of concurrent replications to prevent network starvation

Run Commands as Another User​

Allows the commands to be run as a different user, on most platforms this means the bb-runner instance must run as root.

Size Class Analysis​

Allows executors of different size classes to be used, the scheduler will attempt to utilize executors efficiently but there is an inherent tradeof between throughput and latency. Once configured the scheduler will automatically attempt to keep track of which actions are best run on which executors.

Execution Routing Policy​

The scheduler accepts an execution routing policy configuration that allows it to determine how to defer builds to different executors.

If you see any other changes you feel should get a mention feel free to submit a pull request at github using the link below.