3 posts tagged with "release"

This is a continuation of the previous update article and is a high level summary of what has happened in Buildbarn from 2023-02-16 to 2023-11-14.

Added support for JWTs signed with RSA​

Support for JWTs signed with RSA has been added. The following JWT signing algorithms are now supported:

• HS256
• HS384
• HS512
• RS256
• RS384
• RS512
• EdDSA
• ES256
• ES384
• ES512

Generalized tuneables for Linux BDI options​

Linux 6.2 added a sysfs attribute for toggling BDI_CAP_STRICTLIMIT on FUSE mounts. If using the FUSE backed virtual file system on Linux 6.2 adding { "strict_limit": "0" } to linux_backing_dev_info_tunables will remove the BDI_CAP_STRICTLIMIT flag from the FUSE mount.

This may improve fileystem performance especially when running build actions which uses mmap'ed files extensively.

Add support for injecting Xcode environment variables​

Remote build with macOS may call into locally installed copies of Xcode. The path to the local copy of Xcode may vary and Bazel assumes that the remote execution service is capable of processing Xcode specific environment variables.

See the proto files for details.

Add a minimum timestamp to ActionResultExpiringBlobAccess​

A misbehaving worker may polluted the action cache, after fixing the misbehaving worker we would rather not throw away the entire action cache.

A minimum timestamp in ActionResultExpiringBlobAccess allows us to mark a timestamp in the past before which the action should be considered invalid.

Add authentication to HTTP servers​

Much like the gRPC servers are capable of authenticated configuration the http servers can now also require authentication.

This allows the bb_browser and bb_scheduler UI to authenticate access using OAuth2 without involving any other middleware.

This also allows us to add authorization configuration for administrative tasks such as draining workers or killing of jobs.

Authentication using a JSON Web Key Set​

JSON Web Key Sets (JWKS) is a standard format which allows us to specify multiple different encryption keys that may have been used to sign our JWT authentication.

Buildbarn can load the JWKS specification, either inline or as a file, when specifying trusted encryption keys.

This allows us to have rotation with overlap of encryption keys.

The example configuration project for buildbarn bb-deployments has gotten updates.

This is a continuation of the updates from last year article and is a high level summary of what has happened since April 2022 up to 2023-02-16.

Let ReferenceExpandingBlobAccess support GCS​

ReferenceExpandingBlobAccess already supports S3 so support was extended to Google Cloud Storage buckets.

Support for prefetching Virtual Filesystems​

Running workers with Fuse allows inputs for an action to be downloaded on demand. This significantly reduces the amount of data that gets sent in order to run overspecified actions. This however leads to poor performance for actions which reads a lot of their inputs synchronously.

With the prefetcher most of these actions can be recognized and data which is likely to be needed can be downloaded ahead of time.

Support for sha256tree​

Buildbarn has added support for sha256tree which uses sha256 hashing over a tree structure similar to blake3.

This algorithm will allow large CAS objects to be chunked and decompositioned with guaranteed data integrity while still using sha256 hardware instructions.

Completeness checking now streams REv2 Tree objects​

This change introduces a small change to the configuration schema. If you previous had this:

backend: { completenessChecking: ... },

You will now need to write something along these lines:

backend: {
    completenessChecking: {
        backend: ...,
        maximumTotalTreeSizeBytes: 64 * 1024 * 1024,
    },
},

See also the bb-storage commit 1b84fa8.

Postponed healthy service status​

The healthy and serving status, i.e. HTTP /-/healthy and grpc_health_v1.HealthCheckResponse_SERVING, are now postponed until the whole service is up and running. Before, the healthy status was potentially reported before starting to listen to the gRPC ports. Kubernetes will now wait until the service is up before forwarding connections to it.

Server keepalive parameter options​

The option buildbarn.configuration.grpc.ServerConfiguration.keepalive_parameters can be used for L4 load balancing, to control when to ask clients to reconnect. For default values, see keepalive.ServerParameters.

Graceful termination of LocalBlobAccess​

When SIGTERM or SIGINT is received, the LocalBlobAccess now synchronize data to disk before shutting down. Deployments using persistent storage will no longer observe loss of data when restarting the bb_storage services.

Non-sector Aligned Writes to Block Device​

Using sector aligned storage is wasteful for the action cache where the messages are typically very small. Buildbarn can now fill all the gaps when writing, making storage more efficient.

DAG Shaped BlobAccess Configuration​

Instead of a tree shaped BlobAccess configuration, the with_labels notation allows a directed acyclic graph. See also the bb-storage commit cc295ad.

NFSv4 as worker filesystem​

The bb_worker can now supply the working directory for bb_runner using NFSv4. Previously, FUSE and hard linking files from the worker cache were the only two options. This addition was mainly done to overcome the poor FUSE support on macOS.

The NFSv4 server in bb_worker only supports macOS at the moment. No effort has been spent to write custom mount logic for other systems yet.

Specify forwardMetadata with a JMESPath​

Metadata forwarding is now more flexible, the JMESPath expressions can for example add authorization result data. The format is described in grpc.proto.

A common use case is to replace

{
    forwardMetadata: ["build.bazel.remote.execution.v2.requestmetadata-bin"],
}

with

{
    addMetadataJmespathExpression: '{
        "build.bazel.remote.execution.v2.requestmetadata-bin":
            incomingGRPCMetadata."build.bazel.remote.execution.v2.requestmetadata-bin"
    }',
}

Tracing: Deprecate the Jaeger collector span exporter​

This option is deprecated, as Jaeger 1.35 and later provide native support for the OpenTelemetry protocol.

bb-deployments Ubuntu 22.04 Example Runner Image​

The rbe_autoconfig in bazel-toolchains has been deprecated. In bb-deployments it has been replaced by the Act image ghcr.io/catthehacker/ubuntu:act-22.04, distributed by catthehacker, used for running GitHub Actions locally under Ubuntu 22.04.

bb-deployments Integration Tests​

The bare deployment and Docker Compose deployment have now got tests scripts that builds and tests @abseil-hello//:hello_test remotely, shuts down and then checks for 100% cache hit after restart. Another CI test is checking for minimal differences between the Docker Compose deployment and the Kubernetes deployment.

If there are any other changes you feel deserve a mention feel free to submit a pull request at github using the link below.

The sample configuration project for Buildbarn was recently updated after a long hiatus. As an aid for people to understand which changes have been done see the following high level summary.

April 2022 Updates​

This includes updates to Buildbarn since December 2020.

Authorizer Overhaul​

Authorizers have been rehauled to be more flexible it is now part of each individual cache and execution configuration.

Using a JWT authorization bearer token has been added as an authorization method.

Hierarchical Blob Access​

Using hierarchical blob access allows blobs in instance name foo/bar to be accessed from instance foo/bar/baz but not instance foo or foo/qux.

Action Result Expiration​

An expiry can be added to action result which lets the action cache purge the result of an exection that was performed too far in the past. This can be used to ensure that all targets are rebuilt periodically even if they are accessed frequently enough to not normally be purged from the cache.

Read Only Cache Replicas​

Cache read traffic can now be sent to a read-only replica which is periodically probed for availability.

Concurrency Limiting Blob Replication​

Limit the number of concurrent replications to prevent network starvation

Run Commands as Another User​

Allows the commands to be run as a different user, on most platforms this means the bb-runner instance must run as root.

Size Class Analysis​

Allows executors of different size classes to be used, the scheduler will attempt to utilize executors efficiently but there is an inherent tradeof between throughput and latency. Once configured the scheduler will automatically attempt to keep track of which actions are best run on which executors.

Execution Routing Policy​

The scheduler accepts an execution routing policy configuration that allows it to determine how to defer builds to different executors.

If you see any other changes you feel should get a mention feel free to submit a pull request at github using the link below.