We've also launched new documentation with this release, at https://docs.aspect.build/bazel/javascript

If you're already on Bazel 7+, loading rules_js via MODULE.bazel, and using pnpm 9+, the upgrade should be straightforward.

What’s Changed?

• 🚫 Remove Bazel 6 support

• 🚫 Remove pnpm 8 support

• 🚫 Remove WORKSPACE support

• 🧹 Remove some rarely used APIs

• ✅ Enable enhancements that were previously opt-in

Ecosystem Compatibility

Related rulesets including rules_ts, rules_swc, rules_jest, and rules_webpack have been tested with rules_js 3.0 and are compatible with their current releases.

The dependency on aspect_bazel_lib 2.x has been replaced with bazel_lib 3.0 (2567). With this change, rules_js ensures any aspect_bazel_lib usage remains forward-compatible with bazel_lib, as long as your repository does not override the aspect_bazel_lib module.

The most significant change is the removal of pnpm 8 and WORKSPACE support.

Early versions of rules_js were designed around:

• The pnpm lockfile format at the time

• Limitations of Bazel repository rules in WORKSPACE mode

To avoid diverging code paths:

• When pnpm updated its lockfile format, rules_js downgraded it internally.

• When Bazel MODULEs (bzlmod) were introduced, rules_js delegated much of the logic back to WORKSPACE-based implementations.

That approach kept behavior consistent, but added complexity and limited adoption of newer pnpm and bzlmod features.

With pnpm 8 and WORKSPACE removed:

• The core architecture is simpler

• Legacy compatibility layers are gone

• A single modern code path replaces multiple branches

• Bugs such as the pnpm lockfile being parsed twice under bzlmod (2480) are eliminated

Notable CHANGELOG

📦 Platform-specific optional npm dependencies (2538)

That annoying @esbuild/android-arm64 you always see being fetched and know you really don't need will no longer be fetched. Other common examples are platform specific @swc/*, @rollup/* or the upcoming @typescript/native-preview-* packages.

🔗 proto_library support in JS deps (2721)

Experimental support for directly depending on proto_library targets in js_library(deps) or ts_project(deps). You register a protoc plugin toolchain for your choice of codegen tooling.

🗂 Package exclusion presets + default exclusion list (2652)

That 10mb CHANGELOG.md you always notice? Gone!

The npm_exclude_package_contents API now has multiple presets instead the previous single use_defaults = True which was opt-in. The previous use_defaults = True has been replaced with the "yarn_autoclean" preset which mimics the yarn autoclean command. A new "basic" preset is now available and enabled by default - this is less aggressive then the yarn list while still excluding common unnecessary files often shipped in npm packages.

⚡ Better Bazel 9 compatibility and performance

Bazel 9 has some new features such as the facts API which will now be used to make things like pnpm downloads reproducible even if you don't manually provide SHAs (2698).

Path-mapping support has also been expanded (2575).

🛣 Initial support for Bazel path mapping (2575)

Bazel path mapping allows some actions to be cached and reused across compilation modes such as dbg and release.

🟢 Default Node version bumped to v22 (2649)

Along with upgrading the minimum rules_nodejs version the default node version is now also updated to the LTS v22.

What's Next

rules_js 3.0 is launched and ready for you to upgrade. Check out the updated docs or try a new project using the https://github.com/bazel-starters/js repo which has all the latest versions.

Having problems? Aspect offers a paid support plan with a dedicated Slack channel for your team under an SLA.

We’ve been incredibly gratified by the more than 60 open-source contributors who have improved the ruleset, and everyone who has filed or answered issues. Thank you! We support 30 languages now and continue to grow.

Highlights of Aspect rules_lint release version 2.0

• The Aspect Extension Language (AXL) is now used to provide the lint and format tasks on your command-line. Just install the Aspect CLI and run aspect lint. This is wired in the MODULE.aspect file that sits next to MODULE.bazel.

• Most of the bazel-starters demonstrate rules_lint 2.0, and give an easy playground to try it out.

• The rules_lint examples folder is now broken out into standalone Bazel modules per-language, making it much easier to find the bits you need for your own repo.

• Pythonistas will love the new ty linter support. Ty is a fast Python type-checker written in Rust from Astral - and we’re working with them to add incremental type-check support to avoid quadratic runtime. Thank you to https://github.com/whoahbot for the contribution!

• We’ve added Rust’s clippy tool so you can get auto-fixes like unused imports applied automatically during code review. We expect rules_rust to decrease scope, possibly removing their Clippy support. Thank you to https://github.com/blorente for adding this support!

• Full Bazel 9 support, including Bzlmod module_extension support for fetching all tools, obviating the need for any WORKSPACE file or http_archive rules.

About rules_lint

aspect-build/rules_lint is a Bazel ruleset that makes linting and formatting “first-class” in Bazel — so you can run common static analysis tools via Bazel without wrapping your existing BUILD targets or changing the rulesets you already use.

As before, rules_lint v2 is really two rulesets in one:

Formatting

• Typically one formatter per language; deterministic output; “just apply the changes.”

• Formatting tools are run as side-effects outside of Bazel actions and wired with a git pre-commit hook.

• That means it can run on files not modeled in Bazel’s dependency graph: formatting runs on the file tree (helpful for scripts, docs, etc.).

Get started at https://github.com/aspect-build/rules_lint/blob/main/docs/formatting.md

Linting

• Can run multiple linters per language; may propose fixes; results can be shown as terminal output, failing tests, or code-review feedback.

• No BUILD-file clutter: you lint existing *_library targets rather than adding special wrapper macros.

• Incremental & cache-friendly: lint runs as Bazel actions (works with remote execution/cache).

• Practical for legacy repos: supports “lint only what changed” workflows so you can start without fixing all historic issues. We refer to the “Water Leak Principle” which says to stop the leak before mopping the spill.

Get started at https://github.com/aspect-build/rules_lint/blob/main/docs/linting.md

Wiring into code review

Aspect’s platform includes Marvin, our mascot and helpful Bazel bot. Marvin comments on your pull requests, and includes lint results displayed as GitHub Checks. Even better, when the linter tool has a —-fix mode, Marvin will provide the Suggested Fixes in the GitHub code review so you can just accept the improvements to your code.

Next steps

To learn more about our Aspect Workflows developer productivity platform and expert Bazel support services, talk to us on Bazel Slack or email us at hello@aspect.build.

• Bzlmod is now always enabled, and all WORKSPACE logic has been removed from Bazel (#26131). The Bzlmod migration tool is available.

• All C++-related rules are removed from Bazel and must be loaded from @rules_cc, as part of the Starlark effort.

• Bazel 9 ships with the latest protobuf module (version 33.4), which includes support for a prebuilt protobuf compiler.

This prebuilt protobuf compiler follows up on our blog article https://blog.aspect.build/never-compile-protoc-again which ended with the promise “We’re hoping to upstream our toolchains_protoc to the protobuf repository, so that the default Bazel experience will be fast.”

This is now (nearly) true as of the https://github.com/protocolbuffers/protobuf/releases/tag/v33.4 release.

We thank the Google Protobuf team for sponsoring this work and reviewing the implementation, including changes to the release process.

Enabling the Protobufs toolchains feature

Bazel 9 flips the --incompatible_enable_proto_toolchain_resolution flag to true. This means Bazel is responsible for resolving the symbol @protobuf//bazel/private:proto_toolchain_type to a “concrete” toolchain that provides the right protoc binary for your execution platform. This is also the case for a toolchain_type for each language stub generator.

If you’re on Bazel 9, there’s nothing to do, but earlier Bazels require you set the flag.

Opting-in

Until https://github.com/protocolbuffers/protobuf/pull/25313 lands and is released, you need to opt-in to using the protoc binaries from https://github.com/protocolbuffers/protobuf/releases. Add to your .bazelrc:

common --@protobuf//bazel/toolchains:prefer_prebuilt_protoc

note, if you have a repo_name=com_google_protobuf you’ll have to adapt the @protobuf name

Enforcing

You can enforce that no rules do the “Wrong Thing” of directly referencing the cc_binary target @protobuf//:protoc by following our snippets: https://github.com/aspect-build/toolchains_protoc#ensure-protobuf-and-grpc-never-built

If this fails to build, it means that there’s a bug you should find or report.

The https://github.com/aspect-build/toolchains_protoc and https://github.com/bazelbuild/rules_proto repositories are now archived, completing the deprecation period.

Next

Need help migrating from WORKSPACE to Bzlmod or other steps to move to Bazel 9? View Aspect Build services and email us at hello@aspect.build or ping us on Bazel Slack. We’re happy to support you.

This is exactly where Bazel, the open-source, high-performance build and test system originally created at Google, can play a transformative role. Bazel offers SONiC (Software for Open Networking in the Cloud) contributors, adopters and vendors the reliability, scalability, and repeatability needed to sustain a world-class network OS at global scale.

Below, we explore the case for adopting Bazel across SONiC Foundation projects and how it can meaningfully improve developer productivity, platform compatibility, security posture, and release engineering.

1. Reproducible Builds Are No Longer Optional

SONiC today supports an expanding matrix of devices, ASICs, and software packages. That's its strength—but also its build challenge. Makefiles, ad-hoc scripts, and hand-rolled toolchains multiply variability and increase onboarding time, especially for new vendors.

Bazel provides hermetic, reproducible builds, ensuring that:

• The same inputs always produce the same outputs

• Dependencies are fetched deterministically and cached

• Toolchains and container images are versioned and immutable

• Builds can run anywhere—from a developer's laptop to CI runners to cloud build farms

This consistency dramatically reduces "it works on my machine" issues. For SONiC's multi-vendor ecosystem, it means any contributor can build and test with confidence against a shared standard.

2. A Universal Build System for a Polyglot Codebase

SONiC involves C++, Python, Go, Rust, Docker, kernel modules, switch-vendor SDKs, and more. Bazel excels here:

• First-class multi-language support

• Extensibility through Starlark rules

• Deterministic container and image builds

• Cross-compilation support for ARM, x86, PowerPC, and ASIC-specific toolchains

Instead of maintaining fragmented build logic across repositories, SONiC developers can standardize on a single system that handles everything from low-level DPDK components to high-level services.

3. Cloud-Native CI/CD That Scales With the SONiC Community

Bazel was designed for massive scale and parallelism. For SONiC maintainers, this directly translates into:

• Faster builds through fine-grained caching

• Incremental rebuilds that recompile only what changed

• Remote execution to distribute builds across clusters

• Remote caching to avoid duplicated work between developers and CI jobs

• Unified pipelines across all repos and languages

As the community continues to expand, this infrastructure lets new contributors ramp up quickly and ensures that CI is fast, stable, and cost-efficient.

4. Stronger Security and Compliance

Networking software is increasingly subject to regulatory scrutiny. SONiC vendors must prove the provenance, integrity, and patch level of every component.

Bazel strengthens security by:

• Locking down external dependencies

• Ensuring reproducibility (critical for supply-chain audits)

• Providing deterministic SBOM generation

• Enforcing hermetic builds that eliminate environment drift

• Integrating easily with tools for SLSA, sigstore, and in-toto

For organizations integrating SONiC into large-scale infrastructure, this reduces risk and simplifies compliance workflows.

5. Better Collaboration Between Vendors and the Community

One of SONiC's greatest strengths is its vendor-neutral ecosystem. Bazel reinforces this mission:

• All vendors compile with the same rules and toolchains

• Contributors can share Starlark extensions for SDKs or hardware targets

• Build logic becomes collaborative, reviewable, and testable—just like code

• Onboarding new vendors becomes faster and less error-prone

Instead of each party maintaining private build scripts, SONiC can establish a shared, open-standard build vocabulary.

6. Future-Proofing SONiC for the Next Decade

SONiC is evolving quickly—into new form factors, new silicon, new service models, and new deployment patterns. With Bazel, SONiC gains a foundation that:

• Scales horizontally as code volume increases

• Easily supports new languages or frameworks

• Provides deterministic releases that downstream integrators can trust

• Enables advanced workflows like distributed testing or reproducible builds in the cloud

Bazel is not a short-term patch; it is an investment in SONiC's long-term velocity and stability.

The Legacy Build System: Complexity at Scale

To understand why Bazel is necessary, it's worth examining what SONiC's build process looked like before: a Makefile-based system built around a "slave" container image that encapsulated system dependencies.

Arbitrary Commands, Arbitrary Problems

The legacy system consists of 317 Makefile rules (.mk files) and dependency files (.dep files), one for nearly every package, container, and component in SONiC. Each rule is allowed to invoke arbitrary commands: apt-get install, pip install, dget, shell scripts, custom build logic. On the surface, this sounds reasonable—the "slave" container provides isolation. But at SONiC's scale, this approach becomes a liability:

Brittleness: Each recipe fetches dependencies from the internet on-demand. A network hiccup, a removed package from Debian archives, a deprecated Python module on PyPI, and the entire build fails. There's no guarantee that tomorrow's build will work the same way as today's. No pinning, no snapshot repositories, no explicit version control of dependencies.

Non-Hermeticity: Because recipes are allowed to fetch and execute arbitrary commands, the build's output depends on what's available on the internet right now. Build two identical SONiC commits a week apart, and they may produce different binaries. The build machine's OS and installed packages are implicit dependencies—if you upgrade Ubuntu on your build server, you risk silently changing SONiC's outputs.

Unmaintainability at Scale: With 317 separate rule files, coordinating changes is nightmarish. A Debian package update might require changes to multiple recipes. Dependency injection during build (where rules can declare new dependencies at build time) means the full dependency graph is opaque until runtime. You don't know what's actually going to be built until you run make.

Hidden Complexity: The "slave" container approach obscures the real problem. Developers think they're building in a consistent environment because it's Docker-based, but the container itself is built by the same brittle, non-hermetic process. You're wrapping chaos in a box and calling it reproducibility.

Recipes Inject Dependencies During Build

Making matters worse, the legacy system allows rules to inject dependencies dynamically during the build process. A recipe isn't just a fixed input-output mapping; it's a script that can decide at runtime what to depend on. This makes the dependency graph fundamentally unknowable until build execution. You can't reason about what the artifact will contain—you have to run it and see.

The Scale Problem

SONiC spans dozens of C/C++ components, multiple Python packages, Go binaries, container images, kernel modules, and ASIC-specific toolchains. The legacy system required maintainers to write and maintain Makefiles for all of this, with no unifying framework. Each component has its own recipe, its own build logic, its own fragile internet-based dependency fetching. When SONiC grew to support ARM architectures, multiple Debian versions (Stretch, Buster, Bullseye, Bookworm), and dozens of ASIC vendors, the number of recipes multiplied.

The build system became a maintenance burden that grew faster than the codebase itself.

Bazel is decades ahead as a build system—but it’s not autopilot. You still need a good engineer to make it fit the problem.

Bazel is uniquely suited to address the intricate build and test challenges faced by the SONiC project. As SONiC evolves to support a growing range of devices, languages, and architectures, the need for a robust build system becomes paramount. Bazel’s hermetic, reproducible builds help ensure that the same inputs consistently yield the same outputs—an essential property in a diverse ecosystem. With deterministic dependency management and strong cross-compilation support, Bazel aligns well with SONiC’s goals of consistency across platforms and contributors.

But Bazel isn’t a silver bullet. It’s decades ahead as a build system, yet its architecture doesn’t solve the problem automagically—real success still depends on good engineering: clear rule boundaries, disciplined dependencies, and a toolchain/packaging strategy that makes Bazel’s guarantees real in practice.

The Deep Technical Challenge: Hermetic Package Dependency Resolution

Building a polyglot, multi-architecture system like SONiC using Bazel exposed a fundamental tension between how traditional Linux package managers work and how Bazel is designed to operate. The solution has required solving multiple interconnected problems that most build systems never encounter.

The sections below summarize some of the Bazel for SONiC issues that Aspect Build and has helped to identify and address.

The .deb Package Overlay Problem

Debian packages are designed to be installed sequentially into a shared filesystem. When you run apt-get install, each package unpacks its files into the same root directory—creating a virtual "overlay" of files from potentially hundreds of packages. Circular dependencies are resolved through this overlay: library A might contain a symlink to library B, which is provided by a completely different package. The order of installation handles conflicts, and the running system sees a unified merged filesystem.

Bazel, by contrast, abhors large, opaque target outputs. Each target should be hermetic and reproducible, with explicit dependencies and minimal side effects. Representing "unpack all these packages and overlay them" as a single Bazel target creates an enormous, unwieldy build artifact that defeats caching, incremental builds, and reproducibility. Yet SONiC needs precisely this—a complete, consistent sysroot with hundreds of interdependent Debian packages.

The solution: Bazel needed to compute the transitive closure of package dependencies, resolve symlinks ahead of time, and produce a metadata representation of the merged filesystem without materializing the entire overlay**.** Rather than creating a giant unpacked sysroot target, rules_distroless generates a "Contents" file—a snapshot mapping every filename to the package that provides it. This allows Bazel to:

• Resolve symlink chains before build time (avoiding "dangling symlink" errors that plague container builds)

• Determine the final location of each library or header file despite circular package dependencies

• Share the metadata across builds without replicating hundreds of MB of unpacked packages

• Enable fine-grained caching at the package level, not the sysroot level

The RPATH Nightmare

The ELF dynamic linker, ld.so, searches for shared libraries in a precise order:

1. Directories in the binary's DT_RPATH attribute (if DT_RUNPATH is absent)

2. LD_LIBRARY_PATH environment variable

3. Directories in DT_RUNPATH (the modern preferred approach)

4. System cache (/etc/ld.so.cache)

5. Default paths (/lib, /usr/lib, and architecture-specific variants)

By default, binaries built in SONiC expect their dependencies in /usr/lib, /lib, and architecture-specific variants like /usr/lib/aarch64-linux-gnu. But Bazel builds place outputs in bazel-out/, a completely different path structure. The linker has no idea where to find libyang.so.2.0 when it's buried in bazel-out/aarch64-linux-gnu/bin/external/com_github_sonic_net_sonic_mgmt_common/lib/libyang.so.2.0.

The fix requires embedding RPATH entries directly into the binary at link time—telling the linker "look in $ORIGIN/../lib for my dependencies." But this introduces new challenges:

• Relocation: A binary linked with RPATH=$ORIGIN/../lib expects a specific directory structure. If that binary is later used as a tool in another Bazel action (where it's relocated to a different path), the RPATH becomes invalid.

• Transitive Dependencies: When a binary depends on library A, which depends on library B, the linker must find all three—but each may have different RPATH settings. Bazel must ensure the transitive closure is visible to the linker without creating massive, monolithic targets.

SONiC solved this through Bazel Configurations and Transitions —a mechanism that applies different compiler flags to different parts of the dependency graph. Some targets (like Python's C extensions) are compiled with -fPIC and packaged into a sysroot. Everything else uses the standard compilation model. Bazel's transitions ensure these different compilation modes never mix, preventing linker errors where position-dependent code tries to reference position-independent code.

The RPATH Nightmare Continues: Finding Dependencies in the Sandbox

When Bazel runs a test target, it constructs a temporary sandbox directory containing only the files that test explicitly depends on. A test might link against a dynamic C++ library, which depends on libc.so.6, which transitively depends on ld-linux-x86-64.so.2. At test runtime, none of these libraries are in the system /usr/lib—they're scattered across bazel-out/ in the test's sandbox.

The breakthrough: Rather than relying on LD_LIBRARY_PATH (which is fragile and defeats reproducibility), rules_distroless ensures that when a test binary is linked, its RPATH contains the exact paths to all transitive dependencies within that test's sandbox. The binary becomes self-contained—it knows exactly where to find every library it needs, relative to its own location. A test can now be run anywhere, on any machine, and it will find its dependencies without environment variable manipulation. This is genuinely hermetic testing: the test's success or failure depends only on the code and its declared dependencies, not on what happens to be installed on the developer's machine.

Container Runtime: Binaries Finding Dependencies Across Layers

When a SONiC container starts, thousands of binaries are present—p4rt, swss, gnmi, monitoring agents, and hundreds of utilities. None of them are statically linked. Each binary must find its shared libraries at runtime.

In a traditional Docker/container image built with a Dockerfile, everything is installed into standard locations (/lib, /usr/lib). The dynamic linker searches these paths by default. Simple and naive.

But SONiC containers built with Bazel take a different approach. Multiple packages provide the same functionality (e.g., multiple versions of libprotoc.so), but only one should be "selected" for the final image. Furthermore, if SONiC switches from Debian bookworm to Debian trixie, or moves between Ubuntu LTS versions, the exact libraries available change. Hardcoding /usr/lib/x86_64-linux-gnu/libfoo.so.1 in a binary's RPATH would break when the library moves or changes versions.

The solution: Bazel generates a container image where the RPATH of every binary is already computed to find the exact libraries that will be present in that specific image. When p4rt starts in the container, its RPATH contains the paths where rules_distroless placed libprotoc.so, libyang.so, and everything else. The binary finds its dependencies through the RPATH, not through the system's default search paths. This means:

• The container is self-describing: a binary's dependencies are "baked in" rather than discovered at runtime

• Moving from Debian bookworm to trixie doesn't break existing binaries—Bazel recomputes the RPATH for the new distro

• Libraries can be placed anywhere in the image without breaking binaries

• Container images are reproducible: given the same Bazel build configuration, the same binaries will find the same libraries every time

Liberation from Host Distro Dependencies

Perhaps the most profound impact: rules_distroless decouples the build machine's operating system from the built artifact.

Traditionally, if you build SONiC on Ubuntu 24.04 and then try to run the binaries on Ubuntu 22.04, you hit subtle glibc incompatibilities. Some binaries depend on functions only available in Ubuntu 24's glibc version. Others link against the "wrong" version of OpenSSL. Developers spend weeks debugging "works on my machine" failures.

With rules_distroless, the build process explicitly pins every single Debian package using Debian snapshot repositories. When you build SONiC on an Ubuntu 22.04 machine, the build ignores the system's installed packages and fetches exact, pinned versions from the snapshots. The build output contains binaries linked against those exact packages, regardless of what's on the build machine.

This means:

• A developer on Ubuntu 20.04 can build SONiC targeting Debian bookworm without installing bookworm libraries locally

• CI/CD runners can be any modern Linux distro—the build is isolated from the host

• Build results are reproducible across machines because dependency versions are explicit, not implicit

• Upgrading the build machine's OS doesn't accidentally change SONiC's binaries

The build machine becomes nearly irrelevant. You're no longer asking "how do I build this on my machine?" You're asking "given a Bazel configuration and a snapshot of the Debian archive, what do I build?" The answer is the same everywhere, because Bazel controls everything.

Radical Hermeticity: Explicit Dependencies, Zero Implicit Assumptions

The commitment to hermeticity goes far beyond RPATH tuning and package pinning. SONiC's C/C++ compilation is built with compiler flags that reject the very concept of "system libraries":

-nostdinc -nostdinc++ -nostdlib

These flags tell the compiler: "There are no standard libraries. There are no default include paths. Everything must be explicitly declared." Without these flags, a compiler would silently fall back to the build machine's /usr/include and /usr/lib. A developer on Ubuntu 24.04 might accidentally use headers from Ubuntu 24's glibc, even if the target is Debian bookworm. With -nostdinc -nostdinc++ -nostdlib, that accident becomes impossible—the build fails loudly if a dependency is missing.

Every header file, every standard library function, every bit of C runtime must be explicitly provided as a Bazel dependency. This guarantees that:

• A build on any machine produces identical binaries

• Adding a dependency automatically updates the build configuration (nothing hidden)

• Removing unused dependencies is impossible—they're explicitly declared

• Switching between libc implementations or versions is straightforward—just change the declared dependency

Auto-Generated cc_library Targets: Turning .deb Packages into Bazel Dependencies

But declaring every libc header and every system library explicitly would require thousands of manual cc_library rules. That's where rules_distroless shines.

When rules_distroless processes a Debian package, it doesn't just extract files. It automatically generates Bazel cc_library targets that encapsulate:

• All header files from the package (C standard library headers, C++ standard library headers, architecture-specific headers)

• All shared object libraries

• The correct include paths and linker settings

• All transitive dependencies, automatically resolved

From a SONiC developer's perspective, instead of hoping that libc headers and libraries exist somewhere on the system, they explicitly declare:

cc_binary(
    name = "my_tool",
    srcs = ["main.cc"],
    deps = [
        "@debian//libc6",
        "@debian//libstdc++",
    ],
)

The Bazel build system now understands exactly which Debian packages this binary depends on. If you want to upgrade libc, you change one line in MODULE.bazel. If you want to use a different libc or switch distributions entirely, the build system tracks it explicitly.

This auto-generation is the key to polyrepo hermetic builds. In a monorepo, one team can write all the libc rules. But SONiC spans multiple GitHub repositories. Each repo's BUILD files can independently declare which Debian packages it needs. The rules_distroless machinery automatically generates compatible cc_library targets, and Bazel's module system ensures they all use the same versions across all repositories.

No more silent fallback to system libraries. No more "it works on my machine but not in CI." Every dependency is explicit, versionable, and traceable.

The Performance Paradox

Computing transitive closure of dependencies across a polyrepo, resolving symlink chains, and managing RPATH dynamically sounds expensive — and it is. But doing it at build time, once, and caching the result is far cheaper than doing it at container runtime or — worse — debugging "symbol not found" errors weeks later in production.

The trick is recognizing what can and cannot be cached:

• Package metadata (which files each package provides, symlink targets) is stable and highly cacheable

• Sysroot layout (which package provides which file when all dependencies are considered) is computed once and reused

• Binary relocations (embedding RPATH in binaries) happens at link time and is reproducible

• Container assembly (final layer selection) happens downstream and doesn't affect upstream caching

SONiC's Bazel infrastructure treats these as separate concerns, allowing massive parallelism and cache reuse even though the final build is complex.

Solving the Python Dependency Maze: Explicit, Versioned, Cross-Compilable

Python presents a unique challenge in reproducible builds. Unlike C/C++ where dependencies are system packages installed via a package manager, Python's ecosystem fetches from PyPI—a centralized repository where package availability and versioning can change. The standard approach to managing Python dependencies in Bazel has been rules_python, but it has limitations that made it unsuitable for SONiC's polyglot, multi-architecture environment.

The PyPI Problem at Scale

SONiC uses Python extensively: configuration generation tools, management daemons, testing frameworks, and utilities. Each Python package has transitive dependencies on other packages, many of which are compiled extensions (.so files). The challenge is that rules_python's traditional pip_parse implementation:

• Assumes a single target architecture (no cross-compilation support)

• Doesn't pin package versions deterministically

• Struggles with compiled Python extensions that depend on system libraries

• Doesn't integrate well with polyrepo ecosystems where multiple repositories have different Python dependency requirements

In SONiC's case, you might need to build the same Python package for x86-64 and ARM64 simultaneously. The standard tooling wasn't built for that.

Enter Aspect Rules Python: Modern Dependency Management

To solve these problems, Aspect Rules Python (aspect_rules_py) provides a modern reimplementation of Python dependency management in Bazel. It replaces the traditional pip_parse with a new implementation based on uv, a fast, production-grade Python package resolver.

Key improvements:

Explicit Versioning: Dependencies are pinned in a lock file (similar to requirements.lock), ensuring that every build uses the exact same package versions. No surprises from PyPI changes.

Cross-Compilation Support: Unlike the original rules_python, aspect_rules_py can build Python packages for different architectures simultaneously. This is critical for SONiC: the same Python source code can be compiled as a wheel for x86-64, then recompiled for ARM64, with all dependencies resolved correctly for each target.

Integration with Hermetic Sysroots: When a Python package has compiled extensions that depend on system libraries (e.g., a C extension that links against libyang), the sysroot provided by rules_distroless makes the correct headers and libraries available. aspect_rules_py integrates seamlessly with this sysroot, so the package's build automatically uses the right C compiler flags, header locations, and link paths for the target architecture.

Polyrepo-Friendly: Each SONiC repository declares and resolves its own Python dependency closure independently. aspect_rules_py uses uv to compute the complete transitive dependency graph for each repository, generating a lock file that pins all transitive dependencies. Because each repo has its own lock file, different repositories can use different versions of shared dependencies without conflict—Bazel's module system keeps them isolated.

Enabling Container Cross-Compilation

Combined with the Debian package management improvements (rules_distroless), aspect_rules_py enables a powerful capability: building complete container images for different architectures within a single Bazel build.

Before, cross-compiling a SONiC container for ARM64 required:

1. Running the build on an ARM64 machine (or emulating it, which was slow)

2. Managing different Python dependency versions for different architectures

3. Handling the mismatch between the build machine's Python environment and the target architecture

With aspect_rules_py and rules_distroless, Bazel can:

1. Resolve Python dependencies for the target architecture (e.g., ARM64)

2. Build wheels for that architecture using the sysroot's C compiler and libraries

3. Layer those wheels into the container image alongside the pinned Debian packages

4. All within a single Bazel invocation on any machine

This means a developer on an x86-64 laptop can type bazel build //path/to:docker-swss-arm64 and get a fully cross-compiled container image without any special setup, emulation, or native ARM64 hardware.

A Call to Action for the SONiC Foundation

The SONiC community is at an inflection point. As deployments reach hyperscale and the contributor base grows more diverse, the project needs a build and test system that matches its ambitions.

Bazel offers exactly that: a modern, reproducible, cloud-scale platform that empowers contributors, accelerates releases, and strengthens the entire ecosystem.

Adopting Bazel across SONiC Foundation projects will:

• Reduce fragmentation

• Increase developer productivity

• Improve security and auditing

• Enable faster and more reliable releases

• Provide a consistent experience for every vendor and contributor

The benefits compound over time—and they're aligned with SONiC's vision of an open, interoperable, high-performance network OS. Let's give SONiC the build system it deserves.

Accelerating Innovation Through Reproducible, Scalable, Cloud-Native Build Systems

In this article, we have explored the case for adopting Bazel across SONiC Foundation projects and how it can meaningfully improve developer productivity, platform compatibility, security posture, and release engineering. We’ve also updated the Bazel and SONiC communities on some of our recent contributions to empower Bazel for SONiC.

As the SONiC Foundation continues to drive forward an open, standards-based network operating system for the industry, one challenge has become increasingly visible across the community: how to build, test, and release SONiC components with speed, consistency, and confidence. The project has grown in complexity—diverse hardware platforms, multiple languages and toolchains, distributed teams, and the rising expectation that networking software behave like modern cloud software.

This is exactly where Bazel, the open-source, high-performance build and test system originally created at Google, can play a transformative role. Bazel offers SONiC contributors and vendors the reliability, scalability, and repeatability needed to sustain a world-class network OS at global scale.

Next Steps

Interested in learning more about how to succeed with Bazel for SONiC? Schedule time to talk with us or email us at hello@aspect.build.

Here’s a retrospective of the conference and our highlights including from my Monday morning keynote.

Travel Was a Pain

U.S. flight cancellations and delays brought many attendees to Atlanta in the wee hours of the morning. Either that, or it was a convenient excuse for feeling hungover during the morning sessions.

All told, BazelCon 2025 attracted over 330 in-person attendees in Atlanta. Thankfully everyone arrived safely.

Aspect Extension Language

Bazel is great at two things: Loading & Analysis of the Dependency/Action graphs, and using plugins (“rulesets”) to populate the bazel-out folder from your sources.

A broad class of extensibility has been missing, and as a result most teams have written their own local dev scripts wrapping bazel and also some CI/CD YAML for their pipelines.

Our solution: a new Starlark dialect we call “Aspect Extension Language (AXL)”. Similarly to how .bzl files have some Bazel-specific standard libraries available as global symbols in starlark code, .axl files give you extension points to hook your developer workflows!

Check out my BazelCon 2025 10 minute lightning talk about AXL and our Marvin Saves the BUILD comic (as a pdf file).

We host a collection of extensions at https://github.com/aspect-extensions and you can easily write your own. Start at https://aspect.build/axl.

Notable BazelCon 2025 Content

Check out the conference session recordings. Our Bazel 102 course on Python was recorded, and has lots of updated content.

We recorded Aspect Insights video podcasts with Mícheál, Yun, and Xudong from Google. These episodes will drop soon!

We hosted a Hackathon the day after the conference. It felt like a hit! Notably Juan from Verkada presented an AXL extension he wrote, which will post soon.

We want to continue getting stuff done around Bazel, so we are going to host monthly meetups in the San Francisco Bay Area starting next month at Figma. See https://luma.com/build-meetup-sf for events in San Francisco and https://luma.com/98o72bge for events in Palo Alto and Silly Valley, aka Silicon Valley.

Next year at BazelCon 2026 we’ll host another Hackathon event like this too.

Malte gave a talk on rules_img. We’ve supported his work and intend to make an easy transition for users of rules_oci.

The BUILD Foundation is forming. Uber, Spotify, and Canva already committed to be founding members. The first meeting will be December 4, email foundation@bazel.build if your company would consider participating.

Workflows Free Open-Source Tier

We’ve opened up a free tier of Aspect Workflows, our CI/CD infrastructure platform. Projects like Buildbarn are now getting our warm CI runners, Buildbarn cache, Web Results UI, and observability. Learn more in our blog.

Orion and gazelle-prebuilt

Jason Bedard has worked with our awesome partners at Adobe to extract the Starlark BUILD file generator to a standalone Gazelle extension called Orion. You can now include this in your own custom Gazelle binary, along with custom extensions you wrote in Go.

We also extracted the pre-compiled Gazelle out of Aspect CLI to a standalone repo: aspect-gazelle. This way your engineers can skip the slow Go source builds and we can reliably depend on C extensions like tree-sitter.

Community Contributions

As a leader in the Bazel ecosystem and Aspect’s Developer Evangelist, I’ve been active behind the scenes.

• Suggested that Cloudflare host a mirror of the BCR, since our customers tripped on unreliable hosting from ftp.gnu.org and gitlab.arm.com. They did!

• Bazel docs are the #1 user-reported problem. For initial progress, the Bazel Central Registry (BCR) has added features to improve transparency:

  • Starlark API docs are now visible in the registry for modules that publish their documentation.

  • Deprecation flags now clearly mark deprecated or archived modules in the registry.

• Next step for documentation: move hosting to a place the community can edit more easily, for example by having a preview of docs PRs (tentatively bazel.online). I encouraged Alan Mond to pick up the torch of improving Bazel’s docsite after he wrote an excellent blog post and coordinated with Bazel team to launch it. I arranged for the Rules Authors SIG to pay a third-party doc hosting service, https://mintlify.com. You can see https://preview.bazel.build for what’s coming!

• We donated our bazel-lib library to Linux Foundation. The 3.0 release completes the rename of the module, removing Aspect branding from the name.

• I had a LOT of meetings to sell the BUILD Foundation mission to large enterprises and the community, connecting their resources to the OSS projects that need it.

• In addition, Jason contributed to upstream Gazelle.

Free Marvin Plush

Including free U.S. shipping. While supplies last. Get yours here.

See you at BazelCon 2026!

We participate in the BazelCon 2026 planning meetings. Little birds chirp that it may be in Europe. My colleague Brett Sheppard suggested Bazel, Belgium. He lived nearby in Flanders as a student. No word yet on the final location for next year’s get together.

Happy coding from the Aspect Build team at BazelCon 2025.

To learn more about our Bazel support and platform, visit www.aspect.build or pick a time to talk with us.

Linting is the process of using a static code analysis tool, known as a "linter," to identify and flag potential programming errors, bugs, stylistic issues, and suspicious constructs in source code. It essentially examines code without executing it.

Of course every language needs linting and formatting. Starlark has one too! It was originally created because the Go team at Google wanted to machine-edit BUILD files, but didn’t want to get into code reviews with teams who liked their hand-formatting of files. Read https://laurent.le-brun.eu/blog/the-story-of-reformatting-100k-files-at-google-in-2011 for more on this back-story.

Buildifier is a tool for formatting Bazel BUILD and .bzl files with a standard convention. Buildifier works on the Aspect Extension Language too! Here’s how that looks in VSCode.

Setting it up

There are several ways to install Buildifier for developers. After working with over 50 companies on their Bazel config, we encoded our learnings into our Starter repos: https://github.com/bazel-starters. To save you a bunch of browsing, here’s a summary of what Aspect recommends:

1. Buildifier is written in Go, but most engineers don’t want to wait to compile it when it’s a cache miss. This is why the “official” instructions look HORRIBLE. We like https://github.com/keith/buildifier-prebuilt as an easy way to get pre-built binaries, along with a build rule to run it. Note that you could fetch binaries directly from the project releases as well.

2. Developers will want to run buildifier from their PATH. We recommend https://direnv.net to hook the shell to update PATH as you cd into the workspace folder, then https://github.com/buildbuddy-io/bazel_env.bzl to add tools to the PATH. Here’s the spot in the example that sets up Buildifier.

3. bazel_env.bzl also provides a stable path for the tool that editors can reference. For example in VSCode, add this to your .vscode/settings.json:
   "bazel.buildifierExecutable": "./bazel-out/bazel_env-opt/bin/tools/bazel_env/bin/buildifier",
   you probably also want to enable it on save:
   "bazel.buildifierFixOnFormat": true,

Buildifier formatting

We want to make developers productive! What’s not productive? Discussion of whitespace, or waiting to re-run your CI job because of a formatter nit. We can make these basically disappear.

First, setup Aspect rules_lint, following https://github.com/aspect-build/rules_lint/blob/main/docs/formatting.md. You don’t need this for buildifier itself, but assuming you also want to run other formatters for other languages, it gives you a single setup that formats all files in the repo.

We setup the editor earlier to run buildifier on save, but engineers have a lot of editor choices. As a fallback, add a pre-commit hook so any unformatted files get fixed at git commit time. A couple options for this are documented in formatting.md.

Finally we do want to enforce that files in the repo are formatted. This isn’t because we hate reading code with different whitespace - it’s just to avoid the next engineer who touches the file ending up with spurious deltas in their pull request. You can run the format.check target from rules_lint on your CI. Give developers a nice error message when it fails, guiding them to setup their dev environment so they don’t need to hit a red CI job in the future.

When you first format the repo, it’s good practice to list your commit hash in the .git-blame-ignore-revs - see https://git-scm.com/docs/git-blame#Documentation/git-blame.txt---ignore-revs-filefile. This way you don’t pollute the blame layer.

Buildifier linting

Buildifier has about 100 checks that catch certain coding issues in Starlark files, though many of them are specific to Bazel’s standard library. The list is here: https://github.com/bazelbuild/buildtools/blob/main/WARNINGS.md

Using rules_lint only runs linters over the dependency graph, and you probably didn’t want to have to add your BUILD files to a filegroup in the BUILD file. Too self-referential! So we recommend running buildifier linting as a standalone step, such as this GitHub Actions workflow. Note that this is not incremental - it runs the linter across all the code in the repository, every time it’s run.

When you first setup the linter, it will point out a big pile of issues, and this may result in a massive PR that is hard to rebase and disruptive to engineers when it lands. We recommend enabling a single check at a time, and slowly rinse-and-repeat following the “Ratchet principle”.

It’s easier on our platform!

The Aspect Workflows developer productivity platform includes buildifier as a first-class task type. This lets you skip some of the setup steps above, and get our recommendations running automatically on your continuous integration (CI) system. Check it out at https://aspect.build/platform.

Or talk with us to learn more.

Some tribes are dismissive of old technology. However, a lot of old projects are not abandoned: they are stable, and in many cases are the bedrock on which a whole stack of tools have been developed. So in this article I’ll take you back to the turn of the decade - and not the most recent one. I suggest you listen to 🎶 “Poison” by Bell Biv DeVoe while you read this article — here’s the music video https://www.youtube.com/watch?v=hgnhVcyLy1I. That’s because Poison came out in March 1990, a couple months before 4.3BSD-Reno, which introduced a tool called mtree(5).

From the archive:

https://man.archlinux.org/man/mtree.5.en

The mtree format is a textual format that describes a collection of filesystem objects. Such files are typically used to create or verify directory hierarchies.

Well, that sounds useful to use in a build system like Bazel. We love textual formats for the ease of manipulation, and Bazel is obsessed with small files produced in one place being inputs to various tools.

The mtree format

Here’s a simple mtree file:

usr/bin uid=0 gid=0 mode=0755 time=1672560000 type=dir
usr/bin/bazelisk uid=0 gid=0 mode=0755 time=1672560000 type=file contents=/path/to/download
usr/bin/bazel uid=0 gid=0 mode=0755 time=1672560000 type=link link=bazelisk

The format includes everything we need for reproducible builds, especially that time attribute. We can describe any arbitrary filesystem structure along with permissions, symlinks, and a bunch more.

Now, following the Unix philosophy, Bazel enables you to compose tools or random bash one-liners with ease. So let’s say we need /usr/bin/bazel to be owned by the right user since we’ll stick this filesystem into a Docker container image. It can be as simple as sed:

genrule(
  name = "change_owner",
  cmd = "sed 's/uid=0/uid=1000/;s/gid=0/gid=500/' <$< >$@",
  ...
)

Where mtree’s are used

Since the mtree format is 35 years old, it can run for President of the United States! And with age comes wisdom — it also has a lot of useful utilities, for example https://github.com/vbatts/go-mtree is a Go utility to interact with manifests. The most useful one, however is tar - at least BSD tar (the GNU tar is not very reproducible and doesn’t have an intermediate representation of the filesystem being archived)

We wrote a Bazel rule, https://registry.bazel.build/modules/tar.bzl, which is a simple starlark wrapper around a hermetic BSD tar toolchain and the mtree format. This rule doesn’t have to be smart, because it just takes an mtree to describe how the inputs should be laid out into the archive.

I gave that sed example earlier - it turns out another Very Old tool is a great way to make more general edits to mtrees - awk. We built this into tar.bzl to provide various mutations of the filesystem layout:

https://github.com/bazel-contrib/tar.bzl/blob/main/tar/private/modify_mtree.awk

And wrapping that with a minimal mutate API lets us replace most of rules_pkg in a simple way:

https://github.com/bazel-contrib/tar.bzl/blob/main/examples/migrate-rules_pkg/BUILD

The philosophy

This post isn’t just about mtree. It turns out many problems were solved long ago in compilers, optimization, packaging, verification, and so on. It’s easy to be tribal and only look for solutions in the ecosystem you’re familiar with, like GitHub repos with a bunch of social followers and recent commits. Often the best tool for the job is on a website that looks like the Wayback Machine, and will serve you a lot better than The Latest Rewrite in Rust.

Shoutout to http://github.com/thesayyn for finding all the clever ideas I’ve described in this post!

What’s next

Meet the Aspect Build team at BazelCon 2025.

I didn’t make any public announcement at the time. This past year I’ve been continuing to evolve the design, and I’m excited to have finally launched it!

For example here is https://registry.bazel.build/modules/tar.bzl:

Documenting Bazel APIs

Some documentation for the Bazel “core” appears on https://bazel.build. However Bazel is extended by rulesets which are published as modules. How do developers find the API documentation for these?

The answer has been quite fragmented, with the community taking several approaches:

1. Markdown or ReStructured Text files checked into the repo like https://github.com/bazel-contrib/rules_go/tree/master/docs/go/core. These rely on some testing that the checked-in files match the stardoc output. So a stardoc_with_diff_test is commonly included, like this one for the Go example. This is not great for contributors who submit a pull request (PR) with a minor correction to some Starlark code — they are met with a red PR and have to update the codegen. Worse, that process uses a Java program in the stardoc module which has to be built from source. This has a bunch of dependencies like protoc which also builds from source. It can take 10min to update the codegen following a 10sec minor correction. This leads to contributors abandoning the fix.

2. Using GitHub Pages like https://bazelbuild.github.io/rules_rust/ or adding a versioning scheme like https://bazelbuild.github.io/rules_pkg/

3. Using Sphinx and a custom Bazel publishing pipeline like https://rules-python.readthedocs.io/en/latest/

4. Aspect had a legacy docsite where we rendered API docs.

Most of these were not great, missing features like versioning, full-text search, a button to copy a code sample, or nav-to-edit.

And most importantly, there was no one place to look. As a developer, if you don’t know which Bazel Module contains the doc you need, you end up searching around all these places. Bazel Modules are commonly published on the Bazel Central Registry (BCR) at https://registry.bazel.build. So, that’s where I added them!

How it Works

Generating the Docs

Behind the scenes, Bazel has a built-in rule starlark_doc_extract, in the Java core code, which runs Bazel’s Starlark interpreter over a given Starlark file. The interpreter is required because .bzl files use a standard library which is Bazel-specific and not part of the Starlark language spec, and the documentation needs to be aware of that. It also needs to read from transitively-loaded files in the deps of a bzl_library target.

As a Starlark author, you don’t really want to think about generating the API docs, it should just work! So writing bzl_library in your BUILD files is sufficient - and there’s a Bazel Gazelle extension to write those for you.

The implementation of bzl_library in bazel-skylib doesn’t actually do any documentation generation, or even validation of the inputs.

So we have an improved one in bazel-lib. Here’s my first opportunity to link you to the documentation on the BCR: https://registry.bazel.build/modules/bazel_lib#-bzl_library-bzl

At this point, a rule author can bazel query ‘kind(starlark_doc_extract, //...)’ to see what APIs have their documentation available.

Publishing the Docs

We want to include these docs in the releases, next to the artifact users download. Most rules use the Publish-to-BCR workflow or app which require a release_prep.sh script, so we can just add a snippet there:

# Add generated API docs to the release
# See https://github.com/bazelbuild/bazel-central-registry/blob/main/docs/stardoc.md
docs="$(mktemp -d)"; targets="$(mktemp)"
bazel --output_base="$docs" query --output=label --output_file="$targets" 'kind("starlark_doc_extract rule", //...)'
bazel --output_base="$docs" build --target_pattern_file="$targets"
tar --create --auto-compress \
    --directory "$(bazel --output_base="$docs" info bazel-bin)" \
    --file "$GITHUB_WORKSPACE/${ARCHIVE%.tar.gz}.docs.tar.gz" .

Note: the --output_file flag was added in Bazel 7.5.0

Next, we want the module's metadata to point to that docs.tar.gz file, by editing the source.template.json to include `"docs_url": "https://github.com/{OWNER}/{REPO}/releases/download/{TAG}/{REPO}-{TAG}.docs.tar.gz",`

Note: you need publish-to-bcr version v0.2.3 or greater to pick up a fix for multiple replacements in source.template.json

Now the module just gets published as usual. The docs_url property points to an archive of all the stardocs in their binaryproto format.

Rendering the Docs

Google maintains https://github.com/bazelbuild/stardoc which is a Java implementation of a Velocity template renderer for stardoc binaryprotos. However the BCR UI is a TypeScript Next.js application. We don’t want to introduce a Java source dependency. Not only that — but look at the “Legacy WORKSPACE setup” on https://github.com/bazelbuild/stardoc/releases/tag/0.8.0 for an idea of the mass of transitive dependencies it requires at runtime.

As I pointed out in that blog post I linked at the start, we can just use the @buf/bazel_bazel.bufbuild_es NPM package to parse the binaryproto’s, so we get a short implementation of data fetching: https://github.com/bazel-contrib/bcr-ui/blob/main/data/stardoc.ts

Now that we have the Stardocs in a TypeScript object, we just need a standard React component to render them: https://github.com/bazel-contrib/bcr-ui/blob/main/components/Stardoc.tsx — this one is a little longer, but it’s mostly written and maintained by AI.

Come Use It and Contribute!

The BCR-UI project is governed by the Rules Authors SIG under Linux Foundation: https://github.com/bazel-contrib/bcr-ui

Your help is greatly appreciated!

• Update a ruleset to publish its docs

• Improve ruleset documentation, like by adding more copy-paste examples

• Suggest usability improvements to the Next.js rendering

Today, we’re excited to offer Aspect Workflows free for a limited number of open-source projects.

Try Our CI/CD Platform—Built for Bazel, Ready to support OSS

Aspect Workflows provides a modern CI/CD experience powered by Buildkite—the same engine we use to run our own projects. It also supports other CI systems like CircleCI and GitHub Actions, so you can integrate with the workflows you already use.

We’ve already been dogfooding this setup with our own bazel-examples repo and many of our Bazel rulesets—including several in the bazel-contrib GitHub org, such as rules_oci. The Buildbarn project is also on board, using it for their bb-storage CI. You can see the full list at https://buildkite.com/aspect-build.

To access the platform, head over to https://docs.aspect.build/ and create a free account.

What You Get

Projects on our OSS tier get access to:

• 🚀 Fast CI that powers our persistent runners that avoid cold-start latency

• 🔎 Detailed UI that shows exactly which test, or build step failed

• 💬 Streaming feedback in GitHub that posts real-time results directly to your PRs

• 🧰 Managed Buildbarn that provides a remote cache and execution backend, fully hosted

A Few Things to Know

This program runs in a shared (co-tenant) environment with other open-source workloads, which makes it ideal for public projects but not suitable for confidential code. There’s no formal SLA, and because open-source workloads are often idle and Workflows scales down to zero, it can sometimes take a couple of minutes for new machines to spin up. But once your build kicks off, it’s fast and smooth. ⚡️

Ready to Get Started?

If you're maintaining an open-source project and want to level up your CI/CD, we’d love to hear from you. Start at https://portal.aspect.build to begin the sign-up process.

We think this is a great way to customize your new workspace, but it has some downsides:

• You have to start by installing the Aspect Build CLI (or the scaffold tool that the templating is based on).

• It’s hard to point to a link on GitHub that says “here’s why we have Bazel configured this way” or to copy-paste some boilerplate from it.

• You don’t get a green “Use this Template” button on GitHub to press that creates a new repo, which is useful for one-off reproductions, playgrounds, or training courses.

• We didn’t have a nice README explaining how you can use the new workspace.

• It’s Aspect-branded, and so any training courses we might donate to Linux Foundation would likely trigger objections from competitors.

Introducing template repositories

The continuous integration (CI) setup for our template has always had “presets” — one for each language, plus a zero-language “minimal” preset and one with everything thrown in (the “kitchen sink”). We now publish the output of the generator for each preset to its own repo under its own org:

https://github.com/bazel-starters

The README here explains a whole bunch of features that are included:

• 📦 Curated bazelrc flags via bazelrc-preset.bzl

• 🧰 Developer environment setup with bazel_env.bzl

• 🎨 Formatting and linting built-in, using rules_lint

• ✅ Pre-commit hooks for automatic linting and formatting

• 📚 Language-specific package manager integration (e.g. pip, go.mod, npm, etc.)

• 🧱 Latest Bazel version pinned and working

• 🐳 Docker container support using rules_oci

• 🧪 Code generation tools (e.g. copier, scaffold, yeoman) to help you and your team stamp out new services or components quickly

We also found a clever trick to make a Markdown file directly executable as a Bourne Shell script. So each of the README files, such as https://github.com/bazel-starters/cpp are actually executed as part of our CI process. That means you can trust those instructions will actually work after you clone the repo.

What’s next

We are always improving the state-of-the-art for to get started more easily with Bazel. Look for an Aspect Build announcement coming up in November for BazelCon 2025 where these starters will get a lot more useful!

However, I often see developers struggling to model something that’s not a tree. Sometimes it’s a bush, or a chandelier. This is usually a sign that Bazel’s dependency + action graphs won’t work well, due to bad ergonomics and ruined incrementality.

Bazel dogma teaches us that all logic should be in Starlark (Bazel’s extension language) and described in BUILD files. I’ll show a few examples where this isn’t the “Right Tool for the Job”.

However, I’ll make a stronger case: Bazel is the inner core of a wider system. The core really only performs two jobs well:

1. Inspect the dependency and action graphs (aquery and cquery)

2. Populate a subset of bazel-bin and bazel-testlogs (build and test - though the latter can really be thought of as “build text files containing all the test runner exit codes”)

The wider system is a “task runner”. Makefile commonly serves this purpose, surrounding Bazel commands, but it has a trap: it overlaps with Bazel’s capabilities and makes it impossible to ensure you don’t have Build steps sneaking into the outer layer. At BazelCon this year, I’ll present a better task runner that lets you write tasks in Starlark. In the meantime, I’ll just illustrate the task runner layer with some Bash one-liners.

An archive of the whole repo

Our first example is common in Bazel rulesets. You want an archive that represents the whole source repository, so the shape is “take all the leaves and connect them directly to the root”.

This doesn’t work well in Bazel because packages are encapsulated, so a glob([**/*]) doesn’t gather up all the sources in subpackages. Instead you need an awkward tree of filegroup targets in every package that are linked together.

The alternative is to use the Right Tool for the Job: git archive. It has some lesser-known configuration options you can set in the .gitattributesfile (see https://git-scm.com/docs/git-archive#ATTRIBUTES) that help a bunch:

• Filtering out contents: instead of Bazel glob(excludes=[]) you can use export-ignore patterns

• Version Stamping the result: use export-subst to configure which file is stamped, then something like the following in that file:

•         _VERSION_PRIVATE = "$Format:%(describe:tags=true)$"
  
          VERSION = "0.0.0" if _VERSION_PRIVATE.startswith("$Format") else _VERSION_PRIVATE.replace("v", "", 1)
  

In an earlier post I covered this in more detail, including real-life examples: https://blog.aspect.build/releasing-bazel-rulesets-rust

All the “Something” targets

This one comes up a lot. Recently I’ve been working on distributing API documentation which is generated for code across the repo. You’ll recognize this pattern whenever it seems like bazel query 'some expression` | xargs bazel build is the model of what you want to ask Bazel for.

It’s tempting to model this is a “collector” target that’s just a long list of deps and then wonder “how am I going to keep this list of deps up-to-date as we add more something targets?” You can’t and shouldn’t.

The biggest reason to avoid this one is the shape of dependency graph you end up with. Developers will commonly trip over analyzing this target (just doing a query over the repo, or loading that package will do it). Then Bazel goes from incremental “only do the minimal work for the targets I requested” to performing a whole-repo step that downloads gigabytes of irrelevant tooling.

This time, the Right Tool for the Job is a small workflow outside of Bazel. Create a query expression that matches the targets you care about, and selects the outputs you need from them. We do this for the lint command to select “all the report files” for example. Here’s a full code listing for the API docgen task (note that we don’t literally use xargs because of the length of the command line can exceed ARG_MAX and spill into multiple spawns):

docs="$(mktemp -d)"; targets="$(mktemp)"
bazel --output_base="$docs" query --output=label --output_file="$targets" 'kind("starlark_doc_extract rule", //...)'
bazel --output_base="$docs" build --target_pattern_file="$targets"
tar --create --auto-compress \
    --directory "$(bazel --output_base="$docs" info bazel-bin)" \
    --file "$GITHUB_WORKSPACE/${ARCHIVE%.tar.gz}.docs.tar.gz" .

Compare with another version of the code

buf_breaking is a good example. It wants to see the prior state of the output (say at the Base commit of a Pull Request), and compare with the current one. Bazel sees a single snapshot of the source code for a given build. I’ve seen some customers write a repository rule to clone a different commit of the repository, which seems very brittle to me. Checking in the prior output is too hard to automate.

The Right Tool for this Job is to find CI artifacts from the Base commit for a given change, and run a comparison/validation tool after the build runs. Then write an updated artifact from builds on the main branch for subsequent comparisons.

gazelle

BUILD file generation is in this category, because Bazel has always refused to allow dynamic dependency graphs based on file contents. The team argues that the “no-op” build has to remain fast.

But it doesn’t matter how fast their tool is, if every user is then forced to wrap it in something slower. In this case, we always want something to run before Bazel’s loading phase: a step like how C++ builds run autoconf with a ./configure && make workflow.

Today engineers mostly have to discover their BUILD files are outdated (maybe there’s a compilation error about a missing dependency) and then do a manual bazel run //:gazelle - but if we had a Task Runner layer around Bazel, we’d just setup a step to run ahead of time.

(By the way, at BazelCon I’ll present two things: we can use Starlark to write the task that invokes Gazelle, and we can also extend Gazelle’s BUILD generation logic in Starlark!)

Coverage

Bazel has a coverage command, so why is this example here? Well, in my experience, it was a mistake - this command wanted to live in the task runner layer, but Google never wrote one. The coverage system is bad mostly because of things like lcov transformation and merging, and how difficult it is to configure.

Coverage really should have been formulated as a Task Runner that:

1. Builds the code under a Transition that enables an Instrumentation Configuration (pokes counters into the executable to track how many times a line or statement executes)

2. Runs the tests as usual. The coverage data files are configured to be additional outputs (using the TEST_UNDECLARED_OUTPUTS feature my intern added (hi John!))

3. After the tests are complete, collects the resulting data files. They might be LCOV format, or something else that needs to be transformed.

4. Presents the results, frequently by consulting the VCS so you can show incremental coverage (how many of the added/edited lines were tested)

Run

Even the bazel run command is a mistake in my opinion. It’s a nice “syntax sugar” for building a single target, then spawning it as a subprocess. But it’s missing things like watch mode, we needed a separate rules_multirun to get multiple servers to start, and has a ton of bugs around how the working directory is selected.

If we had a Task Runner, it would clearly not be Bazel’s job to do these things.

print

Buildozer is a great tool for machine-editing BUILD files, but also for quickly inspecting their contents in a purely syntactic pass that doesn’t trigger Bazel’s fetching, Loading, and Analysis phases. With a Task Runner layer, we can easily expose these Bazel-adjacent tools through the same interface engineers use to request build outputs, instead of making them install and learn about a variety of other tools. So we’d add a print task that doesn’t even invoke Bazel at all!

Next

To learn more about Aspect Build's Bazel developer workflow platform and professional services, visit aspect.build.

This release brings notable improvements in hermeticity, performance, and reliability, aligning pnpm more closely with the expectations and requirements of Bazel-based workflows that use rules_js.

(Only) Built Dependencies

Since pnpm v9 rules_js has required packages with build steps to be manually declared in the pnpm.onlyBuiltDependencies field of package.json. Now pnpm v10 has the exact same requirements as rules_js, see pnpm #8897 (as well as #7710 and #7716 for more background).

Explicitly declaring which packages require a build step improves determinism and hermeticity in both pnpm and rules_js. This allows rules_js to model Bazel build actions without first needing to download and inspect package contents.

This change was followed-up with pnpm.neverBuiltDependencies (#8958) in pnpm 10.1 to suppress pnpm install warnings about packages containing install logic without being listed in pnpm.onlyBuiltDependencies.

Secure SHA256 Hashing

Pnpm v10 has switched to more secure SHA256 hashing of content in the pnpm-lock.yaml file, see #8530. Bazel and rules_js already use sha256/512 for integrity checks, and rules_js will continue to align with pnpm lockfiles where pnpm v10 has upgraded to sha256.

Configuration changes

Pnpm v10 has made many other configuration related changed that do not directly effect integration with rules_js, but may effect your experience when upgrading such as:

• default hoisting has changed #8378

• NODE_ENV is now ignored on install #8827

• the @yarnpkg/extensions package was upgraded, this may alter resolved dependencies in edge cases

Catalogs

While actually a pnpm v9.5 feature, pnpm catalogs is a feature worth mentioning again. Catalogs have provided a way to stop repeating version numbers throughout your package.json files and declare a single version for a package in a single location, while keeping fine grained dependencies in your projects.

Catalogs are especially useful in large monorepos where Bazel and rules_js are normally used.

Catalogs are used by pnpm when generating the pnpm-lock.yaml file and does not change the underlying lockfile format, so rules_js supports comes for free.

Final Thoughts

Like pnpm v9 last year, pnpm v10 continues to move toward more deterministic builds that continue to align with Bazel and rules_js, further proving pnpm was the right choice for the Bazel and rules_js ecosystems.

See the pnpm v10 release and followup releases for a full list of changes.

Aspect recommends that Continuous Delivery is modeled as the step of the pipeline where built artifacts are uploaded from the build machine to a well-known repository location. This could be a container image registry like Docker Hub, a blob store like AWS S3, or even a database.

This makes a clear separation of responsibilities between CI, CD and Deployment:

• The CI pipeline runs all the tests to confirm the repo is in a shippable state

• The CD pipeline should then upload only the artifacts that are:

  • configured with BUILD.bazel files. Product engineers don't need to worry about setting up CD

  • green: it can prove that all relevant tests are passing

  • changed from a previous build

• The deployment system

  • locates and "promotes" artifacts to the next environment, such as "dev", "staging", or "prod".

Build vs. Buy

The recommendations in this guide can be applied in two ways:

• DevInfra teams may wish to implement and operate a custom system for their organization, or

• Use Aspect Workflows, which provides this feature out-of-the-box.

What is "deliverable"

A deliverable artifact is one that contains both the binary or files to push, as well as the "pushing" logic that knows how to perform the upload. It might also send a message to the deployment system to trigger an auto-deployment of the new artifact.

In Bazel terms, this means a deliverable should be an executable program that can be bazel run.

Container Images

The rules_oci oci_push or rules_docker container_push rules can both be executed with bazel run to push a Docker image to a registry like Docker Hub.

Therefore these rules are considered "deliverable".

Git Push

Sometimes artifacts belong in a separate code repository. For example, an SDK built from the API definitions in a monorepo needs to be published.

See an example git_push executable in this repository.

S3 upload

See s3_sync.

Which targets to deliver

A bazel query expression is the most convenient way to locate deliverable targets. Users may choose a tagging scheme for their workspace (i.e. "all targets with tags = ['artifact']"), or deliver well-known rule kinds (i.e. oci_push), or both.

Which changes to deliver

To optimize time and money, it's best to deliver only "changed" targets. This avoids wasted time and resources uploading the same artifact repeatedly. It also means that release engineers won't have to sort through a massive list of duplicates when choosing a release.

There are two approaches for choosing "changed" targets:

1. Predict the changes based on a version control delta. For example you could git diff between the hash being delivered and the "prior successful" delivery hash, then use a tool like bazel-diff or target-determinator to produce a list of targets that might be affected by those changes.

2. Determine empirically based on what is actually different. This requires determinism, so it must only use unstamped build results (--nostamp). In most cases a green CI run just completed, so these unstamped outputs are easily available.

Aspect recommends following the second approach because the first has some downsides:

• bazel-diff is incorrect and will sometimes miss affected targets, so they aren't delivered.

• target-determinator is slow and may hurt the "service level indicator" of time between pushing a hotfix and being able to release that fix.

• It will over-deliver, because sometimes a source change doesn't actually factor into whether the release binary changes, such as for a comment-only change.

The rest of this section provides more details about the second approach (determine empirically).

To determine whether Workflows should deliver that executable target on a particular commit, it is first hashed using the aspect outputs command with a special pseudo-mnemonic "ExecutableHash", for example:

$ aspect outputs 'attr("tags", "\bdeliverable\b", //...)' ExecutableHash
//cli:release h1:cj8OUC3l3fIr3Zxnffk6y7gukLOJmiWRCAQoqadg66Y=
//workflows/rosetta:release h1:kjHVajw+Nta2kh3Epcd32DkZxTE1NHA8b5N7hCNFNSM=

You need a lookup database to store previously delivered hashes. If the hash value matches one previously seen, then skip delivery of that target.

Debugging changes to deliver

You can run the aspect outputs command locally to understand whether a given change to a source file results in a new executable. Sometimes the result may be surprising. For example, if a comment in a .go source file is changed, the compiler produces the same .a file as a result, so the hash seen on the uploader executable is unchanged.

Another scenario that won't change the executable is when some production configuration is changed. For example, you may use Helm charts to deploy to Kubernetes. If these aren't included inside the image, then changes to these files won't cause a new delivery.

Perform the delivery

Run each deliverable target with stamping enabled. You can do this in a script which reads the targets from a manifest file, essentially cat $delivery_manifest | xargs -N1 bazel run --stamp.

Some folks use a devcontainer or VDI (Virtual Desktop Infrastructure) to describe the tooling you need installed, but that’s heavy and slow. If we’re using Bazel, it already has features to give a hermetic environment, right?

A year ago, I wrote an article describing how to get tools on your developers local machine using Bazel, and today I have an update.

In the technique from my original post (https://blog.aspect.build/run-tools-installed-by-bazel), users have to change their behavior, typing ./tools/my-tool rather than just my-tool. Retraining developers to have a different command under their fingers is hard, especially for things they run all the time.

Shortly after I wrote that post, Fabian Meumertzheim created https://github.com/buildbuddy-io/bazel_env.bzl. This is an alternative technique I’ll write about today. It puts the tools on the $PATH instead, fixing the ergonomic issue from the first technique — however there are always trade-offs! This one requires engineers manually install the direnv tool before they get setup.

How direnv works

direnv is a command-line tool that automatically sets and unsets environment variables when you cd into or out of a directory. It’s especially useful for managing project-specific environment variables, such as secrets, configuration settings, or language versions (like Python or Node.js versions).

• You place a .envrc file in your project directory.

• This .envrc file contains shell commands to export environment variables or run setup scripts.

• When you enter the directory, direnv loads the .envrc file and applies the environment changes.

• When you leave the directory, it automatically reverts those changes.

Here’s how it looks when I enter Aspect’s monorepo (named “silo”):

alexeagle@aspect-build ~ % cd Projects/silo
direnv: loading ~/Projects/silo/.envrc
direnv: export ~PATH

Installing direnv isn’t just a matter of getting the program on your machine. It also needs to hook into your shell (it supports bash, zsh and many others). And finally, you must explicitly allow each .envrc to be trusted.

To follow this pattern, you’ll need to instruct your developers to install this first tool manually. Fortunately they’ll get a reminder of the instructions in the next step.

bazel_env creates a .envrc

After installing bazel_env.bzl you’ll have a runnable Bazel target, typically bazel run //:_bazel_env or bazel run //tools:bazel_env.

The bazel_env target is defined with a dictionary that maps a tool name to put on the PATH, to some other target that provides it. What kinds of targets can those be? Take a look at the example: https://github.com/buildbuddy-io/bazel_env.bzl/blob/main/examples/BUILD.bazel

1. Binary targets for programs you author yourself in the monorepo

2. Tools provided by a toolchain, like go, node, pnpm, cargo, etc

3. With https://github.com/theoremlp/rules_multitool you can run multitool to update a tools.lock.json file, and all of these tools are installed.

4. CLI utilities distributed by a package manager, like console scripts from PyPI, bin entries from the package.json of NPM packages, Go utilities (see scaffold example here), and so on.

% bazel run //tools:bazel_env
INFO: Analyzed target //tools:bazel_env (580 packages loaded, 72963 targets configured).
INFO: Found 1 target...
Target //tools:bazel_env up-to-date:
  bazel-bin/tools/bazel_env_all_tools
INFO: Elapsed time: 9.399s, Critical Path: 0.45s
INFO: Running command line: bazel-bin/tools/bazel_env.sh

====== bazel_env ======

✅ direnv is installed
✅ direnv added bazel-out/bazel_env-opt/bin/tools/bazel_env/bin to PATH

Tools available in PATH:
  * aws:             @aws
  * pnpm:            @pnpm
  * gofumpt:         @@rules_multitool~~multitool~multitool//tools/gofumpt:gofumpt
  * jsonnetfmt:      @@rules_multitool~~multitool~multitool//tools/jsonnetfmt:jsonnetfmt
  * shfmt:           @@rules_multitool~~multitool~multitool//tools/shfmt:shfmt
  * terraform:       //tools:terraform
  * yamlfmt:         @@rules_multitool~~multitool~multitool//tools/yamlfmt:yamlfmt
  * ruff:            @@rules_multitool~~multitool~multitool//tools/ruff:ruff
  * shellcheck:      @@rules_multitool~~multitool~multitool//tools/shellcheck:shellcheck
  * buf:             @@rules_multitool~~multitool~multitool//tools/buf:buf
  * buildozer:       @@rules_multitool~~multitool~multitool//tools/buildozer:buildozer
  * docker-compose:  @@rules_multitool~~multitool~multitool//tools/docker-compose:docker-compose
  * diesel-cli:      @@rules_multitool~~multitool~multitool//tools/diesel-cli:diesel-cli
  * etcdctl:         @@rules_multitool~~multitool~multitool//tools/etcdctl:etcdctl
  * grpcurl:         @@rules_multitool~~multitool~multitool//tools/grpcurl:grpcurl
  * ibazel:          @@rules_multitool~~multitool~multitool//tools/ibazel:ibazel
  * multitool:       @@rules_multitool~~multitool~multitool//tools/multitool:multitool
  * otel-cli:        @@rules_multitool~~multitool~multitool//tools/otel-cli:otel-cli
  * otelcol-contrib: @@rules_multitool~~multitool~multitool//tools/otelcol-contrib:otelcol-contrib
  * promtool:        @@rules_multitool~~multitool~multitool//tools/promtool:promtool
  * pyrra:           @@rules_multitool~~multitool~multitool//tools/pyrra:pyrra
  * tflint:          @@rules_multitool~~multitool~multitool//tools/tflint:tflint
  * tfsec:           @@rules_multitool~~multitool~multitool//tools/tfsec:tfsec
  * buildifier:      @buildifier_prebuilt//:buildifier
  * scaffold:        @com_github_hay_kot_scaffold//:scaffold
  * node:            $(NODE_PATH)
  * cargo:           $(CARGO)
  * rustfmt:         $(RUSTFMT)

Toolchains available at stable relative paths:
  * nodejs: bazel-out/bazel_env-opt/bin/tools/bazel_env/toolchains/nodejs
  * rust:   bazel-out/bazel_env-opt/bin/tools/bazel_env/toolchains/rust

direnv: loading ~/Projects/silo/.envrc
direnv: export ~PATH

Try it out!

This is now the pattern we stamp out in new Bazel repositories generated by running our CLI, aspect init.

See https://docs.aspect.build/guides/getting-started/

Guardrails

There are a few things that can go wrong, which are worth pointing out.

You’ll note that the tools are actually installed under a named output folder, bazel-out/bazel_env-opt - what if the user runs a bazel clean? This case is well handled, since direnv is able to report errors in the .envrc file with a custom message:

alexeagle@aspect-build silo % bazel clean
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
direnv: loading ~/Projects/silo/.envrc
direnv: ERROR[bazel_env.bzl]: Run 'bazel run //tools:bazel_env' to regenerate bazel-out/bazel_env-opt/bin/tools/bazel_env/bin
direnv: export ~PATH

Any time the tools change, everyone has to run it again.

It’s also not lazy (at least not yet). When you run bazel_env it needs to fetch all the tools, even those you never plan to run. See https://github.com/buildbuddy-io/bazel_env.bzl/issues/14

As a workaround, you can have multiple bazel_env targets, but users have to choose the right one for the work they intend to do.

Protocol Buffers needs a code generation step, turning your .proto files into “stub” code in your language of choice which provides marshaling/unmarshaling of proto messages (binary or JSON typically) into language-idiomatic data structures. If you use services, then you also want “stub” code to implement clients and/or servers that use gRPC.

This code generation needs a “compiler” for the protobuf language and the idiomatic compiler provided by the Protobuf team at Google is protoc. They provide binary releases of this tool on https://github.com/protocolbuffers/protobuf/releases and that’s where most protobuf users get it (maybe via a plugin for their Build system, such as https://github.com/google/protobuf-gradle-plugin)

However under Bazel, the common means of getting the tool is to compile it yourself from this cc_binary target. This is what Googlers expect, having lived in an extreme monorepo where everything is vendored, and security is paramount. See an interesting comment I got in a discussion:

Interesting. I wasn't aware there was a contingent of Bazel rules maintainers who are opposed to google3-style source-only builds.

Yes! I’m one of those maintainers. That’s because most of us don’t have perfect caching. Ideally you don’t notice protoc compilation other than “the first time”. In practice, I wait for protoc to build all the time, and it spams my CI log with gcc warnings that are irrelevant to me. And on some machines, the compilation fails. If you haven’t listened to the Aspect Insights podcast, this topic was all the way back at episode 1.

Avoiding it

The exciting news is that many Bazel projects can stop compiling protoc TODAY.

Aspect engineer Sahin worked with the Bazel team long ago to help introduce a new flag, allowing the protoc binary to be registered as a toolchain. This gives us the flexibility of registering something other than the cc_binary target. In fact, you’re free to register /bin/false as your protocol buffer compiler, if you want something very fast and very broken.

I then wrote https://github.com/aspect-build/toolchains_protoc which gives the convenience of downloading the official binary release, and registering that. See the docs on that repo, and especially the examples folder.

Unfortunately we haven’t made as much progress as we’d like, because there’s an ecosystem-wide change required. Everywhere that references the protobuf compiler needs to use the toolchain to resolve it. As of this writing, https://github.com/protocolbuffers/protobuf/pull/19679 is pending as a fix for the worst culprit.

Then there is a long, long tail of issues. For example, I was waiting 10 minutes to cut releases of our Bazel OSS rulesets, just because the stardoc documentation generator is a java_binary target with a hard-coded transitive dep on the cc_binary(name=”protoc”) target and the CD step wasn’t getting a cache hit for it. So I worked around that with another pre-build: https://github.com/alexeagle/stardoc-prebuilt and then applied a small patch to our rulesets to override the renderer attribute, for example: https://github.com/aspect-build/rules_js/pull/2156. It’s unfortunate to have to work around the problem, but in this case it’s worth it to fix the slowest step in our releases.

Making it official

We’re hoping to upstream our toolchains_protoc to the protobuf repository, so that the default Bazel experience will be fast. Of course, you can still choose to register the cc_binary target as your proto compiler toolchain, if you want to build it from source.

With Bazel, this story is different. All *_binary targets have a well known directory structure called Runfiles which makes it insanely easy to decide what structure the Javascript container will have. You just take the Runfiles directory tree, put it into a tar archive, add it to your oci_image and call it a day, right?

Though this is a fine approach for small applications (<50MB), it does not scale well beyond a few gigabytes because of increased build and deploy times. You could take a nap waiting for the whole layer to be uploaded and redeployed after a single line change.

The JsImageLayer rule from rules_js keeps you moving. It is a packaging rule that efficiently creates JavaScript containers using the Runfiles structure.

In the early days, JsImageLayer created two layers for the whole container. The node_modules layer contained everything that changed infrequently such as npm dependencies and node interpreter (yes, rules_js includes a hermetic Node.js interpreter) and the app layer contained first party JavaScript code.

This worked fairly well because a single line change did not cause node and node_modules to be uploaded and redeployed. However, we realized that it was not good enough. node binary rarely changes and node_modules changes more frequently than node, so it is not economical to bundle them together as a single layer.

That’s exactly what prompted the rules_js maintainers (us) to add more layers, ordered from infrequently changed to frequently changed.

In version 2.0 of rules_js, js_image_layer created more layers for better build and deploy time performance. It worked well for most JavaScript containers, but there is no one-size-fits-all approach. People reached the limit of that optimization too.

$$\[ \begin{array}{ccccc} \text{node} & \text{package_store_3p} & \text{package_store_1p} & \text{node_modules} & \text{app} \\ \uparrow & \uparrow & \uparrow & \uparrow & \uparrow \\ \text{interpreter} & \text{3rd party npm} & \text{1st party npm} & \text{symlinks} & \text{application code} \\ \end{array} \]$$

What happens if you have 150,000 files from 3rd party npm packages? Changing one npm package led to the whole layer being rebuilt and sent over the network, causing unwelcome flashbacks to the early days of js_image_layer. The problem was even worse if you had npm packages that shipped with prebuilt binaries or .node bindings. In my consulting work with AI companies, I learned how monstrous pip packages can be (👀 CUDA). I knew what I had to do.

Introducing JsImageLayer layer groups, a new rules_js feature that allows fine grained control over the number of layers created. Users can create additional layers to further optimize JsImageLayer by supplying a dictionary of names and regex that is evaluated against the path.

An example of putting @huge/pkg into its own layer can be written as follows.

$$\[ \begin{array}{cc} \text{layer_groups} & \text{default layers} \\ \uparrow & \uparrow \\ \text{any number of additional layers} & \text{the layers shown above} \\ \end{array} \]$$

We also took this as an opportunity to optimize how we generate layers. Previously, js_image_layer had a custom Node.js program to create layers (.tar archives). Though it worked great for medium-size archives (<= 200MB), streaming backpressure greatly reduced its efficiency. Fixing this was as easy as building the archives with good ol’ libarchive (also known as bsdtar).

One of our customers saw a 40% speed improvement with no additional configuration change. With some additional layers, it became 50% faster due to better parallelization of build actions.

A benchmark with the cold build times for a js_image_layer target with no change to the BUILD file demonstrates 52% speed improvement for overall build time.

You can now add as many layers as you want based on size, how frequently they change, or any other criteria. You can even override the default layers by using the same keys in the dictionary.

The Layer Groups feature is now available in rules_js version v2.3.5!

But a couple weeks ago, there was yet another wake-up call: a vulnerability in tj-actions infected an unknown number of open-source projects, whose CI pipelines probably exposed tokens that can be used to hijack their projects.

Like the xz vulnerability last year, this happened in an OSS repo where a tiny number of individuals (often hobbyists) are responsible for holding up the security posture of enterprise users. And many Bazel rulesets are similarly-staffed projects! Google is transferring repositories to the Linux Foundation (under the bazel-contrib GitHub org) and this comes with no resources for maintenance.

What would happen if a Bazel ruleset author leaks a Personal Access Token? Rules do things like download toolchains, and invoke them to produce your build outputs. A malicious release artifact could easily inject code into the binaries you build and ship. This attack vector is outside of your code, and outside of the third-party packages that your security scanner runs on. I don’t know of any companies who are taking steps to prevent this today.

What can you do about it?

Whenever taking a new dependency, an enterprise security team will scan the sources for exploits, because they cannot trust the maintainers have a security posture that meets their requirements. When the dependency is updated by dependabot or renovate, hopefully they’re reviewing the delta to the sources since the prior trusted version.

It’s convenient that rulesets (and C++ packages) are published on the Bazel Central Registry so users don’t have to vendor the sources of all the modules they depend on. However as a consequence of GitHub’s checksum stability outage two years ago, Bazel modules generally publish a release artifact, which is constructed from the sources. How can an enterprise consumer know whether that artifact is truly built from the sources they’ve reviewed?

This is the assurance we can get from a framework like https://slsa.dev/. Thanks to a grant from Google and a bunch of help from Appu on the Distroless team, Aspect has upgraded the supply-chain for Bazel rulesets to optionally include an attestation, which is a cryptographic proof-of-trust. It allows an end-user of the module to place their trust in the build system vendor (GitHub in this case) that the build artifacts are provably constructed on a secure machine, given the sources and build scripts that exist in the repository for that release.

Attestations of BCR modules

As an example, let’s look at https://registry.bazel.build/modules/aspect_rules_lint/1.3.4. To be secure, one should follow the “Release Notes” link to navigate to the GitHub repo, and use the “Full Changelog” link to scan through the source code changes since the prior release we trusted. Okay, nothing looks malicious. How can we trust the release artifact on BCR?

That release also has three new files, with a .intoto.jsonl extension. These are attestations of provenance. What does that mean? Since the release was built on GitHub Actions, using GitHub-hosted runners, GitHub is providing a proof that the release artifact is constructed from the sources in the repo, using the workflow definition in the repo.

GitHub CLI provides the attestation verify command. If we download the release artifact we can try it out:

% gh attestation verify ~/Downloads/rules_lint-v1.3.4.tar.gz --owner aspect-build --signer-repo bazel-contrib/.github       
Loaded digest sha256:a7dfbfe0aa2fb960911a1589fa2ebc4f9fd0e25b0090edb54a9dfac73fdd6444 for file:///Users/alexeagle/Downloads/rules_lint-v1.3.4.tar.gz
Loaded 1 attestation from GitHub API

The following policy criteria will be enforced:
- Predicate type must match:................ https://slsa.dev/provenance/v1
- Source Repository Owner URI must match:... https://github.com/aspect-build
- Subject Alternative Name must match regex: (?i)^https://github.com/bazel-contrib/.github/
- OIDC Issuer must match:................... https://token.actions.githubusercontent.com

✓ Verification succeeded!

The following 1 attestation matched the policy criteria

- Attestation #1
  - Build repo:..... aspect-build/rules_lint
  - Build workflow:. .github/workflows/release.yml@refs/tags/v1.3.4
  - Signer repo:.... bazel-contrib/.github
  - Signer workflow: .github/workflows/release_ruleset.yaml@refs/tags/v7.1.0

A similar verification is built-in to the BCR presubmit process as well, ensuring that the attestations in the repository (https://github.com/bazelbuild/bazel-central-registry/blob/main/modules/aspect_rules_lint/1.3.4/attestations.json) may be trusted. Ultimately we expect that the Bazel client itself will transparently verify modules as they are downloaded, which will also support private registries.

Rolling out to the ecosystem

We didn’t just make this work for rules_lint. The changes needed have been made to the Publish to BCR helper, which is now a reusable workflow rather than a GitHub App. That means we’ll soon be able to trust releases from all the rules that reuse this release and publish workflow.

Thanks to Derek for doing the engineering work! If you’re a module author, follow that link to the docs for Publish to BCR.

In this post, I look at what (the new) Dagger is, how it works, and how it compares to Bazel.

Dagger history and concepts

Dagger, started in 2022 by Solomon Hyke, the same creator as Docker, is part of the ecosystem of tools I call "code as infrastructure." In this model, you use your programming language of choice to define your infrastructure and test and build pipelines.

Under the hood, everything runs in containers, making Dagger pipelines portable and moveable between local runs or running on CI.

Dagger defines every task and workflow (including "infrastructure" setup) as a Dagger Function written in one of the currently available SDKs: Go, Typescript, and Python. Yes, this means you write functions to create Functions. It's a little confusing.

You can extend the core Dagger functionality with Daggerverse modules, which include a wide variety of use cases, from using helm charts to spinning up Kubernetes servers and various package managers.

At the center of everything is the Dagger Engine, which is open source and handles maintaining the connections between functions, caching, state, telemetry, and more.

There's also the Dagger Cloud, which currently provides a browser-based interface for tracing and debugging issues with Dagger Functions. This is Dagger's monetization strategy and is free for one user.

Example

This is the TypeScript example Daggerized application pipeline from the Dagger documentation. It builds two containers, runs tests, and then publishes one of the built containers.

import { dag, Container, Directory, object, func } from "@dagger.io/dagger"

@object()
class HelloDagger {
  /**
   * Publish the application container after building and testing it on-the-fly
   */
  @func()
  async publish(source: Directory): Promise<string> {
    await this.test(source)
    return await this.build(source).publish(
      "ttl.sh/myapp-" + Math.floor(Math.random() * 10000000),
    )
  }

  /**
   * Build the application container
   */
  @func()
  build(source: Directory): Container {
    const build = this.buildEnv(source)
      .withExec(["npm", "run", "build"])
      .directory("./dist")
    return dag
      .container()
      .from("nginx:1.25-alpine")
      .withDirectory("/usr/share/nginx/html", build)
      .withExposedPort(80)
  }

  /**
   * Return the result of running unit tests
   */
  @func()
  async test(source: Directory): Promise<string> {
    return this.buildEnv(source)
      .withExec(["npm", "run", "test:unit", "run"])
      .stdout()
  }

  /**
   * Build a ready-to-use development environment
   */
  @func()
  buildEnv(source: Directory): Container {
    const nodeCache = dag.cacheVolume("node")
    return dag
      .container()
      .from("node:21-slim")
      .withDirectory("/src", source)
      .withMountedCache("/root/.npm", nodeCache)
      .withWorkdir("/src")
      .withExec(["npm", "install"])
  }
}

You chain Functions together, meaning other Functions can call them. To run this example, you call the publish Function, passing a local code directory:

dagger call publish --source=.

The publish Function, in turn, calls test, which calls build, which calls buildEnv. You can pass variables between them, such as the code directory, and use any other aspect of the programming language you're using should you need them.

The Dagger-specific methods in each Function are fairly self-explanatory and often mirror their name and function in a Dockerfile. Again, they use chaining from the base dag client and any of its core types, such as a Container.

Once you run the pipeline and it is complete, besides anything retained for caching, the Dagger engine tears everything down. However, like with Docker, you can maintain state by mounting local volumes from the host system.

Bazel and/or Dagger

I initially set out to write this post looking at how to use Bazel and Dagger together. But I almost immediately came across this line in the documentation:

Before going any further, you should look for reasons not to adopt Dagger. Your project may not be a good fit for Dagger if: …

• You are happily using a monolithic toolchain, such as Gradle, Nix or Bazel, with no exception and no fragmentation within the team

There are also many discussions within the community on how the tools compare and contrast, often coming down to "stick with what works for you".

So, instead, what are the key differences and overlaps? In the general spirit of technology and developer tools, there are no hard and fast "best" answers. Often, it depends.

Native verses Containers

Dagger runs tasks in containers, which makes its pipelines portable. However, not everything can run in containers. They add their own overhead, such as container runtimes.

Bazel uses native environments to run tasks which are more performant, but can have inconsistencies between platforms and require accommodations for portability.

Configuration language

Bazel uses Starlark, which is Python-like but requires understanding some complex new concepts. It has widespread support for plugins and linters in IDEs.

Dagger uses Python, Go, or TypeScript with the relevant SDK, making it simpler to start. The world of "code as infrastructure" tools is interesting and includes something like Pulumi. I'm unsure if it has widespread adoption as a concept or if it's something that dev and DevOps teams actually want to mix together.

Language support

Support for most aspects of Bazel comes in the form of "rules". By default, Bazel has support for C, C++, Java, Objective-C, Proto Buffer, Python, and Shell. Through 3rd party community rules, Bazel can support many other languages natively.

While you can only write pipelines in one of the supported SDKs (or call the Dagger API directly), Dagger can build, test, and run whatever programming language you can run in containers.

Reproducibility

One of Bazel's main features is its reliable reproducibility no matter when or where you run a build or test.

Dagger relies on the reproducibility of containers for its own guarantees. As it has this reproducibility, it can add caching, helping speed up subsequent runs.

Community

Bazel originates in Google, but now has a large community of users, contributors, and a large ecosystem of rulesets and plugins around it to extend core functionality. Bazel is about nine years old. This is relatively new in the grand scheme of similar tools such as Nix or Gradle, but also perhaps considered "old" by some cutting-edge development teams.

Dagger is new and largely directed by one company. However, it is open source and has a healthy mix of contributors. The module ecosystem is reasonable, but it's always hard to predict how much maintainers will keep those up to date.

DevEx

Bazel treats most things as artifacts that depend on each other, which can require some rethinking for your processes but is also similar to Nix. Bazel often needs a reasonable amount of configuration files and rulesets to start.

While it uses common language patterns and containers (which are fairly established now), Dagger also requires some rethinking, especially if you're coming from "as code" tools. I also found that whilst the practice of chaining was conceptually straightforward, I found myself getting lost in it sometimes and passing lots of variables back and forth. Perhaps the biggest blocker is if you don't use Python, Go, or TypeScript, writing pipelines is more challenging. It's still possible to use it by calling the GraphQL API, but then you lose a lot of the advantages. And while containers are fairly flexible, there are still certain tasks that you can't or don't want to run in them.

Stick with what you know

Dagger and other "code as infrastructure" tools like it are interesting and tend to attract a lot of tech media attention. However, they are still new and often developed largely by one company funded by venture capital. This can mean the tool won't stick around, no matter how promising it is.

Echoing the statements I read in the Dagger community, but also those of any pragmatic developer tool community. There are always new shiny tools to try. They may offer benefits over what you use already, but if what you have works reliably, and you have staff who understand it, is it worth the effort? That's time you could spend on servicing and improving customer's needs.

Enhanced Security

Customer cloud accounts have security protocols to isolate networks on Virtual Private Clouds (VPCs), enforce custom IAM roles, and firewall applications to prevent unintended access. CI/CD systems are core to the software supply-chain, so vulnerabilities matter! Self-hosted CI infrastructure is subject to the same policies.

It requires less trust from the vendor. While Aspect is SOC2 certified of course, there are always other security and business risks of relying on a vendor to operate the infrastructure. The reduced risk in self-hosting has advantages in legal and procurement processes as well.

Self-hosted infrastructure-as-code (IaC) also allows your security scanning tools to operate over the vendor’s infrastructure definitions, including any co-maintenance policies granted to the vendor’s on-call engineers.

Data Control and Compliance

Data is retained in the customer’s cloud, and is subject to the access auditing, encryption, and retention policies enforced by their platform team. This is especially useful in industries where regulations like GDPR, HIPAA, FINRA require constraints around data management.

Cloud Pricing

Many companies have a cloud contract that reserves some capacity to get a better pricing agreement. Others like startups have credits given by the cloud sales team or from their investors. Sometimes there’s a requirement to utilize all the compute credits or reservations.

Hosting CI infrastructure in the customers cloud account lets them save on hosting costs, compared with a vendor-hosted model.

Honest Billing

Running tests can be resource-intensive, and you want to be able to run as many as you need. When a vendor meters their service based on usage, it’s difficult to budget for predicted SaaS costs.

In the case of Build & Test, the vendor should be expected to reduce the cloud compute costs by optimizing use of caching and right-sized, elastic-scaling instances. If the vendor is rewarded for consuming compute, they have a perverse incentive to use more resources rather than less.

Self-hosting infrastructure puts the resource billing under the customers control, and allows them to scrutinize the optimizations the vendor provides.

Low-latency, high-bandwidth integrations

Running infrastructure in the customers cloud makes it much more straightforward to introduce components developed in-house or by other vendors. For example, a Build system infrastructure should be network-adjacent to a remote development cluster or remote IDE backends.

Addressing the Productivity Bottlenecks

It’s 2024, and many engineering organizations still struggle with slow, costly, and unpredictable developer workflows! The causes vary from unreliable dependency management, brittle IDE configuration, slow builds, and the impracticality of automated integration testing. Despite advancements in the DevOps stack like containerization, CI/CD, and infrastructure as code, the increasing complexity of software projects is driving up build times and slowing down teams. As projects grow, particularly with the rise of monorepos and reliance on third-party dependencies, idiomatic language-specific tooling is becoming a major bottleneck for companies striving to deliver features quickly and securely at scale.

At Aspect, we are uncomfortably excited to solve these problems. By mastering advanced techniques such as remote execution and caching and hermetic dependency management, we help teams eliminate the delays caused by slow and flaky builds. We bring developers along with us thanks to unrelenting focus on ergonomics, documentation, and support. Our system optimizes every step of the build process, ensuring that development teams can focus on creating and shipping great products.

Our Journey: bootstrapping from Google

Greg Magolan and I first worked together at Google on the Angular team adopting Bazel, a large-scale multi-language build system that pioneered how massive organizations like Google handled complex software builds. We loved that Google decided to open-source the tool, but it was very clear that companies would have difficulty achieving the benefits promised in the Bazel 1.0 blog post.

Rather than start with venture funding, we bootstrapped for three years by providing Bazel consulting services to over 50 companies. This gave us a front-row seat to witness painful bottlenecks that processes like dependency management, builds and Continuous Integration can create, and also the first taste of success in helping early customers overcome them.

Aspect placed a big bet on our Open Source Software, which for us is both a passion and our code of ethics. We’ve been the authors of Bazel’s JavaScript and TypeScript support from the beginning, and have added Docker (OCI), Python, and more. Just recently, the Bazel Central Registry got its 2,000th entry, and we’re proud that 23% of those are from Aspect! I was also the catalyst for moving Bazel’s community and yearly conference to the Linux Foundation, setting the stage for meaningful community governance.

With Aspect Workflows, we’re layering on our open-source and bringing that same level of optimization from our consulting days to a wider audience, helping software teams scale their workflows to manage scale and complexity.

Since launching, Aspect Workflows has deployed at numerous customers, provided 40-60% cost savings, accelerated builds 10x and tests 2-3x, all while maintaining the reproducibility and security necessary for complex, large-scale development.

Fueling Our Growth with Seed Funding

We met David Waltcher over two years ago. His insight was immediately obvious, as he wrote in his introduction to us, “I’ve been spending a lot of time around Bazel… as the Google family has permeated other organizations, everyone in our network has been raving about the multi-language support and caching.” We’re glad he followed us through our bootstrap journey and our gradual conversion to recurring revenue, and we’re proud to partner with FirstMark and Preston-Werner Ventures, who bring a wealth of experience in developer tools and share our vision for transforming how software teams operate. With their support, we’re excited to accelerate our product roadmap, expand our team, and push the boundaries of what’s possible in large-scale multi-language software development.

We frequently share exciting announcements, so follow us at https://www.linkedin.com/company/aspect-build/

-Alex Eagle

CEO, Aspect Build