Bazel for SONiC Foundation Projects

As the Linux Foundation’s SONiC Foundation continues to drive forward an open, standards-based network operating system for the industry, one challenge has become increasingly visible across the community: how to build, test, and release SONiC components with speed, consistency, and confidence. The project has grown in complexity—diverse hardware platforms, multiple languages and toolchains, distributed teams, and the rising expectation that networking software behave like modern cloud software.

This is exactly where Bazel, the open-source, high-performance build and test system originally created at Google, can play a transformative role. Bazel offers SONiC (Software for Open Networking in the Cloud) contributors, adopters and vendors the reliability, scalability, and repeatability needed to sustain a world-class network OS at global scale.

Below, we explore the case for adopting Bazel across SONiC Foundation projects and how it can meaningfully improve developer productivity, platform compatibility, security posture, and release engineering.

1. Reproducible Builds Are No Longer Optional

SONiC today supports an expanding matrix of devices, ASICs, and software packages. That's its strength—but also its build challenge. Makefiles, ad-hoc scripts, and hand-rolled toolchains multiply variability and increase onboarding time, especially for new vendors.

Bazel provides hermetic, reproducible builds, ensuring that:

The same inputs always produce the same outputs
Dependencies are fetched deterministically and cached
Toolchains and container images are versioned and immutable
Builds can run anywhere—from a developer's laptop to CI runners to cloud build farms

This consistency dramatically reduces "it works on my machine" issues. For SONiC's multi-vendor ecosystem, it means any contributor can build and test with confidence against a shared standard.

2. A Universal Build System for a Polyglot Codebase

SONiC involves C++, Python, Go, Rust, Docker, kernel modules, switch-vendor SDKs, and more. Bazel excels here:

First-class multi-language support
Extensibility through Starlark rules
Deterministic container and image builds
Cross-compilation support for ARM, x86, PowerPC, and ASIC-specific toolchains

Instead of maintaining fragmented build logic across repositories, SONiC developers can standardize on a single system that handles everything from low-level DPDK components to high-level services.

3. Cloud-Native CI/CD That Scales With the SONiC Community

Bazel was designed for massive scale and parallelism. For SONiC maintainers, this directly translates into:

Faster builds through fine-grained caching
Incremental rebuilds that recompile only what changed
Remote execution to distribute builds across clusters
Remote caching to avoid duplicated work between developers and CI jobs
Unified pipelines across all repos and languages

As the community continues to expand, this infrastructure lets new contributors ramp up quickly and ensures that CI is fast, stable, and cost-efficient.

4. Stronger Security and Compliance

Networking software is increasingly subject to regulatory scrutiny. SONiC vendors must prove the provenance, integrity, and patch level of every component.

Bazel strengthens security by:

Locking down external dependencies
Ensuring reproducibility (critical for supply-chain audits)
Providing deterministic SBOM generation
Enforcing hermetic builds that eliminate environment drift
Integrating easily with tools for SLSA, sigstore, and in-toto

For organizations integrating SONiC into large-scale infrastructure, this reduces risk and simplifies compliance workflows.

5. Better Collaboration Between Vendors and the Community

One of SONiC's greatest strengths is its vendor-neutral ecosystem. Bazel reinforces this mission:

All vendors compile with the same rules and toolchains
Contributors can share Starlark extensions for SDKs or hardware targets
Build logic becomes collaborative, reviewable, and testable—just like code
Onboarding new vendors becomes faster and less error-prone

Instead of each party maintaining private build scripts, SONiC can establish a shared, open-standard build vocabulary.

6. Future-Proofing SONiC for the Next Decade

SONiC is evolving quickly—into new form factors, new silicon, new service models, and new deployment patterns. With Bazel, SONiC gains a foundation that:

Scales horizontally as code volume increases
Easily supports new languages or frameworks
Provides deterministic releases that downstream integrators can trust
Enables advanced workflows like distributed testing or reproducible builds in the cloud

Bazel is not a short-term patch; it is an investment in SONiC's long-term velocity and stability.

The Legacy Build System: Complexity at Scale

To understand why Bazel is necessary, it's worth examining what SONiC's build process looked like before: a Makefile-based system built around a "slave" container image that encapsulated system dependencies.

Arbitrary Commands, Arbitrary Problems

The legacy system consists of 317 Makefile rules (.mk files) and dependency files (.dep files), one for nearly every package, container, and component in SONiC. Each rule is allowed to invoke arbitrary commands: apt-get install, pip install, dget, shell scripts, custom build logic. On the surface, this sounds reasonable—the "slave" container provides isolation. But at SONiC's scale, this approach becomes a liability:

Brittleness: Each recipe fetches dependencies from the internet on-demand. A network hiccup, a removed package from Debian archives, a deprecated Python module on PyPI, and the entire build fails. There's no guarantee that tomorrow's build will work the same way as today's. No pinning, no snapshot repositories, no explicit version control of dependencies.

Non-Hermeticity: Because recipes are allowed to fetch and execute arbitrary commands, the build's output depends on what's available on the internet right now. Build two identical SONiC commits a week apart, and they may produce different binaries. The build machine's OS and installed packages are implicit dependencies—if you upgrade Ubuntu on your build server, you risk silently changing SONiC's outputs.

Unmaintainability at Scale: With 317 separate rule files, coordinating changes is nightmarish. A Debian package update might require changes to multiple recipes. Dependency injection during build (where rules can declare new dependencies at build time) means the full dependency graph is opaque until runtime. You don't know what's actually going to be built until you run make.

Hidden Complexity: The "slave" container approach obscures the real problem. Developers think they're building in a consistent environment because it's Docker-based, but the container itself is built by the same brittle, non-hermetic process. You're wrapping chaos in a box and calling it reproducibility.

Recipes Inject Dependencies During Build

Making matters worse, the legacy system allows rules to inject dependencies dynamically during the build process. A recipe isn't just a fixed input-output mapping; it's a script that can decide at runtime what to depend on. This makes the dependency graph fundamentally unknowable until build execution. You can't reason about what the artifact will contain—you have to run it and see.

The Scale Problem

SONiC spans dozens of C/C++ components, multiple Python packages, Go binaries, container images, kernel modules, and ASIC-specific toolchains. The legacy system required maintainers to write and maintain Makefiles for all of this, with no unifying framework. Each component has its own recipe, its own build logic, its own fragile internet-based dependency fetching. When SONiC grew to support ARM architectures, multiple Debian versions (Stretch, Buster, Bullseye, Bookworm), and dozens of ASIC vendors, the number of recipes multiplied.

The build system became a maintenance burden that grew faster than the codebase itself.

Bazel is decades ahead as a build system—but it’s not autopilot. You still need a good engineer to make it fit the problem.

Bazel is uniquely suited to address the intricate build and test challenges faced by the SONiC project. As SONiC evolves to support a growing range of devices, languages, and architectures, the need for a robust build system becomes paramount. Bazel’s hermetic, reproducible builds help ensure that the same inputs consistently yield the same outputs—an essential property in a diverse ecosystem. With deterministic dependency management and strong cross-compilation support, Bazel aligns well with SONiC’s goals of consistency across platforms and contributors.

But Bazel isn’t a silver bullet. It’s decades ahead as a build system, yet its architecture doesn’t solve the problem automagically—real success still depends on good engineering: clear rule boundaries, disciplined dependencies, and a toolchain/packaging strategy that makes Bazel’s guarantees real in practice.

The Deep Technical Challenge: Hermetic Package Dependency Resolution

Building a polyglot, multi-architecture system like SONiC using Bazel exposed a fundamental tension between how traditional Linux package managers work and how Bazel is designed to operate. The solution has required solving multiple interconnected problems that most build systems never encounter.

The sections below summarize some of the Bazel for SONiC issues that Aspect Build and has helped to identify and address.

The .deb Package Overlay Problem

Debian packages are designed to be installed sequentially into a shared filesystem. When you run apt-get install, each package unpacks its files into the same root directory—creating a virtual "overlay" of files from potentially hundreds of packages. Circular dependencies are resolved through this overlay: library A might contain a symlink to library B, which is provided by a completely different package. The order of installation handles conflicts, and the running system sees a unified merged filesystem.

Bazel, by contrast, abhors large, opaque target outputs. Each target should be hermetic and reproducible, with explicit dependencies and minimal side effects. Representing "unpack all these packages and overlay them" as a single Bazel target creates an enormous, unwieldy build artifact that defeats caching, incremental builds, and reproducibility. Yet SONiC needs precisely this—a complete, consistent sysroot with hundreds of interdependent Debian packages.

The solution: Bazel needed to compute the transitive closure of package dependencies, resolve symlinks ahead of time, and produce a metadata representation of the merged filesystem without materializing the entire overlay**.** Rather than creating a giant unpacked sysroot target, rules_distroless generates a "Contents" file—a snapshot mapping every filename to the package that provides it. This allows Bazel to:

Resolve symlink chains before build time (avoiding "dangling symlink" errors that plague container builds)
Determine the final location of each library or header file despite circular package dependencies
Share the metadata across builds without replicating hundreds of MB of unpacked packages
Enable fine-grained caching at the package level, not the sysroot level

The RPATH Nightmare

The ELF dynamic linker, ld.so, searches for shared libraries in a precise order:

Directories in the binary's DT_RPATH attribute (if DT_RUNPATH is absent)
LD_LIBRARY_PATH environment variable
Directories in DT_RUNPATH (the modern preferred approach)
System cache (/etc/ld.so.cache)
Default paths (/lib, /usr/lib, and architecture-specific variants)

By default, binaries built in SONiC expect their dependencies in /usr/lib, /lib, and architecture-specific variants like /usr/lib/aarch64-linux-gnu. But Bazel builds place outputs in bazel-out/, a completely different path structure. The linker has no idea where to find libyang.so.2.0 when it's buried in bazel-out/aarch64-linux-gnu/bin/external/com_github_sonic_net_sonic_mgmt_common/lib/libyang.so.2.0.

The fix requires embedding RPATH entries directly into the binary at link time—telling the linker "look in $ORIGIN/../lib for my dependencies." But this introduces new challenges:

Relocation: A binary linked with RPATH=$ORIGIN/../lib expects a specific directory structure. If that binary is later used as a tool in another Bazel action (where it's relocated to a different path), the RPATH becomes invalid.
Transitive Dependencies: When a binary depends on library A, which depends on library B, the linker must find all three—but each may have different RPATH settings. Bazel must ensure the transitive closure is visible to the linker without creating massive, monolithic targets.

SONiC solved this through Bazel Configurations and Transitions —a mechanism that applies different compiler flags to different parts of the dependency graph. Some targets (like Python's C extensions) are compiled with -fPIC and packaged into a sysroot. Everything else uses the standard compilation model. Bazel's transitions ensure these different compilation modes never mix, preventing linker errors where position-dependent code tries to reference position-independent code.

The RPATH Nightmare Continues: Finding Dependencies in the Sandbox

When Bazel runs a test target, it constructs a temporary sandbox directory containing only the files that test explicitly depends on. A test might link against a dynamic C++ library, which depends on libc.so.6, which transitively depends on ld-linux-x86-64.so.2. At test runtime, none of these libraries are in the system /usr/lib—they're scattered across bazel-out/ in the test's sandbox.

The breakthrough: Rather than relying on LD_LIBRARY_PATH (which is fragile and defeats reproducibility), rules_distroless ensures that when a test binary is linked, its RPATH contains the exact paths to all transitive dependencies within that test's sandbox. The binary becomes self-contained—it knows exactly where to find every library it needs, relative to its own location. A test can now be run anywhere, on any machine, and it will find its dependencies without environment variable manipulation. This is genuinely hermetic testing: the test's success or failure depends only on the code and its declared dependencies, not on what happens to be installed on the developer's machine.

Container Runtime: Binaries Finding Dependencies Across Layers

When a SONiC container starts, thousands of binaries are present—p4rt, swss, gnmi, monitoring agents, and hundreds of utilities. None of them are statically linked. Each binary must find its shared libraries at runtime.

In a traditional Docker/container image built with a Dockerfile, everything is installed into standard locations (/lib, /usr/lib). The dynamic linker searches these paths by default. Simple and naive.

But SONiC containers built with Bazel take a different approach. Multiple packages provide the same functionality (e.g., multiple versions of libprotoc.so), but only one should be "selected" for the final image. Furthermore, if SONiC switches from Debian bookworm to Debian trixie, or moves between Ubuntu LTS versions, the exact libraries available change. Hardcoding /usr/lib/x86_64-linux-gnu/libfoo.so.1 in a binary's RPATH would break when the library moves or changes versions.

The solution: Bazel generates a container image where the RPATH of every binary is already computed to find the exact libraries that will be present in that specific image. When p4rt starts in the container, its RPATH contains the paths where rules_distroless placed libprotoc.so, libyang.so, and everything else. The binary finds its dependencies through the RPATH, not through the system's default search paths. This means:

The container is self-describing: a binary's dependencies are "baked in" rather than discovered at runtime
Moving from Debian bookworm to trixie doesn't break existing binaries—Bazel recomputes the RPATH for the new distro
Libraries can be placed anywhere in the image without breaking binaries
Container images are reproducible: given the same Bazel build configuration, the same binaries will find the same libraries every time

Liberation from Host Distro Dependencies

Perhaps the most profound impact: rules_distroless decouples the build machine's operating system from the built artifact.

Traditionally, if you build SONiC on Ubuntu 24.04 and then try to run the binaries on Ubuntu 22.04, you hit subtle glibc incompatibilities. Some binaries depend on functions only available in Ubuntu 24's glibc version. Others link against the "wrong" version of OpenSSL. Developers spend weeks debugging "works on my machine" failures.

With rules_distroless, the build process explicitly pins every single Debian package using Debian snapshot repositories. When you build SONiC on an Ubuntu 22.04 machine, the build ignores the system's installed packages and fetches exact, pinned versions from the snapshots. The build output contains binaries linked against those exact packages, regardless of what's on the build machine.

This means:

A developer on Ubuntu 20.04 can build SONiC targeting Debian bookworm without installing bookworm libraries locally
CI/CD runners can be any modern Linux distro—the build is isolated from the host
Build results are reproducible across machines because dependency versions are explicit, not implicit
Upgrading the build machine's OS doesn't accidentally change SONiC's binaries

The build machine becomes nearly irrelevant. You're no longer asking "how do I build this on my machine?" You're asking "given a Bazel configuration and a snapshot of the Debian archive, what do I build?" The answer is the same everywhere, because Bazel controls everything.

Radical Hermeticity: Explicit Dependencies, Zero Implicit Assumptions

The commitment to hermeticity goes far beyond RPATH tuning and package pinning. SONiC's C/C++ compilation is built with compiler flags that reject the very concept of "system libraries":

-nostdinc -nostdinc++ -nostdlib

These flags tell the compiler: "There are no standard libraries. There are no default include paths. Everything must be explicitly declared." Without these flags, a compiler would silently fall back to the build machine's /usr/include and /usr/lib. A developer on Ubuntu 24.04 might accidentally use headers from Ubuntu 24's glibc, even if the target is Debian bookworm. With -nostdinc -nostdinc++ -nostdlib, that accident becomes impossible—the build fails loudly if a dependency is missing.

Every header file, every standard library function, every bit of C runtime must be explicitly provided as a Bazel dependency. This guarantees that:

A build on any machine produces identical binaries
Adding a dependency automatically updates the build configuration (nothing hidden)
Removing unused dependencies is impossible—they're explicitly declared
Switching between libc implementations or versions is straightforward—just change the declared dependency

Auto-Generated cc_library Targets: Turning .deb Packages into Bazel Dependencies

But declaring every libc header and every system library explicitly would require thousands of manual cc_library rules. That's where rules_distroless shines.

When rules_distroless processes a Debian package, it doesn't just extract files. It automatically generates Bazel cc_library targets that encapsulate:

All header files from the package (C standard library headers, C++ standard library headers, architecture-specific headers)
All shared object libraries
The correct include paths and linker settings
All transitive dependencies, automatically resolved

From a SONiC developer's perspective, instead of hoping that libc headers and libraries exist somewhere on the system, they explicitly declare:

cc_binary(
    name = "my_tool",
    srcs = ["main.cc"],
    deps = [
        "@debian//libc6",
        "@debian//libstdc++",
    ],
)

The Bazel build system now understands exactly which Debian packages this binary depends on. If you want to upgrade libc, you change one line in MODULE.bazel. If you want to use a different libc or switch distributions entirely, the build system tracks it explicitly.

This auto-generation is the key to polyrepo hermetic builds. In a monorepo, one team can write all the libc rules. But SONiC spans multiple GitHub repositories. Each repo's BUILD files can independently declare which Debian packages it needs. The rules_distroless machinery automatically generates compatible cc_library targets, and Bazel's module system ensures they all use the same versions across all repositories.

No more silent fallback to system libraries. No more "it works on my machine but not in CI." Every dependency is explicit, versionable, and traceable.

The Performance Paradox

Computing transitive closure of dependencies across a polyrepo, resolving symlink chains, and managing RPATH dynamically sounds expensive — and it is. But doing it at build time, once, and caching the result is far cheaper than doing it at container runtime or — worse — debugging "symbol not found" errors weeks later in production.

The trick is recognizing what can and cannot be cached:

Package metadata (which files each package provides, symlink targets) is stable and highly cacheable
Sysroot layout (which package provides which file when all dependencies are considered) is computed once and reused
Binary relocations (embedding RPATH in binaries) happens at link time and is reproducible
Container assembly (final layer selection) happens downstream and doesn't affect upstream caching

SONiC's Bazel infrastructure treats these as separate concerns, allowing massive parallelism and cache reuse even though the final build is complex.

Solving the Python Dependency Maze: Explicit, Versioned, Cross-Compilable

Python presents a unique challenge in reproducible builds. Unlike C/C++ where dependencies are system packages installed via a package manager, Python's ecosystem fetches from PyPI—a centralized repository where package availability and versioning can change. The standard approach to managing Python dependencies in Bazel has been rules_python, but it has limitations that made it unsuitable for SONiC's polyglot, multi-architecture environment.

The PyPI Problem at Scale

SONiC uses Python extensively: configuration generation tools, management daemons, testing frameworks, and utilities. Each Python package has transitive dependencies on other packages, many of which are compiled extensions (.so files). The challenge is that rules_python's traditional pip_parse implementation:

Assumes a single target architecture (no cross-compilation support)
Doesn't pin package versions deterministically
Struggles with compiled Python extensions that depend on system libraries
Doesn't integrate well with polyrepo ecosystems where multiple repositories have different Python dependency requirements

In SONiC's case, you might need to build the same Python package for x86-64 and ARM64 simultaneously. The standard tooling wasn't built for that.

Enter Aspect Rules Python: Modern Dependency Management

To solve these problems, Aspect Rules Python (aspect_rules_py) provides a modern reimplementation of Python dependency management in Bazel. It replaces the traditional pip_parse with a new implementation based on uv, a fast, production-grade Python package resolver.

Key improvements:

Explicit Versioning: Dependencies are pinned in a lock file (similar to requirements.lock), ensuring that every build uses the exact same package versions. No surprises from PyPI changes.

Cross-Compilation Support: Unlike the original rules_python, aspect_rules_py can build Python packages for different architectures simultaneously. This is critical for SONiC: the same Python source code can be compiled as a wheel for x86-64, then recompiled for ARM64, with all dependencies resolved correctly for each target.

Integration with Hermetic Sysroots: When a Python package has compiled extensions that depend on system libraries (e.g., a C extension that links against libyang), the sysroot provided by rules_distroless makes the correct headers and libraries available. aspect_rules_py integrates seamlessly with this sysroot, so the package's build automatically uses the right C compiler flags, header locations, and link paths for the target architecture.

Polyrepo-Friendly: Each SONiC repository declares and resolves its own Python dependency closure independently. aspect_rules_py uses uv to compute the complete transitive dependency graph for each repository, generating a lock file that pins all transitive dependencies. Because each repo has its own lock file, different repositories can use different versions of shared dependencies without conflict—Bazel's module system keeps them isolated.

Enabling Container Cross-Compilation

Combined with the Debian package management improvements (rules_distroless), aspect_rules_py enables a powerful capability: building complete container images for different architectures within a single Bazel build.

Before, cross-compiling a SONiC container for ARM64 required:

Running the build on an ARM64 machine (or emulating it, which was slow)
Managing different Python dependency versions for different architectures
Handling the mismatch between the build machine's Python environment and the target architecture

With aspect_rules_py and rules_distroless, Bazel can:

Resolve Python dependencies for the target architecture (e.g., ARM64)
Build wheels for that architecture using the sysroot's C compiler and libraries
Layer those wheels into the container image alongside the pinned Debian packages
All within a single Bazel invocation on any machine

This means a developer on an x86-64 laptop can type bazel build //path/to:docker-swss-arm64 and get a fully cross-compiled container image without any special setup, emulation, or native ARM64 hardware.

A Call to Action for the SONiC Foundation

The SONiC community is at an inflection point. As deployments reach hyperscale and the contributor base grows more diverse, the project needs a build and test system that matches its ambitions.

Bazel offers exactly that: a modern, reproducible, cloud-scale platform that empowers contributors, accelerates releases, and strengthens the entire ecosystem.

Adopting Bazel across SONiC Foundation projects will:

Reduce fragmentation
Increase developer productivity
Improve security and auditing
Enable faster and more reliable releases
Provide a consistent experience for every vendor and contributor

The benefits compound over time—and they're aligned with SONiC's vision of an open, interoperable, high-performance network OS. Let's give SONiC the build system it deserves.

Accelerating Innovation Through Reproducible, Scalable, Cloud-Native Build Systems

In this article, we have explored the case for adopting Bazel across SONiC Foundation projects and how it can meaningfully improve developer productivity, platform compatibility, security posture, and release engineering. We’ve also updated the Bazel and SONiC communities on some of our recent contributions to empower Bazel for SONiC.

As the SONiC Foundation continues to drive forward an open, standards-based network operating system for the industry, one challenge has become increasingly visible across the community: how to build, test, and release SONiC components with speed, consistency, and confidence. The project has grown in complexity—diverse hardware platforms, multiple languages and toolchains, distributed teams, and the rising expectation that networking software behave like modern cloud software.

This is exactly where Bazel, the open-source, high-performance build and test system originally created at Google, can play a transformative role. Bazel offers SONiC contributors and vendors the reliability, scalability, and repeatability needed to sustain a world-class network OS at global scale.

Next Steps

Interested in learning more about how to succeed with Bazel for SONiC? Schedule time to talk with us or email us at hello@aspect.build.