GitHub Actions Dynamic Matrix

We needed to configure GitHub actions to run a matrix of jobs, but the values weren't static. For example:

  • one job requires a secret auth token, thus it can't be run on untrusted code from pull requests (for example, a private NPM registry is used, or Bazel's Remote Build Execution)
  • we might want to read a line from a config file like .bazelversion and use that value in a matrix dimension

GitHub actions themselves don't document this at all. Maybe they should!

The answer here is fantastic, but really long:

It's also a bit out-of-date since GitHub is deprecating the syntax it uses:

And it uses a separate JSON file which felt like overkill for my case. Here's a quicker recipe:

Define one or more "matrix-prep" jobs

Each of these contributes the values needed by one dimension of the matrix. It just runs bash one-liners, then the results are aggregated into a JSON array. For example to make a value conditional on having some secret available:

    # Prepares the 'config' axis of the test matrix
    runs-on: ubuntu-latest
      # Grab a secret from the GitHub environment, will be empty string if the secret isn't
      # visible such as for untrusted code in a Pull Request
      - id: local
        run: echo "config=local" >> $GITHUB_OUTPUT
      - id: rbe
        run: echo "config=rbe" >> $GITHUB_OUTPUT
        # Don't run RBE if there are no EngFlow creds which is the case on forks
        if: ${{ env.ENGFLOW_PRIVATE_KEY != '' }}
      # Result will look like '["local", "rbe"]' if the secret was present or
      # '["local"]' otherwise
      configs: ${{ toJSON(steps.*.outputs.config) }}

or another example where we need to read a file from the repo to find the values:

    # Prepares the 'bazelversion' axis of the test matrix
    runs-on: ubuntu-latest
      # Need the repo checked out in order to read the file
      - uses: actions/checkout@v3
      - id: bazel_6
        run: echo "bazelversion=$(head -n 1 .bazelversion)" >> $GITHUB_OUTPUT
      - id: bazel_5
        run: echo "bazelversion=5.3.2" >> $GITHUB_OUTPUT
      # Will look like '["6.0.0rc1", "5.3.2"]'
      bazelversions: ${{ toJSON(steps.*.outputs.bazelversion) }}

Use that JSON value in the matrix definition

We'll use needs to wait for the above jobs to complete and to read the values produced.

    runs-on: ubuntu-latest

      - matrix-prep-config
      - matrix-prep-bazelversion

        # Reads the value saved by "outputs" of the jobs above
        config: ${{ fromJSON(needs.matrix-prep-config.outputs.configs) }}
        bazelversion: ${{ fromJSON(needs.matrix-prep-bazelversion.outputs.bazelversions) }}

        # Another dimension with static values
          - "."
          - "e2e/bzlmod"
          - "e2e/copy_to_directory"

        # Exclusions work like normal
          - config: rbe
            bazelversion: 5.3.2
            folder: e2e/bzlmod

Here's the full example: