GitHub Actions Dynamic Matrix
We needed to configure GitHub actions to run a matrix of jobs, but the values weren't static. For example:
- one job requires a secret auth token, thus it can't be run on untrusted code from pull requests (for example, a private NPM registry is used, or Bazel's Remote Build Execution)
- we might want to read a line from a config file like
.bazelversion
and use that value in a matrix dimension
GitHub actions themselves don't document this at all. Maybe they should!
The answer here is fantastic, but really long: stackoverflow.com/questions/65384420/how-to..
It's also a bit out-of-date since GitHub is deprecating the syntax it uses: github.blog/changelog/2022-10-11-github-act..
And it uses a separate JSON file which felt like overkill for my case. Here's a quicker recipe:
Define one or more "matrix-prep" jobs
Each of these contributes the values needed by one dimension of the matrix. It just runs bash one-liners, then the results are aggregated into a JSON array. For example to make a value conditional on having some secret available:
jobs:
matrix-prep-config:
# Prepares the 'config' axis of the test matrix
runs-on: ubuntu-latest
env:
# Grab a secret from the GitHub environment, will be empty string if the secret isn't
# visible such as for untrusted code in a Pull Request
ENGFLOW_PRIVATE_KEY: ${{ secrets.ENGFLOW_PRIVATE_KEY }}
steps:
- id: local
run: echo "config=local" >> $GITHUB_OUTPUT
- id: rbe
run: echo "config=rbe" >> $GITHUB_OUTPUT
# Don't run RBE if there are no EngFlow creds which is the case on forks
if: ${{ env.ENGFLOW_PRIVATE_KEY != '' }}
outputs:
# Result will look like '["local", "rbe"]' if the secret was present or
# '["local"]' otherwise
configs: ${{ toJSON(steps.*.outputs.config) }}
or another example where we need to read a file from the repo to find the values:
jobs:
matrix-prep-bazelversion:
# Prepares the 'bazelversion' axis of the test matrix
runs-on: ubuntu-latest
steps:
# Need the repo checked out in order to read the file
- uses: actions/checkout@v3
- id: bazel_6
run: echo "bazelversion=$(head -n 1 .bazelversion)" >> $GITHUB_OUTPUT
- id: bazel_5
run: echo "bazelversion=5.3.2" >> $GITHUB_OUTPUT
outputs:
# Will look like '["6.0.0rc1", "5.3.2"]'
bazelversions: ${{ toJSON(steps.*.outputs.bazelversion) }}
Use that JSON value in the matrix definition
We'll use needs
to wait for the above jobs to complete and to read the values produced.
jobs:
[...]
test:
runs-on: ubuntu-latest
needs:
- matrix-prep-config
- matrix-prep-bazelversion
strategy:
matrix:
# Reads the value saved by "outputs" of the jobs above
config: ${{ fromJSON(needs.matrix-prep-config.outputs.configs) }}
bazelversion: ${{ fromJSON(needs.matrix-prep-bazelversion.outputs.bazelversions) }}
# Another dimension with static values
folder:
- "."
- "e2e/bzlmod"
- "e2e/copy_to_directory"
# Exclusions work like normal
exclude:
- config: rbe
bazelversion: 5.3.2
folder: e2e/bzlmod
Here's the full example: github.com/aspect-build/bazel-lib/blob/0c8e..