Skip to content
hyperspool

Build only what changed

In a monorepo most commits touch one or two packages, and building everything on every push wastes time. The set of packages to build is only known once the run has the commit in hand, so the run works it out first and then generates the jobs: a script: job diffs the commit, maps changed paths to packages, writes a pipeline with one job per affected package, and runs it with spool call.

The examples assume a repo connected with spool repo connect (CI from a repo), which starts .spool/ci.yaml with the pushed commit as input.sha. To skip a build entirely when a push only touches docs, filter with --when on the connection instead (Branch routing).

Any repo: diff, map, generate

Section titled “Any repo: diff, map, generate”

This repo keeps services under services/<name>/ and shared code under libs/. A change to a service tests that service; a change under libs/ tests all of them. Each service has a test.sh.

name: ci


env:
  REPO_URL: "https://github.com/${{ has(input.repo) ? input.repo : 'acme/mono' }}.git"
  SHA: "${{ has(input.sha) ? input.sha : '' }}"
  BASE_REF: "${{ has(input.base_ref) ? input.base_ref : '' }}"


jobs:
  affected:
    script: |
      #!/usr/bin/env bash
      set -eu


      # Clone, diff and map paths to packages in one recorded step, so a
      # replay of this script reuses the answer instead of cloning again.
      plan="$(spool step affected -- bash -c '
        set -eu
        repo="$1" sha="$2" base_ref="$3"
        cd "$(mktemp -d)"
        git clone --quiet "$repo" .
        if [ -n "$sha" ]; then git checkout --quiet "$sha"; fi
        if [ -n "$base_ref" ]; then
          git fetch --quiet origin "$base_ref"
          base="$(git merge-base FETCH_HEAD HEAD)"
        else
          base="HEAD^"
        fi
        changed="$(git diff --name-only "$base" HEAD)"
        if grep -q "^libs/" <<<"$changed"; then
          pkgs="$(ls services)"
        else
          pkgs="$(grep "^services/" <<<"$changed" | cut -d/ -f2 | sort -u || true)"
        fi
        echo "$(git rev-parse HEAD)" $pkgs
      ' _ "$REPO_URL" "$SHA" "$BASE_REF")"
      read -r sha pkgs <<<"$plan"


      if [ -z "$pkgs" ]; then
        echo "no package changed"
        exit 0
      fi


      # One job per affected package. Jobs with no depends_on run in parallel.
      {
        echo "name: ci-affected"
        echo "jobs:"
        for p in $pkgs; do
          echo "  test_$p:"
          echo "    steps:"
          echo "      - name: test"
          echo "        run: |"
          echo "          git clone --quiet $REPO_URL src && cd src"
          echo "          git checkout --quiet $sha"
          echo "          services/$p/test.sh"
        done
      } > affected.yaml


      spool call affected.yaml
      spool output set packages="$pkgs"

For a commit that changed services/api/main.go, services/web/index.html and README.md, the job writes and runs:

name: ci-affected
jobs:
  test_api:
    steps:
      - name: test
        run: |
          git clone --quiet https://github.com/acme/mono.git src && cd src
          git checkout --quiet c77e6857cc096deb8d3cbe7627ab2e8a7f89328d
          services/api/test.sh
  test_web:
    steps:
      - name: test
        run: |
          git clone --quiet https://github.com/acme/mono.git src && cd src
          git checkout --quiet c77e6857cc096deb8d3cbe7627ab2e8a7f89328d
          services/web/test.sh

How it fits together:

  • spool call needs a script: job. A steps: job runs once and can’t wait on a child, so spool call refuses to run there.
  • The diff and the mapping sit inside one spool step, so the script records the answer once. A step body runs in a sealed environment without the job’s env:, which is why the repo, commit and base ref go in as arguments (spool step).
  • The base is the parent commit for a push, and the merge base with the target branch for a pull request (input.base_ref, set when the repo is connected with --pull-requests). A push of several commits is compared against its last commit’s parent only; to cover all of them, diff a branch against origin/main instead.
  • The generated jobs pin the commit the plan was made from, so a push that lands while the run is going doesn’t change what gets tested.
  • Generated jobs have no depends_on, so they run in parallel. If one fails, the child run fails, spool call exits non-zero and the affected job fails with the child’s error.
  • A commit that touches no package ends the run green with “no package changed”.

For a private repo, give the parent and each generated job a credentials: block with the clone token, such as GH_TOKEN: github-app:acme/mono for a repo connected through the GitHub App, and clone from https://x-access-token:${GH_TOKEN}@github.com/acme/mono.git.

The same thing as a graph

Section titled “The same thing as a graph”

In a graph, a code step can produce the list and a fan_out runs one shell step per package. The fan-out stays inside the run, so there is no separate child run to open:

version: hyperspool-graph/v1
name: ci
vars:
  REPO_URL: https://github.com/acme/mono.git
nodes:
  - id: affected
    type: code
    config:
      env:
        REPO_URL: { $expr: vars.REPO_URL }
        SHA: { $expr: "has(input.sha) ? input.sha : ''" }
      script: |
        set -eu
        cd "$(mktemp -d)"
        git clone --quiet "$REPO_URL" .
        if [ -n "$SHA" ]; then git checkout --quiet "$SHA"; fi
        changed="$(git diff --name-only HEAD^ HEAD)"
        if grep -q '^libs/' <<<"$changed"; then
          pkgs="$(ls services)"
        else
          pkgs="$(grep '^services/' <<<"$changed" | cut -d/ -f2 | sort -u || true)"
        fi
        list=""
        for p in $pkgs; do list+="\"$p\","; done
        spool output set --values "{\"sha\": \"$(git rev-parse HEAD)\", \"packages\": [${list%,}]}"
  - id: test
    type: fan_out
    config:
      items: { $expr: nodes.affected.output.packages }
      max_parallel: 8
      body:
        type: shell
        config:
          run:
            $template: |
              git clone --quiet {{ vars.REPO_URL }} src && cd src
              git checkout --quiet {{ nodes.affected.output.sha }}
              services/{{ item }}/test.sh
edges:
  - { from: affected, to: test }

A failing package fails the test step once the packages in flight settle (on_error: halt, the default). An empty list runs nothing and succeeds. A fan_out takes at most 100 items, and a computed list longer than that fails the step, so use this shape when the package count stays well under it. Use the generated pipeline when packages need different jobs, or when one job per package should depend on another.

Bazel: query the affected targets

Section titled “Bazel: query the affected targets”

Bazel already knows the dependency graph, so the plan asks it. rdeps(//..., set(<files>)) finds every target that depends on a changed file, and kind(".*_test", …) keeps the tests. The targets then go out in batches, one job per batch, since one bazel test over fifty targets shares analysis and cache lookups that fifty separate invocations would repeat.

This one is a sketch. The Hyperspool side (the recorded query step, the batching, the generated pipeline and spool call) runs as shown, but check the bazel query lines against your own workspace.

name: ci


env:
  REPO_URL: "https://github.com/${{ has(input.repo) ? input.repo : 'acme/mono' }}.git"
  SHA: "${{ has(input.sha) ? input.sha : '' }}"


jobs:
  affected:
    script: |
      #!/usr/bin/env bash
      set -eu


      # First line: the commit. Then one test target per line.
      plan="$(spool step query -- bash -c '
        set -eu
        cd "$(mktemp -d)"
        git clone --quiet "$1" .
        if [ -n "$2" ]; then git checkout --quiet "$2"; fi
        git rev-parse HEAD
        files="$(git diff --name-only --diff-filter=d HEAD^ HEAD | tr "\n" " ")"
        if grep -qE "(^| )(MODULE\.bazel|WORKSPACE|\.bazelrc|\.bazelversion) " <<<"$files "; then
          bazel query "kind(\".*_test\", //...)"
        elif [ -n "$files" ]; then
          # --keep_going skips files no package owns; exit 3 means partial results.
          bazel query --keep_going "kind(\".*_test\", rdeps(//..., set($files)))" \
            || [ $? -eq 3 ]
        fi
      ' _ "$REPO_URL" "$SHA")"
      sha="$(head -n 1 <<<"$plan")"
      targets="$(tail -n +2 <<<"$plan")"


      if [ -z "$targets" ]; then
        echo "no test target affected"
        exit 0
      fi


      # Fifty targets per job.
      {
        echo "name: ci-affected"
        echo "jobs:"
        n=0
        while read -r batch; do
          n=$((n + 1))
          echo "  batch_$n:"
          echo "    steps:"
          echo "      - name: test"
          echo "        run: |"
          echo "          git clone --quiet $REPO_URL src && cd src"
          echo "          git checkout --quiet $sha"
          echo "          bazel test $batch"
        done < <(xargs -n 50 <<<"$targets")
      } > affected.yaml


      spool call affected.yaml

The query step and every batch job need bazel (or bazelisk) on the agent that runs them. The managed containers don’t have it: install it on a build VM with requires: [build], or run your own agent with Bazel installed and a capability of your own naming, and add that to requires: on the affected job and on each generated job. Every job starts from a fresh checkout, so a remote cache is what keeps the batches from rebuilding the same dependencies.

Pick the batch size from your test times. Fewer, larger batches spend less on checkout and analysis; more batches finish sooner when there are agents to run them.

On the run page and the commit

Section titled “On the run page and the commit”

The pipeline run shows one step, affected, with the planning step’s log and the spool call waiting on the child. The generated ci-affected run has a run page of its own with one job per package, and links back to the run that started it. spool run list shows the parent’s id in its last column:

dev:default:run-92cbedef4c7a8348  ci-affected  failed  pipeline  0s ago  call-parent:…  dev:default:run-FmisKpLAX6jbFRd1
dev:default:run-FmisKpLAX6jbFRd1  ci           failed  pipeline  0s ago  kind:pipeline  -

The commit status reports the pipeline run, under hyperspool/ci for .spool/ci.yaml, and links to its run page. The generated run gets no status of its own: when a package fails, the parent fails with that job’s error and the status says failure. To see which package broke, open the child from there.

Limits

Section titled “Limits”
  • The generated work is a separate run. It has its own page, log and rerun, and the parent can only wait on the whole of it: no step in the parent can start after one generated job and before the rest.
  • A child started with spool call takes one of your plan’s concurrent run slots while it runs. It never waits for one, since the parent is waiting on it, but it can hold up the next push’s run.
  • fan_out and matrix: stop at 100 items. A generated pipeline or graph can have up to 512 steps, and one with more is refused before its run starts. Batch the work anyway when the list can grow into the hundreds, so each job does a useful amount of it.
  • Rerunning the parent plans again and tests every affected package. To retry one flaky package, rerun the child from that job (spool run rerun <child> --from test_web, or Rerun from here on its page), which leaves the jobs that passed alone. The commit status still reports the original parent run; push again to update it.