Durable patterns

Short recipes for things durable workflows are often asked to do: undo work when a later step fails, wait for a deadline, ask a person, handle each event once. Each one is a complete file you can run, followed by what happens when something fails or the run replays, and a link to the reference.

The recipes are bash scripts that call spool commands, plus one pipeline. echo stands in for the real work (and a placeholder URL in the retry recipe), so swap in your own commands. Run a script on its own with spool run <file>, or on spool dev with spool submit <file> when the recipe needs a signal, an event or several runs. Scripts run under bash -eu. A few read the event that started the run with python3, since the managed environments have python3 and no jq.

If you haven’t read the authoring model yet, read it first: every recipe here relies on checkpoints and replay.

Saga: undo what succeeded when a later step fails

When you want to book a flight, a hotel and a car, and cancel whatever was booked if a later booking fails:

booked=()   # booking ids, in the order they were made

rollback() {
  for (( i=${#booked[@]}-1; i>=0; i-- )); do
    spool step cancel --label "${booked[i]}" -- echo "cancelled ${booked[i]}"
  done
  exit 1
}

flight="$(spool step book-flight -- echo FL-881)" || rollback
booked+=("$flight")

hotel="$(spool step book-hotel -- echo HT-2207)" || rollback
booked+=("$hotel")

car="$(spool step book-car -- sh -c 'echo "no cars left" >&2; exit 1')" || rollback
booked+=("$car")

spool step confirm -- echo "booked $flight, $hotel and $car"

The car booking fails, so the hotel and then the flight are cancelled, newest first, and the run ends failed:

$ spool run saga.sh
  run: dev:default:run-z7pn2N97Vw718Ixi
no cars left
cancelled HT-2207
cancelled FL-881
  dev:default:run-z7pn2N97Vw718Ixi failed: script exited 1

On replay, booked is rebuilt from the recorded booking steps, so a pass that resumes after a crash or a wait cancels the same bookings without booking anything again. Each cancel is a step of its own, so one that already ran is not repeated. A cancel that fails ends the run with the older bookings still in place, so make cancels retry, inside the step (curl --retry) or with spool activity run --attempts.

Reference: spool step.

A deadline that an event can cut short

When you want a trial that ends in 14 days unless the customer upgrades first. A handler starts one run per trial.started event, and the run waits for plan.upgraded with a timeout:

customer="$(python3 -c 'import json, os; print(json.load(open(os.environ["SPOOL_INPUT_FILE"]))["payload"]["customer"])')"
spool step welcome -- echo "trial started for $customer"

if spool wait event plan.upgraded --timeout 14d > /dev/null; then
  spool step thank -- echo "$customer upgraded"
else
  spool step downgrade -- echo "trial over for $customer"
fi

spool handler set trials ./trial.sh --on trial.started
spool event emit trial.started --key cust-1 --payload '{"customer":"cust-1"}'
spool event emit trial.started --key cust-2 --payload '{"customer":"cust-2"}'

# later, when cust-1 pays
spool event emit plan.upgraded --key cust-1

The --key is the event’s subject. A run a handler starts takes the subject of the event that started it, and its spool wait event then only accepts events with the same subject (or with none). So the upgrade for cust-1 ends that run’s wait and leaves cust-2 waiting, until its 14 days run out and it takes the downgrade branch. To try it, change 14d to 30s. A wait that times out also prints a line saying so to stderr, which shows in the run’s log.

The run holds no agent while it waits. The outcome is recorded the first time the wait resolves, so an upgrade that lands just after the timeout doesn’t change a branch already taken. An event that fires before the run reaches the wait is not seen; for a value that has to be kept until the run asks for it, send a signal instead (spool wait signal, which queues).

Reference: spool wait event, subjects.

Approval with a timeout and escalation

When you want a person to approve something, and to ask someone else when the first person doesn’t answer in time. An approval job gives the person a form and Approve and Reject buttons; its timeout: lets the pipeline go on without an answer:

name: refund
jobs:
  ask:
    approval:
      prompt: "Refund order 1042, EUR 480?"
      timeout: 2d
  escalate:
    depends_on: [ask]
    if: ${{ steps.ask.output.timed_out }}
    steps:
      - name: page-lead
        run: echo "no answer on the refund for order 1042, asking the team lead"
  ask_lead:
    depends_on: [escalate]
    if: ${{ steps.ask.output.timed_out }}
    approval:
      prompt: "Refund order 1042, EUR 480? The first reviewer didn't answer."
      timeout: 1d
  refund:
    depends_on: [ask_lead]
    if: ${{ !steps.ask.output.timed_out || !steps.ask_lead.output.timed_out }}
    steps:
      - name: refund
        run: echo "refunded order 1042"
  give_up:
    depends_on: [ask_lead]
    if: ${{ steps.ask.output.timed_out && steps.ask_lead.output.timed_out }}
    steps:
      - name: close
        run: echo "nobody answered, refund for order 1042 left open"

Answer from the run page, or from a terminal with spool approve <run> ask (or ask_lead). The three outcomes:

What happened	Jobs that ran
`ask` approved	`ask`, `refund`
`ask` timed out, `ask_lead` approved	`ask`, `escalate`, `ask_lead`, `refund`
both timed out	`ask`, `escalate`, `ask_lead`, `give_up`

A gate that times out succeeds with timed_out: true, which is why every job below it checks the flag. Rejecting either gate cancels the run. The run holds no agent while a gate waits, and Approvals in Slack and Mattermost puts the buttons in a channel. Job names in expressions use underscores, since steps.ask-lead would read as a subtraction.

In a script, the same shape is two bounded waits on a signal: if ! spool wait signal approve --timeout 2d; then …; fi. A signal has no form and no reject button.

Reference: approval:, bounded waits.

Handle each event once

When you want an event handler that does its work once per order, even if the same order arrives twice:

order="$(python3 -c 'import json, os; print(json.load(open(os.environ["SPOOL_INPUT_FILE"]))["payload"]["order"])')"

rc=0
spool state add "order-$order" --entity orders || rc=$?
case $rc in
  0) ;;
  3) echo "order $order already handled"; exit 0 ;;
  *) exit 1 ;;   # couldn't reach the server: fail, don't skip
esac

spool step charge -- sh -c 'echo "charged order $1, key $SPOOL_IDEMPOTENCY_KEY"' _ "$order"
spool step ship -- echo "shipped order $order"

spool handler set orders ./on-order.sh --on order.placed
spool event emit order.placed --payload '{"order":1042}'
spool event emit order.placed --payload '{"order":1042}'

Two events, two runs. The first claims order-1042 and does the work; the second finds the claim taken:

charged order 1042, key spool-4bd07bd392872025b5be2041abc64281
shipped order 1042

order 1042 already handled

Three layers of deduplication are at work here, each for a different duplicate:

The same event delivered twice resolves to the same run, because a handler run’s id comes from the event id and the handler name.
Two different events about the same order are caught by spool state add, which claims a key in an entity-scoped cell once for the whole project. Its answer is recorded, so a replay of the winning run still sees that it won.
A step that dies after calling the payment API but before its result is recorded runs again. $SPOOL_IDEMPOTENCY_KEY is the same on that second attempt, so an API that dedupes on it charges once.

When you submit runs yourself, spool submit --idempotency-key <key> answers a repeat submit with the first run instead of starting another.

Reference: claiming a key, spool step, delivery guarantees.

Fan out over a list and collect the results

When you want to run the same command for every item in a list, side by side, and use all the results:

regions="eu-north us-east ap-south"

# Submit every item before waiting on any, so they run side by side.
for region in $regions; do
  spool activity submit "probe-$region" --requires shell --attempts 3 --backoff 5s \
    -- sh -c 'echo "$1: $((RANDOM % 50 + 10))ms"' _ "$region"
done

for region in $regions; do
  spool activity wait "probe-$region"
done

eu-north: 29ms
us-east: 50ms
ap-south: 53ms

Each activity submit queues one command for any agent with the capabilities it asks for, and each activity wait prints that item’s output once it’s done. The script waits holding no agent. Results come back in the order you wait for them, whatever order the items finished in. A failed item makes its activity wait exit non-zero, which ends the run under errexit; catch it to carry on, or use spool activity wait --all when you only need to know whether any failed. Names have to be unique within the run, which is why the region is in each one.

Which tool to pick:

The work per item is	Use
one command, from a script	`spool activity submit` and `activity wait`, as above
one step, in a graph	a `fan_out` step, with `max_parallel` and `rate`
several steps, each item with its own run page	a `fan_out` with a `workflow` body, which starts one child run per item

spool call waits for each child before the next line runs, so calling in a loop runs the items one after another. spool spawn starts a child and moves on without its result. fan_out takes at most a hundred items; activity submit has no such cap.

Reference: spool activity submit and wait, repeating work.

One at a time, and no faster than a rate

Three tools, depending on what you’re protecting.

One deploy at a time

When you want at most one run of a workflow at a time, with later ones waiting their turn:

name: deploy-prod
concurrency:
  group: deploy-prod
  policy: queue
jobs:
  deploy:
    steps:
      - name: deploy
        run: echo "deploy started $(date +%T)"; sleep 4; echo "deploy done $(date +%T)"

Submit it twice and the second run shows as queued until the first finishes. policy: cancel-running lets the newest run replace the one in progress instead, and skip drops a run that arrives while the group is busy. The group can be computed from the input, such as one group per branch. A group admits one run at a time; there is no setting for “at most three”.

A rate limit

When you want to cap how often runs start, such as calls to an API that allows six a minute per tenant:

name: enrich
throttle:
  limit: 6
  period: 1m
  key: 'crm-${{ input.tenant }}'
jobs:
  call:
    steps:
      - name: call
        run: echo "called the CRM for ${{ input.tenant }} at $(date +%T)"

Runs over the rate wait as queued and start evenly spaced, one every ten seconds here, and each tenant has its own budget. Three runs for acme submitted together, then one for globex:

called the CRM for acme at 21:41:55
called the CRM for acme at 21:42:05
called the CRM for acme at 21:42:15
called the CRM for globex at 21:41:55

burst: lets that many start together after a quiet spell, and policy: drop refuses runs over the rate instead of queueing them.

A script has no document, so it takes the same limits on submit: spool submit enrich.sh --throttle 6/1m --throttle-key crm-acme.

A lock per customer

When you want only one run at a time to work on a given customer, while runs for different customers go ahead in parallel:

customer=acme
spool step fetch -- echo "fetched for $customer"
spool lock "customer-$customer" -- sh -c 'echo "start $(date +%T)"; sleep 3; echo "end $(date +%T)"'

Three of these submitted together run their locked sections one after another, in arrival order. The work outside the lock doesn’t wait:

start 21:34:19
end 21:34:22
start 21:34:22
end 21:34:25
start 21:34:25
end 21:34:28

A run waiting for the lock holds no agent. The form with a command releases the lock when the command finishes; spool lock <group> with no command holds it until the run ends, which is what you want when the locked section is itself spool calls such as a state read and write. A pipeline job’s concurrency: takes the same lock.

Reference: run admission, spool lock.

Debounce a burst of events

When you want five updates in two seconds to cause one run of the work, on the last of them:

customer="$(python3 -c 'import json, os; print(json.load(open(os.environ["SPOOL_INPUT_FILE"]))["payload"]["customer"])')"

# incr is recorded, so this run keeps its ticket across the sleep.
mine="$(spool state incr "ticket:$customer" --entity reindex --op ticket)"

spool sleep 5s   # the quiet window

# A live read: it sees tickets drawn by later arrivals.
if [ "$mine" = "$(spool state get "ticket:$customer" --entity reindex)" ]; then
  spool step reindex -- echo "reindexing $customer (ticket $mine)"
else
  echo "ticket $mine superseded"
fi

spool handler set reindex ./reindex.sh --on customer.updated
for i in 1 2 3 4 5; do
  spool event emit customer.updated --payload '{"customer":"acme"}'
  sleep 0.5
done

Each event starts a run that draws a ticket and sleeps. Only the run holding the highest ticket when it wakes does the work:

reindexing acme (ticket 5)
ticket 4 superseded
ticket 3 superseded
ticket 2 superseded
ticket 1 superseded

The ticket has to come from spool state incr. Its answer is recorded, so the pass after the sleep holds the number it drew before it. A marker written with spool state set would be written again on that pass, and every run would think it was last. Durable state explains why and has the canvas form. To collect a burst and handle all of it at once instead, see the batch example.

Poll an outside system on a schedule

When you want to pick up new records from an API every few minutes, continuing from where the last check stopped:

since="$(spool state get cursor --entity tickets)"

# Stands in for "GET /tickets?since=$since"; prints the new cursor.
latest="$(spool step fetch -- sh -c 'echo $(( ${1:-0} + 10 ))' _ "$since")"

spool step process -- echo "processed tickets ${since:-0}..$latest"
spool state set cursor "$latest" --entity tickets

spool schedule create --every 5m --name ticket-poll poll.sh

Each tick is a fresh run that reads the cursor the last one wrote. Three ticks:

processed tickets 0..10
processed tickets 10..20
processed tickets 20..30

The cursor lives in an entity-scoped cell, which outlives every run; a step’s recorded result belongs to one run and can’t carry it to the next. The first tick reads an empty cell, which ${since:-0} covers. The fetch is a step, so a pass that replays after a crash doesn’t fetch again. If a tick fails before the last line, the cursor doesn’t move and the next tick fetches the same range again, so make the processing safe to repeat. By default a tick that arrives while the previous one is still running is skipped, so two polls never overlap.

Reference: the poll cursor, schedules.

Retry a flaky call with backoff

When you want a call that sometimes fails to be tried again, with a growing wait between attempts:

rates="$(spool activity run fetch-rates --requires shell \
  --attempts 5 --backoff 2s --retry-on-exit 75 \
  -- sh -c 'curl -fsS https://api.example.com/rates || exit 75')"
spool step store -- echo "stored $rates"

--attempts counts the first try, and the wait doubles after each failure: 2s, 4s, 8s, 16s. With a command that failed twice and then answered, the attempts ran at 21:43:24, 21:43:26 and 21:43:30. The script waits holding no agent, and only the final result is recorded. --retry-on-exit 75 retries that exit code only, so a command that decides its input is wrong (exit 64, say) fails on the first attempt.

The other places a retry can go:

Retry	Where	What runs again
`curl --retry 5` inside a `spool step`	the command	the command, before anything is recorded
`spool activity run --attempts`	the script	that one command
`retry:` on a pipeline job	the job	the whole job; for a `script:` job, every step in it, since recorded steps are cleared between attempts

A spool step on its own doesn’t retry. A failed step records the failure, and a replay returns it.

Reference: spool activity run, retry:, failure and retries.

An agent loop that waits for input

When you want a loop that calls a model, acts on the reply, and stops for a person when the reply needs one, see Agent loops. Each turn is a checkpoint, and a wait for a person holds no agent.