The authoring model

How durable workflows run, and the rules that follow from it. Read it before writing anything longer than a demo.

Checkpoint and re-run

A workflow script is a normal program that the engine can run more than once. Each durable call - step, sleep, wait signal, wait event, lock, spawn, event emit and the rest (full list) - is a checkpoint: its result is recorded in the run’s event log the first time it completes. When the run resumes (after a crash, a sleep, a signal, an agent restart), the script executes again from the top. Durable calls already in the log return their results instantly without re-running; execution races through the recorded prefix and picks up where it left off.

The engine replays recorded results; it never re-executes recorded calls. Your code doesn’t need to be deterministic in any formal sense.

Two consequences shape everything else:

• Between checkpoints, your code runs again on every pass. A log line between two steps prints on every resume. A counter incremented between steps counts passes, not events.
• The suspension points are real process exits. When a sleep or a wait can’t return yet, the script process is terminated and the agent is freed - a run asleep for a month costs nothing. The call you wrote never returns on that pass; it returns on a later pass, from the log.

The rules

Wrap side effects and nondeterministic reads in a step. Anything that touches the outside world (deploys, API calls, file writes) and anything a later decision depends on that could change between passes - time, randomness, environment lookups, network reads - belongs inside a step. The recorded result is then the same on every pass, so decisions made from it are stable. A resumed run goes to whichever agent is free and the recorded result travels with it, so a step’s value holds wherever the resume lands and however long it took.

Make steps idempotent where you can. A step that dies after acting but before its result is recorded will run again. “Create or update”, “push if absent”, version-stamped writes - the usual idempotency toolkit applies.

When the remote system dedupes for you - most payment, messaging and ticketing APIs take an idempotency key or a transaction id - hand it a token that stays the same across every way one call can happen twice. On a graph node that’s node.idempotency_key:

{ "id": "charge", "type": "http",
  "config": { "method": "POST", "url": "https://api.example.com/charges",
              "headers": { "Idempotency-Key": { "$expr": "node.idempotency_key" } } } }

It stays the same across a retry:, a replay after a suspend, and a re-dispatch after an agent dies, since those are all the same call site of the same run. Inside a fan_out it also carries the item’s index, so ten notifications send ten keys. A rerun gets a new token, which is right: a rerun is a deliberate new attempt.

In a script, the same token reaches a spool step body as $SPOOL_IDEMPOTENCY_KEY, scoped to that step of that job of that run (how to pass it). Don’t use $SPOOL_ACTIVITY_ID instead: it is one constant for the whole job, so three charge steps in one job would send one key, the provider would dedupe them into the first charge, and nothing would tell you.

A YAML steps: job has no token of its own - its steps are one atomic script with no durable boundaries between them, so the whole job re-runs together. Put a call that must not double up behind a spool step in a script: job, or on a graph node.

For “has anyone already handled this record?” (the same webhook delivered twice, two schedules picking up one row), take a claim. See Durable state.

Keep between-step code cheap and repeatable. It re-runs on every pass. Pure formatting, branching on recorded results, and building argument lists are all fine; sending an email is not. Branch on values you captured in a step, not on the wall clock or fresh randomness. If a replay takes a different path and reaches a different durable call than the first pass recorded at that position, the run fails with a determinism error. When replay goes wrong walks through one, how to find the branch that caused it, and the lenient mode an agent you run can use to switch the check off.

Names are identity where something refers back to them. These are unique per run, and a second use is rejected rather than silently returning the first call’s result:

• activity submit <name>, because activity wait <name> retrieves that child’s result. Two submits under one name would make the second uncollectable.
• lock <group> with no command, because the lease is held to the end of the run, so there is only one per run.
• state cas / state incr, keyed by --op <name> (defaulting to the cell), because each attempt in a read-modify-write loop is its own question.

Everything else repeats. A step or step_fn name is a place in the script, so a loop reaches one name as many times as it iterates and each iteration gets its own checkpoint; pass --label if you want the run view to say which host or which attempt an iteration was for. A wait signal or wait event name is a queue, so waiting on one twice takes the first signal and then the second. And the primitives that carry no name at all - sleep, event emit, spawn, call - key off call order, so two sleeps are two waits.

All of them need the same call reached in the same order on every pass; replay is built on that. A run a handler started from an event replays under the same rules as one you submit by hand.

Mind result sizes. Step results are stored in the event log. Record a path, an id or a summary rather than a build artifact.

Code changes and what goes unchecked

Code changes don’t strand in-flight runs. Each run replays against the definition it started with and finishes on it, while new runs pick up the change. There is no migration step or version pinning to manage.

The engine catches only some rule violations. A replay that reaches a different durable call at a recorded position fails loudly. A side effect outside a step, or a divergence that rejoins the recorded path before the next durable call, goes unnoticed and produces a run that is plausible but wrong (what the engine does not catch).

Where code runs

A script runs on an agent that advertises polyglot, under the interpreter its file extension picks (interpreters). That interpreter has to exist where the script runs: managed environments lists what Hyperspool’s own environments carry, and an agent you run needs whatever you install on it. A script that imports an SDK also needs the SDK importable; workflows in your language says which SDKs you can get.