Architecture

Hyperspool runs your workflows so they survive restarts and can wait for days. This page describes how: the execution model, where work runs, and the durability and security properties you can rely on. The same engine runs YAML pipelines, polyglot scripts, node graphs authored on the canvas, schedules, webhook-driven CI, and the console. For the authoring side of the execution model, see the authoring model.

The execution model

A run is an append-only journal

Every run’s history is a sequence of events: started, activity scheduled/completed, timer scheduled/fired, signal received/consumed, child scheduled/completed, terminal. There is no state snapshot. Status, the run page in the console, and states like “asleep until X” are all derived by folding over the log.

Recovery replays the program

On resume the workflow body runs again from the top; every durable call whose result is already in the log returns the recorded value without executing. Locals come back because the code re-runs. The journal memoizes completed effects and pins values that must not change across replays: timer deadlines, signal payloads, child outputs.

Durable calls (steps, activities, sleeps, waits, spawns, emits and the rest; the full list) are the only durability boundary. Code between two durable calls re-runs on every replay. annotate is the one deliberate exception: a side channel with no durable boundary.

How replay matches calls

In a graph, a durable call is identified by its node. In a script, durable calls are checked by their position in the run’s sequence of durable calls, so code that changes which call comes next fails loudly on replay. When replay goes wrong has the details.

Waiting runs hold no process

A waiting run holds no process and no agent. A polyglot script that sleeps or waits is suspended and its process exits until the wake time or the signal. Graph runs wait as journaled timers or signal waits. An approval that waits weeks costs the same as a sleep that waits seconds.

Hyperspool keeps the journal

Agents are outbound-only: they long-poll for work, hold a lease with a single-use claim token, and report outcomes for the server to record. Inside a step the agent exposes durable primitives to scripts over a local unix socket. The SDKs are thin clients of that socket and hold no durability logic of their own.

Definitions are pinned per run

The YAML, graph or script source travels in the run’s start event, so an in-flight run replays its submit-time definition whatever has changed since. Agents advertise a build id and self-update by draining, never mid-lease.

Deterministic ids make retries idempotent

Workflow ids are caller-supplied idempotency keys: starting an existing id attaches instead of duplicating. Child ids, activity ids and fan-out child ids derive from the parent and a sequence, and emits and spawns memoize by counter, so retries and replays converge on the same runs and steps.

Three things share the word “event”. A history event is a journal entry inside one run. The event bus is project-scoped pub/sub: spool event emit, handlers, and wait_for_event/cancel_on waiters. The run page’s live stream sends event frames, a projection of the journal for the browser.

Graphs and scripts

There are two execution models: graphs and polyglot scripts. YAML compiles to the graph form and the graph interpreter runs it, so a pipeline job and a canvas node are the same thing to the engine. The two models differ in where a run keeps its position.

A graph run’s position is its journal. The interpreter selects a node, schedules an activity, waits for the result, selects the next, and writes each of those down. A wait node is a decision the interpreter makes itself: nothing is dispatched and no agent is involved.

A script run’s position is a process’s program counter, which can’t be persisted. So the whole script is dispatched as one activity and the position is reconstructed. On a spool wait signal with nothing waiting, the script is killed and the agent is freed. When the signal lands the script runs again from line one, and each durable call it already made returns its recorded answer until execution arrives back at the call that suspended.

Two things follow:

• The engine sees every decision a graph makes, and only a script’s checkpoints. The canvas draws a script run from its checkpoints, plus a marker pinned before each durable call, so a call that hasn’t recorded an answer yet shows as running.
• Anything a script does outside a durable call happens again on every pass. A curl -X POST at the top of a script that waits four times fires four times; inside spool step it fires once. A graph doesn’t ask this of you; it is the price of writing control flow in your own language.

Reach for a graph when the orchestration is the interesting part: fan-out, approvals, per-node retries, anything you want resumable and visible node by node. Reach for a script when the logic is the interesting part and you’d rather write a loop than describe one. They nest either way: a graph node can run a script, and a script can spool submit children.

Topology

       console, CLI, webhooks
                 |
                 |  HTTPS
                 v
         +----------------+
         |   Hyperspool   |
         +----------------+
            ^          ^
            |          |   agents dial out
            |          |   over HTTPS
  +---------+----+  +--+--------------+
  |   managed    |  |    your own     |
  | environments |  |     agents      |
  +--------------+  +-----------------+

spool dev is Hyperspool and an agent in one process on your machine.

Managed environments start on demand

Hyperspool starts an environment for work no agent can pick up: a container for shell, polyglot and connector work, a VM for build. Each has its own credential, scope and trust level, and exits once it idles out. Agents you run yourself are long-lived and connect the same way. See agents: managed and your own.

Agents dial out

Nothing dials in to an agent. A claim long-poll is held for about 20 seconds and work arriving mid-poll is handed over at once. An agent runs one activity at a time; concurrency comes from more agents, which keeps replay, timeouts and leases simple. Capabilities are advertised per claim and intersected with the token’s ceiling.

Capabilities route work

A steps: job needs shell, a script or code node needs polyglot, and http and connector nodes need connector (the resolved credential reaches the agent at claim time). Managed environments provide all three plus build. An agent of your own advertises shell and polyglot by default and needs --cap connector to take connector work. Work nothing can take shows as unroutable.

Restarts and failover

A run doesn’t depend on any one server process. If the one driving it goes away, the run carries on from its journal within seconds. spool dev is a single process, and its runs resume when it starts again.

Workflow types

Four workflow types are registered. The type names the kind of definition that was submitted; every run executes as a node graph over one journal.

• hyperspool/pipeline/v1: a YAML pipeline, compiled to a graph.
• hyperspool-graph/v1: a graph document, drawn on the canvas or written as JSON or YAML.
• hyperspool-node/v1: a single node kind, what spool call --node runs.
• hyperspool/script/v1: a polyglot script, one opaque activity.

Life of a run

A YAML run

Submit → role check → plan limits → concurrency admission (below) → the start event is journaled with the YAML inside it → the graph interpreter walks the graph, and each ready node becomes an activity → an agent claims it; project secrets are injected into the job’s environment at claim time, never earlier → the agent runs it, streaming logs and renewing its lease → the outcome is journaled → the nodes it unblocks start → terminal event → lifecycle events (run.succeeded, run.failed, …) go to the project’s bus.

A script that sleeps

The script calls spool sleep 2h → the agent reports the suspension → the script process exits and the run waits without one. When the time comes the sleep’s checkpoint is recorded first and the work is queued again. Any capable agent claims it; the script re-runs from the top; completed steps return their memos; the sleep returns at once and execution continues.

A signal

For a suspended script, the signal’s checkpoint is recorded and the work queued again. For a graph wait, a signal-received event is journaled and paired with a consumed event, so replay consumes the same signal once. Approvals are signal waits whose form fields the server validates.

A webhook

The webhook’s secret or HMAC signature authenticates the delivery → deliveries are deduplicated on the signed body → the trust rules decide whether the sender is trusted → handlers whose on: and when: match start runs; a run from an untrusted delivery gets no secrets. A repo connected with spool repo connect has a build handler that clones the pushed commit on an agent and spawns the pipeline from it, and a status handler that posts the commit status as runs start and finish. See CI from a repo.

Concurrency admission

A submit with a concurrency group tries an atomic acquire. If the group is held, the policy decides: queue stores the submit and returns a position (the run has no journal yet); cancel-running takes the hold and cancels the incumbent; cancel-queued drops the waiters and takes the queue slot; skip creates no run. When the holder finishes the oldest waiter is submitted, and the next waiter also gets its turn when a holder dies. Cancelling a queued run takes it out of the queue.

Log output

Output streams to the run page as it is produced, and every byte goes to the run’s durable log, up to your plan’s per-run log size limit. The live view keeps the last 4 MiB of a run. If log storage is briefly unavailable, up to 4 MiB of output waits to be written; past that the oldest is dropped and the gap is marked in the stream.

Durability and limits

On the service, everything a run records is durable: its history and step results (encrypted at rest), its logs and artifacts, concurrency queues, event waiters (wait_for_event:, cancel_on:) and bus events. spool dev keeps the same on disk, except bus event history, which it holds in memory and loses on restart.

Limits:

• A run’s input is uncapped. It reaches a pipeline as the input expression scope and any job as the file $SPOOL_INPUT_FILE names.
• The live-log snapshot caps at 4 MiB per run. A run’s durable log has your plan’s size limit.
• A concurrency queue holds at most 100 waiting runs by default and refuses beyond that.
• Event handler chains (a handler’s run emitting an event another handler takes) stop at depth 5.
• Finished runs are kept per plan: 14 days on Free, 90 on Team, 365 on Business. Bus events are kept 30 days.
• Run artifacts go when their run does. Cached artifacts (--key) expire after 30 days.

Step results live in the journal and should stay small: record pointers, and put files in artifacts. Replay cost grows with journal length, so a run that loops on signals for a very long time gets slower to resume; recurring work belongs in a schedule, where each tick is a fresh run.

Security model

Tenancy is org → project → member, with roles Owner, Admin, Member and Viewer. Every user route authorizes the org and project from the path, and a run id’s prefix is proof of its scope. API tokens and agent tokens are separate, and an agent token carries a capability ceiling its claims can’t exceed. Claim tokens are per claim, single-use, and checked on every agent request.

A slug is permanent; to rename, change name. A run id is {org_slug}:{project_slug}:run-{random}, and that string addresses everything the run owns, so no route changes a slug.

Outbound requests the server makes for you, such as registering a forge webhook, go only to public https addresses, with redirects refused. Managed containers reach the public internet only.

Secrets are encrypted at rest with a key of your org’s own. Plaintext exists only inside a claim: secrets are injected when an agent claims the step, so they are never stored with the queued work, and the agent redacts their values from live and stored logs. A job’s environment is sealed too: a fixed allowlist plus the variables you pass through, with agent credentials removed before any job starts.

Runs started by untrusted events (fork pull requests, outside contributors) behave differently. Project secrets are left out of their environment without an error. A credential the run declares is refused, and the step fails, because running without it could quietly change what the step does. Their managed environments are single-use.

Everything a run records is sealed under the same per-org key: step memos, durable state cells and the history itself. A payload that won’t open fails its run rather than reading as absent, because an activity result that can’t be decrypted would otherwise look like one that never ran, and the engine would run it again. Such a run ends as unreadable, a status of its own.