Reshaping data between steps

One step returns a wide record; the next wants three fields under different names. A transform step does that reshaping, so the steps either side of it stay about their own work.

It runs inside the engine, with no agent or container, so it is cheap enough to put wherever the shape is wrong.

The inspector of a transform step: a field table with one row per output field - id, customer, total and big, each computed from an incoming order with its value shown under it - and below it the result of the table on the run input.

Two ways to write one

The default is a table, one row per output field. Each row is a name and either a fixed value or a small expression, which covers most reshaping.

The other is code: a single JavaScript expression that returns the whole output. Use it when the output has fewer rows than the input, or a shape a per-field table can’t describe, such as grouping, joining or filtering.

A step is one or the other. Switch to code on the table hands you the JavaScript your table compiles to, so you start from your own table rather than an approximation of it. There is no way back: going from code to a table would mean guessing an intent you may have edited past.

`each` and `all`

Above the editor is the question this step answers: does it run for each item, or once, over the whole list?

Both modes start from the run’s input, the same input every other expression in a graph reads. Neither reads the step above on its own.

each (the default) runs your expression once per item of the run input. i is the item and index its position. A value that isn’t a list is treated as a list of one, so a single record needs no special case.

all runs it once over the whole value, bound to input. Use it when the answer depends on more than one item, or when the number of rows changes.

To reshape what a step above produced, use all and name it: nodes.<id>.output, with vars and run also in reach. Name a step this one depends on; a step beside it in the graph may not have finished yet, and is not there.

The choice sits above the rows because it decides what every row means: i in i.order_id is the item in one mode and nothing at all in the other.

Worked examples

The test suite runs every example below against its input, and the result shown is the one it produced.

Keep a few fields, renamed

A fetch returns more than the next step needs. Name what comes out.

{
  "mode": "each",
  "fields": [
    { "name": "id", "expr": "i.order_id" },
    { "name": "customer", "expr": "i.buyer.email" },
    { "name": "total", "expr": "i.amount_cents / 100" }
  ]
}

Given:

[
  {
    "order_id": "A-1",
    "buyer": { "email": "kim@example.com" },
    "amount_cents": 2500,
    "internal_note": "ignore me"
  }
]

Result:

[{ "id": "A-1", "customer": "kim@example.com", "total": 25 }]

Add a field, keep the rest

include_input carries the incoming record through, with your fields layered over it. It is off by default, so the table shows everything that comes out.

{
  "mode": "each",
  "include_input": true,
  "fields": [{ "name": "size", "expr": "i.amount > 100 ? 'large' : 'small'" }]
}

Given:

[
  { "id": "A-1", "amount": 250 },
  { "id": "A-2", "amount": 40 }
]

Result:

[
  { "id": "A-1", "amount": 250, "size": "large" },
  { "id": "A-2", "amount": 40, "size": "small" }
]

A named field wins over an incoming key of the same name, so this is also how you overwrite one field and pass the rest along.

A fixed value

A row can carry a literal instead of an expression. It is stored as JSON: 42 is a number, true is a boolean, and anything that isn’t JSON stays the text you typed.

{
  "mode": "each",
  "include_input": true,
  "fields": [
    { "name": "source", "value": "nightly-import" },
    { "name": "priority", "value": 3 }
  ]
}

Given:

[{ "id": "A-1" }]

Result:

[{ "id": "A-1", "source": "nightly-import", "priority": 3 }]

To write the string "42" rather than the number, switch the row to an expression and quote it there.

Keep only some rows

Filtering changes the number of rows, so it needs all and code.

{ "mode": "all", "expr": "input.filter(r => r.status === 'open')" }

Given:

[
  { "id": "A-1", "status": "open" },
  { "id": "A-2", "status": "closed" },
  { "id": "A-3", "status": "open" }
]

Result:

[
  { "id": "A-1", "status": "open" },
  { "id": "A-3", "status": "open" }
]

Group rows and total them

{
  "mode": "all",
  "expr": "Object.entries(Object.groupBy(input, r => r.team)).map(([team, rows]) => ({ team, count: rows.length, total: rows.reduce((s, r) => s + r.n, 0) }))"
}

Given:

[
  { "team": "x", "n": 2 },
  { "team": "y", "n": 3 },
  { "team": "x", "n": 1 }
]

Result:

[
  { "team": "x", "count": 2, "total": 3 },
  { "team": "y", "count": 1, "total": 3 }
]

Look a value up in another list

One step fetched the rows, another fetched what they refer to. Code is a single expression, so statements go in an arrow function you call immediately.

{
  "mode": "all",
  "expr": "(() => { const by = Object.fromEntries(input.teams.map(t => [t.team, t.lead])); return input.items.map(i => ({ ...i, lead: by[i.team] })); })()"
}

Given:

{
  "items": [
    { "name": "a", "team": "x" },
    { "name": "b", "team": "y" }
  ],
  "teams": [
    { "team": "x", "lead": "kim" },
    { "team": "y", "lead": "sam" }
  ]
}

Result:

[
  { "name": "a", "team": "x", "lead": "kim" },
  { "name": "b", "team": "y", "lead": "sam" }
]

Split text into unique values

{
  "mode": "all",
  "expr": "[...new Set(input.flatMap(r => r.tags.split(',').map(t => t.trim())))].sort()"
}

Given:

[{ "tags": "urgent, billing" }, { "tags": "billing" }]

Result:

["billing", "urgent"]

Number the rows

index is the item’s position, bound only in each mode.

{
  "mode": "each",
  "include_input": true,
  "fields": [{ "name": "position", "expr": "index + 1" }]
}

Given:

[{ "id": "A-1" }, { "id": "A-2" }]

Result:

[
  { "id": "A-1", "position": 1 },
  { "id": "A-2", "position": 2 }
]

Building one on the canvas

Start from an example. A transform with an empty table offers eleven working starting points, from keep-and-rename and add-a-field to group, join, sort, dedupe and merge by key. Each carries the data it was written against, so picking one shows a result before anything upstream has run.

Give it an input. A table iterates the run’s input, as it does in a run, so the canvas previews it against the run input you would submit: open input beside Run and paste a sample.

Run the step above. On a graph that has never run, this is where the first real data comes from. Select the step and press Run this node; while you edit, every step below reads its output as nodes.<id>.output, as a run would. It really runs: an http step here calls the service.

Or try the whole graph. Try it beside Run starts a dry run of the draft, where transforms run for real against what the stubbed steps returned in the last real run.

Pick a field instead of spelling it. Click into a row’s expression and the fields the incoming data has are offered underneath, each with its value:

order_id  "A-1"    buyer.email  "kim@example.com"    amount_cents  2500

Picking one inserts i.buyer.email at the caret, so you can pick the field and then type / 100 after it. The list comes from the data a row iterates (the run input, or the example you started from) rather than from a declared schema.

Watch the result as you type. Every row shows what it produced:

id        =fx  i.order_id             → "A-1"
total     =fx  i.amount_cents / 100   → 25

Each row is evaluated on its own, so a mistake in one is reported against that row and the others keep showing their values. The evaluation has no effects, so it can fire on every keystroke.

What an expression can use

The JavaScript here is deliberately small. Object, Array, String, Number, Math, JSON, Map, Set, RegExp and the typed arrays are present, and so are modern methods: Object.groupBy, Array.prototype.flatMap, spread, optional chaining.

Anything that reaches outside is absent: no network, no filesystem, no Date, no Math.random, no timers, no imports.

A transform is re-derived rather than recorded, so it runs again on every replay of the run. A value from a clock or a coin flip would differ the second time, and the run would diverge from its own history. If you need the current time or a random id, take it in a code step, where the result is recorded once and replayed. See the authoring model.

Two limits bound a runaway expression: a tick budget that stops an infinite loop, and a 64MB ceiling on what it can allocate. Both fail the step with a message rather than hanging the run.

What gets checked, and when

Save refuses what it can see is wrong, instead of leaving a run to fail minutes later:

an expression that isn’t valid JavaScript, naming the row it is in
a table row with no name, or with both a fixed value and an expression
two rows with the same name (in JavaScript the later one silently wins, a mistake nobody finds by reading the output)
mode that isn’t each or all
a config with both expr and fields, or neither

A row you have added but not filled in saves and compiles to nothing, so a table can be built one click at a time. A name typed before its expression is fine too: that row holds null until you finish it, visible in the result rather than silently dropped.

When something is wrong

Messages point at what you wrote:

you wrote	you get
`i.order_id +`	this isn’t valid JavaScript yet - check for a missing operand, quote or bracket
`i.name.toUpperCse()`	`i.name.toUpperCse` is not a function - check the spelling
`i.customer.email`	`i.customer` has no value here, so `.email` cannot be read - this item has `order_id`, `buyer` and `amount_cents`

The last one lists the fields the data has, the same list the field picker offers.

In a graph document

This is the document the canvas stores, here as JSON. The workflow page’s Graph YAML view shows and edits it as YAML; spool workflow export <name> prints it as YAML, and spool workflow set takes either. You can also write it by hand, or generate it.

{
  "version": "hyperspool-graph/v1",
  "nodes": [
    {
      "id": "fetch",
      "type": "http",
      "config": { "url": "https://api.example.com/orders" }
    },
    {
      "id": "shape",
      "type": "transform",
      "config": {
        "mode": "all",
        "expr": "nodes.fetch.output.body.map(o => ({ id: o.order_id, total: o.amount_cents / 100 }))"
      }
    },
    {
      "id": "notify",
      "type": "shell",
      "config": { "run": "./post-summary.sh" }
    }
  ],
  "edges": [
    { "from": "fetch", "to": "shape" },
    { "from": "shape", "to": "notify" }
  ]
}

The edge decides the order; it carries no value, so the transform names the step it reads. Downstream steps read this one the same way, as nodes.shape.output.

To try one on its own, hand it the upstream it reads. --wait -o json prints the finished run, its output included:

spool submit --node transform \
  --config '{"mode":"all","expr":"nodes.fetch.output.body.map(o => o.order_id)"}' \
  --nodes '{"fetch":{"status":"success","output":{"body":[{"order_id":"a1"}]}}}' \
  --wait -o json

Without --nodes that config fails saying there is no upstream here, rather than reporting the node it names as missing.

Any part of the config can itself be computed, with the usual markers:

- id: shape
  type: transform
  config:
    mode: each
    expr: { $expr: vars.shaping_expression }

Authoring on the canvas - the palette, the inspector, validation and publish.
The authoring model - why a transform may not read a clock.
Polyglot scripts - when the reshaping wants a real program, a library, or the network.

Reshaping data between steps

Two ways to write one

each and all

Worked examples

Keep a few fields, renamed

Add a field, keep the rest

A fixed value

Keep only some rows

Group rows and total them

Look a value up in another list

Split text into unique values

Number the rows

Building one on the canvas

What an expression can use

What gets checked, and when

When something is wrong

In a graph document

Related

`each` and `all`