8. Probes and data-driven fan-out

A probe is a named observation that the build graph can see: the output of a shell command, the contents of a JSON file, a recorded tool identity, a Lua computation. A probe re-runs on every invocation in which it’s reached — nothing about a probe’s own value is ever cached. Recipes consume probes three ways (reference, fan out, seal); seal is the one that folds a probe’s value into a unit’s cache key, so a unit that only observes an unchanged world still hits.

The probe shapes

Four producer forms run a body, fresh on every invocation that reaches them, and return its value:

probe greeting
    { echo hello }                        # bare: a string

probe target_list
    seal "data/targets.txt"
    lines { cat data/targets.txt }        # an array, one per line

probe services
    seal "data/services.json"
    json { cat data/services.json }       # structured data

probe count: services
    >{ return { n = #cook.probes.get("services") } }   # a Lua value

A seal line on a probe declares the files its body reads, so the value recomputes exactly when they change. Probes depend on other probes with the same colon syntax as recipes; a >{ lua } body reads its upstreams with cook.probes.get.

Top-level declarations can record file and tool determinant sets:

tools compiler
    cc                                     # the resolved identity of an executable

files sources
    "src/**/*.c" !"src/gen/**"             # a file set, hashed per file

probe build_env
    lines { echo "$HOSTNAME"; echo "$TERM" } # grouped environment values

tools records which cc resolved and its content hash. The ordinary named shell probe groups environment observations for sealing. files records each matched path’s content hash (gather glob syntax), so editing, adding, or removing any matched file changes the value.

Probes are demand-driven: one only runs when something scheduled actually consumes it. Which brings us to the three ways of consuming.

Using probes

Reference one in a step. A $<...> sigil with a colon in its name is a probe reference (the colon is how cook tells it apart from a recipe or config reference, so name probes namespace:thing when you intend to reference them). One sigil interpolates the value and records the dependency edge; pair it with seal (below) to also fold the probe’s value into the unit’s cache key:

probe stack:node
    { node --version }

recipe banner
    seal stack:node
    cook "build/banner.txt" {
        echo "built with node $<stack:node>" > $<out>
    }

Upgrade node and the banner rebuilds; nothing else does — the probe itself re-observes on every build, but it’s seal that makes the rebuild happen only when what it observed actually changed. $<key> substitutes a scalar value (a string verbatim, a number or boolean as its JSON literal); $<key.field> and $<key.field[1]> address into a table value, one-based. A placeholder that lands on a whole table, a null, or a missing field is an error naming the placeholder, not silently-interpolated text. (Execute-phase Lua bodies have no sigils; they read the same values with cook.probes.get("key"), which cook detects and wires identically.)

Fan out over one. Point a recipe’s gather at a probe (bare name, not a quoted glob) and the recipe runs once per member. Record fields are addressable as $<in.FIELD>:

recipe render
    gather services
    cook "build/$<in.name>.conf" {
        printf 'url = %s\n' "$<in.url>" > $<out>
    }

A field holding a nested value (a list or an object) renders as compact, key-sorted JSON, so $<in.tags> on {"tags": ["a", "b"]} is ["a","b"]; a string field renders bare, with no surrounding quotes.

Add one entry to services.json and exactly one new unit builds; remove one and cook sweeps its orphaned output. A downstream recipe that iterates the same probe can join to a sibling’s per-member output with $<render[in]>:

recipe summary: render
    gather services
    cook "build/$<in.name>.txt" { cat $<render[in]> > $<out> }

This is the shape behind eval suites, per-package monorepo tasks, and any “one unit per record” build.

Seal one into the key. Some values should invalidate the cache without appearing in any command: the compiler’s identity, the environment, the files a tool reads implicitly. seal folds the probe’s value into the key of every unit in the recipe:

recipe app
    gather "src/*.c"
    seal compiler build_env
    cook "build/$<in.stem>.o" { cc -c $<in> -o $<out> }

Swap cc for a different binary, or change a sealed environment value, and every unit misses, attributably: cook why prints the sealed determinant that changed. A sealed probe is never interpolated; it is a declared determinant, nothing more. Sealing is the backbone of shared-cache correctness, and Caching and cache trust shows how it composes with files to cover inputs your gather line can’t hold.