§ 3 — Lexical structure

3. Lexical structure

Normative. This chapter defines how a Cookfile’s text is broken into tokens. The productions in § 4, § 5, § 6, § 8, and Appendix A consume the token classes defined here.

3.1. Source representation

A Cookfile is a sequence of Unicode characters encoded as UTF-8. A conforming implementation MUST accept any valid UTF-8 input and MUST NOT require a byte-order mark. The behaviour on input that is not valid UTF-8 is unspecified.

Lexical analysis is line-oriented: the source is divided into lines by the newline character U+000A (\n), and each line is classified independently as a single token (§ 3.2). A conforming implementation MUST accept \r\n as a line terminator by treating the carriage return as part of the preceding line’s trailing whitespace, which is trimmed before classification (§ 3.10). The final line of the source need not be terminated by \n.

Leading and trailing whitespace on a line are not significant to token classification, with the following exception already stated in § 3.9: leading whitespace is preserved inside brace-balanced block bodies (§ 3.9) because the body is reassembled from the raw source lines. In all other cases, a line is trimmed of leading and trailing whitespace before its prefix is tested (§ 3.7, § 3.10).

Note 3.1.1

The reference tokenizer in cli/crates/cook-lang/src/lexer.rs iterates source.lines(), which is defined by Rust to split on \n and strip a trailing \r. The observable effect matches the rule above: \r\n and \n produce the same token stream (informative).

3.2. Tokens

The Cookfile lexer produces exactly one token per source line. A token carries its 1-indexed line number; downstream phases (§ 4, § 6) use that number for diagnostics and for locating multi-line block bodies (§ 3.9). There is no concept of “multiple tokens on one line” at the Cookfile layer — the interior structure of a line (for example, the output patterns of a cook step) is parsed at the syntactic layer on top of the single Content token.

The token classes produced by the lexer are:

Token class	Produced for	Defined in
`Comment`	Line whose first non-whitespace character is `#`	§ 3.6
`RecipeHeader`	Line beginning with keyword `recipe` followed by a separator	§ 6.1
`ConfigHeader`	Line `config` or `config` + name	§ 5.3
`UseDecl`	Line `use` + name	§ 5.1
`ImportDecl`	Line `import` + `BARE_IDENTIFIER` + path	§ 5.2
`LuaLine`	Line whose first non-whitespace characters are `>` (but not `>{`)	§ 3.7, § 8.7
`LuaBlockOpen`	Line whose first non-whitespace characters are `>{`	§ 3.7, § 3.9
`Blank`	Line containing only whitespace	§ 3.10
`Content`	Any other line; its classification within a recipe body is deferred to § 8.1	§ 8.1

A conforming implementation MAY represent these classes with any internal data structure. The token names above are used descriptively throughout this Standard; they do not mandate a specific variant name in an implementation.

Note 3.2.1

Content is deliberately broad: the lexer does not distinguish gather "…", cook "…" …, @vim notes.txt, or gcc -o main main.c — all four produce Content tokens whose string form is the line after trimming. The step-dispatch cascade in App. A.4 resolves which step kind applies.

3.3. Identifiers

A BARE_IDENTIFIER is a sequence of ASCII characters matching the regular expression in App. A.5:

BARE_IDENTIFIER ::= /[A-Za-z_][A-Za-z0-9_.\-]*/

The first character MUST be an ASCII letter (A–Z, a–z) or underscore (_); each subsequent character MUST be an ASCII letter, ASCII digit, underscore, hyphen (-), or dot (.). A conforming implementation MUST reject any other character. Unicode letters and digits are NOT part of BARE_IDENTIFIER.

BARE_IDENTIFIER is used in four syntactic roles, each with its own supplementary constraints documented where that role is defined:

Role	Supplementary constraint
Import alias (`import_declaration`)	MUST be unique across imports within the Cookfile (App. A.2).
Recipe / chore name (declaration site)	MUST NOT contain `.` (App. A.2). The name is a single undotted segment; namespacing is achieved via `import` (§ 11.2), not via dotted recipe names.
Dependency name / cross-recipe reference	MAY contain `.`. A dotted dependency name resolves through an `import` alias per § 11.3; the segment before the leftmost `.` selects the import, the remainder is the recipe name in the imported Cookfile.
First segment of a `module_call`	MUST NOT contain `.` or `-` (App. A.4).

Where the grammar admits either a BARE_IDENTIFIER or a double-quoted STRING (for example, a recipe name or a dependency), the two forms are equivalent in meaning; the quoted form exists to admit names that would not otherwise match BARE_IDENTIFIER. The “no dots” restriction on declared recipe and chore names applies to both forms — a quoted recipe name MUST also be undotted.

Example 3.3.1

import backend ./services/backend

recipe ship: backend.build lib setup
    cook "ship.out" { echo deploying > $<out> }

ship, lib, and setup are valid undotted recipe / dependency names. backend.build is a valid dotted dependency name: it is not a recipe defined here, but a qualified reference to the build recipe of the Cookfile imported under the alias backend. By contrast, a top-level declaration recipe backend.build would be rejected at parse time — namespaced recipes live behind import, not in dotted top-level definitions.

3.4. Keywords

The Cookfile language reserves seven identifiers contextually, each in a single role: the final dot-separated segment of a recipe name. Outside that role the identifiers are ordinary tokens.

Reserved recipe segment	Role where reserved
`stem`	Final segment of a recipe name
`name`	Final segment of a recipe name
`ext`	Final segment of a recipe name
`dir`	Final segment of a recipe name
`in`	Final segment of a recipe name
`out`	Final segment of a recipe name
`env`	Final or first segment of a recipe name

all is NOT a reserved recipe segment (CS-0130): recipe all and a dependency segment all are both well-formed. The all-spelled sigil was removed as a placeholder form by the same change; it now resolves as an ordinary name reference — a recipe if all is in scope, otherwise an env var lookup (§ 9.1, App. E CS-0130).

A conforming implementation MUST reject a placeholder identifier (§ 9.1) or a dependency name whose final dot-separated segment is a reserved recipe segment (App. A.2). The reserved-segment rule is contextual to the resolution surface — it constrains what dotted reference forms are well-formed, not the lexical class of BARE_IDENTIFIER itself. (At a top-level recipe or chore declaration site the name is undotted by § 3.3, and the reserved-segment rule does NOT constrain it: recipe out, chore name, and recipe env are all well-formed declaration names (CS-0132). The reservation binds only the reference surfaces below — a dependency_list final segment, a $<...> placeholder identifier, and the env-first prefix — never a declaration name or a chore parameter name.) The identifier env is further restricted: it also MUST NOT appear as the first segment of a dotted reference, so $<env.foo> is the explicit env-var-lookup form and a dependency env.foo is rejected at parse time. This restriction reserves the env. prefix for the explicit env-var-lookup form and eliminates the resolution ambiguity that would otherwise arise between $<env.X> (env-var lookup) and an accessor on a recipe named env.

The keywords that introduce other top-level or recipe-body productions — recipe, chore, config, gather, cook, use, import, test, files, tools — are recognised as keywords only when followed by a keyword separator (space, tab, or "). (The former body introducer using was removed in CS-0099 and is no longer a keyword; at its old position it is rejected with a migration diagnostic, § A.4.) Among these, the top-level keywords recipe, chore, config, use, import, register, files, and tools are recognised as keywords only when their line begins at column 0 (no leading whitespace); an indented line whose first word is a top-level keyword is a Content token, dispatched per § 8.1. The recipe-body keywords (gather, cook, test) are not subject to the column-0 rule. The register keyword is recognised when followed by space, tab, or NEWLINE — register alone on a line is admitted as the canonical header form because the block takes no name.

Note 3.4.1

The separator test is the governing subtlety. See the test test_recipe_prefix_is_shell_command in cli/crates/cook-lang/src/lexer.rs, which confirms that recipes_cleanup produces a Content token and not a RecipeHeader (informative).

3.5. Strings

A STRING token is a double-quoted sequence of characters defined in App. A.5:

STRING ::= /"[^"]*"/

A STRING opens with U+0022 ("), contains zero or more characters none of which is ", and closes with U+0022 ("). A conforming implementation MUST NOT interpret backslash escapes inside a STRING: the character sequence is taken literally. A conforming implementation MUST reject a STRING that is not terminated on the same line — STRING tokens do not span lines.

The empty string "" is a valid STRING. A double quote inside a STRING is not expressible in the current grammar; authors needing such content can use the inline-block forms of § 3.9, whose interior language (Lua or shell) has its own quoting rules.

STRING appears in recipe and dependency names, in gather, cook, and test steps, and in the optional name of a config_block header. In each of those roles, a conforming implementation parses the STRING inside the single line that produced the governing token. The cook_step body (§ 8.4) takes a brace-block form, not a STRING; the former single-string body was removed in CS-0022, and the using introducer itself in CS-0099.

Example 3.5.1

recipe "build"
    gather "src/*.c" !"src/legacy.c"
    cook "bin/app" { gcc $<in> -o $<out> }

Each "…" is a STRING. The !"src/legacy.c" form is an input exclusion, not a distinct string class (App. A.4): it is the literal ! followed by an ordinary STRING. The { … } body is a shell block (§ 3.9), not a STRING.

Note 3.5.1

The reference implementation of the string parser is in cli/crates/cook-lang/src/cook_line.rs, function parse_single_quoted_string. The lack of escape processing is observable: the test test_cook_step_shell asserts that a STRING value is delivered verbatim to the step (informative).

3.6. Comments

A comment is a line whose first non-whitespace character is #. The comment extends to the end of the line. A conforming implementation MUST produce a Comment token for such a line and MUST NOT treat the # character as beginning a comment in any other lexical position.

In particular, a # that appears after the first non-whitespace character of a line is part of that line’s content and is not stripped by the lexer. Downstream phases (notably the shell runtime) MAY assign their own meaning to in-line # characters; that meaning is not part of Cookfile lexical syntax.

A Comment token is accepted anywhere a toplevel_item (App. A.1) or a recipe_item (App. A.3) is admitted, and is discarded at the syntactic layer. At top level, however, a contiguous Comment block immediately preceding a recipe or chore header, with no intervening Blank token, attaches as that declaration’s full description. The reference human listing renders the description’s first line. Comments are permitted inside config_body (App. A.2) and inside brace-balanced blocks (§ 3.9), where they are preserved verbatim in the collected block body because their removal is the interior language’s concern, not the Cookfile lexer’s.

Example 3.6.1

# top-level comment: discarded at the syntactic layer

chore "build"
    # indented comment: also a Comment token
    gcc -o main main.c   # not a comment — part of the shell command

The first two # lines produce Comment tokens. The # not a comment sequence on the third source line is part of the Content token for the gcc command (a chore-body shell_command); its treatment is up to the shell runtime (§ 10). The lexer classifies Content tokens the same way regardless of the enclosing body; a chore body is used here because a recipe body admits no loose shell line (§ 8.1).

3.7. Line prefixes

A line prefix is a character sequence that, when it appears as the first non-whitespace characters of a Cookfile line, selects a token classification other than Content. The Cookfile language recognises two lexical line prefixes:

Prefix	Token produced	Phase	Defined in
`>{`	`LuaBlockOpen`	execute	§ 3.9, § 8.7
`>`	`LuaLine`	execute	§ 8.7

A conforming implementation MUST test these prefixes in longest-prefix order: >{ before >. Without this ordering, a >{ would be consumed as > followed by a literal {. A conforming implementation MUST apply these prefixes only after leading whitespace has been trimmed from the line; thus >{ opens an inline Lua block identically to >{ at column zero.

A run of three or more > characters at the start of a trimmed line — >>>, >>>>, etc. — is reserved and MUST be rejected with a parse error. The reservation keeps space for future prefixes without ambiguity.

The phase column above is informative; the normative phase rule is in § 8.7 and § 13.1. Both >{ and > are execute-phase prefixes; as of CS-0134 a conforming implementation MUST admit them only in body position — inside a cook, test, or probe body, or inside a chore body — and MUST NOT admit them as a standalone recipe-body step. The position constraint is normative in § 8.7; this chapter defines only the prefix-to-token mapping. See § C.6.8 for background on the surface split.

The @ character is not a lexical prefix, and — as of CS-0134 — carries no meaning at any layer. A line whose first non-whitespace character is @ produces a Content token whose string form includes the leading @. CS-0134 removed the @ interactive prefix and the interactive_command step kind it once introduced (Note 3.7.1); an @ line is therefore never a distinct step kind. Inside a chore body it is an ordinary shell_command whose body text begins with a literal @ and is passed verbatim to the shell (which will typically report @…: command not found); inside a recipe body it is a Content line matching no declarative step kind and is rejected (§ 8.1); outside any body it is rejected because Content tokens are not valid at the top level (§ 4.2).

Note 3.7.1 — Removal of the doubled-`>` prefixes (CS-0134)

Prior to CS-0134, the language additionally recognised a register-phase pair of doubled-> prefixes: >>{ (InlineLuaBlockOpen) and >> (InlineLuaLine), alongside the execute-phase >{ / > above. CS-0134 REMOVED both doubled-> prefixes entirely: the recipe body is now purely declarative, and a bare ident.ident(...) module-call line auto-classifies as register-phase (§ 8.1) in the position that >> / >>{ used to occupy.

A trimmed line whose first non-whitespace token is >> — and that is not one of the reserved >>>+ forms above — is still recognised by the lexer (§ 3.10), but a conforming implementation MUST reject it with a migration diagnostic directing the author to a top-level register block (§ 5.4) or to a bare module.call() line. The same applies to >>{.

A LuaLine body MAY be empty: the line > alone produces a LuaLine whose body is the empty string. See § 8.7.

3.8. Numbers

A NUMBER token is a non-empty sequence of ASCII digits defined in App. A.5:

NUMBER ::= /[0-9]+/

The NUMBER token denotes an unsigned integer: signs, decimal points, fractional digits, and unit suffixes are NOT permitted. As of CS-0135 the NUMBER token has no grammar consumer: its sole use was the timeout operand of a test_step, and the test modifiers (as / timeout / should_fail) were removed in the v1.0 language cut (§ 8.6, App. E CS-0135). The token class is retained in the lexical grammar in anticipation of the planned 1.x re-add of a per-test time bound.

No step kind currently accepts a bare NUMBER; digits that appear inside a shell command, a STRING, or a Lua block are the interior language’s concern and are not Cookfile tokens.

3.9. Brace-balanced blocks

A brace-balanced block is a span of source text delimited by { and a matching }, where matching is defined by a stateful brace-counting algorithm that ignores braces appearing inside certain syntactic contexts. Both the LUA_BLOCK_CONTENT (inside >{ … }) and the SHELL_BLOCK_CONTENT (inside { … }) are brace-balanced blocks; they differ in their interior language (Lua vs. shell) and in how their collected body is normalised after the block closes.

The opening brace (a LuaBlockOpen token for >{, or the { that opens a shell_block body) establishes initial depth 1, and scanning begins immediately after it, with the remainder of the opening line. The block body is the character span from that point to the block’s matching } (CS-0154): the opening-line remainder, any number of full interior lines, and the closing line’s prefix (the text before the matching }) are all body text. The span is examined character by character in source order; { increments depth and } decrements depth, except that braces are ignored when they appear inside one of the opaque spans enumerated in § 3.9.1 (for Lua blocks) or § 3.9.2 (for shell blocks). The brace counter MUST carry opaque-span state across lines so that a span opened on one line — including one opened on the opening-line remainder — and closed on a later line keeps every brace byte between the open and the close opaque.

A conforming implementation MUST apply this exclusion consistently for both Lua blocks and shell blocks, so that a block’s closing } is never mistaken for a literal } inside string, comment, or heredoc content. The block closes at the first character at which the running depth returns to zero. Body text MAY precede the closing brace on its line. Text following the closing brace on the same line is NOT body text: it is the enclosing production’s trailer. A production that admits a trailer consumes it — the cook step’s modifier tail (§ 8.4.3) and the test step’s rejected-modifier diagnostics read from exactly this position; in every other context (a probe producer’s block, a chore-body Lua block) a conforming implementation MUST reject non-whitespace trailer text with a parse error (before CS-0154 stray trailer text in those contexts was silently discarded). A block that opens and closes on one line ({ cmd }, >{ EXPR }) is the degenerate single-line case of the same span, not a separate production, and its interior is scanned with the same opaque-span rules. A conforming implementation MUST reject a block whose depth never returns to zero by end of input.

The body’s segments are the opening-line remainder, each full interior line, and the closing-line prefix, in source order. The LUA_BLOCK_CONTENT body is the concatenation of its segments joined by newlines, preserving interior whitespace, except that a boundary segment (the opening-line remainder or the closing-line prefix) that is empty or whitespace-only is dropped — >{ alone on its opening line contributes no leading blank line. The SHELL_BLOCK_CONTENT body is further normalised per App. A.5: each segment is trimmed of leading and trailing whitespace, and segments that are empty after trimming are discarded; the result is an ordered sequence of command strings, not a single raw string. The per-segment trim applies uniformly, including to lines interior to a multi-line quoted string (§ 3.9.2).

A conforming implementation MUST reject a block whose depth never returns to zero by end of input.

3.9.1. Opaque spans inside Lua blocks

For a LUA_BLOCK_CONTENT body and for the body of a multi-line module call (App. A.4 module_call), braces are ignored when they appear:

inside a Lua line comment, beginning at -- and extending to the end of the line (single-line span);
inside a double-quoted string "…", with \-escapes honoured (single-line span);
inside a single-quoted string '…', with \-escapes honoured (single-line span);
inside a Lua long string [==[ … ]==] of any =-level — including the level-0 form [[ … ]] (single-line or multi-line span);
inside a Lua block comment --[==[ … ]==] of any =-level — including the level-0 form --[[ … ]] (single-line or multi-line span).

For long strings and block comments, the closing bracket MUST have the same =-level as the opener: a ]] token does not close a [==[ opener, and a ]==] token does not close a [[ opener. This is the long-bracket rule of the Lua 5.4 reference manual; a conforming implementation MUST honour it for any number of = characters.

When a long string or block comment is opened on one source line and closed on a later one, every brace byte in the intervening lines is opaque to the brace counter. CS-0035 establishes this multi-line carry as a normative requirement; see App. D.

3.9.2. Opaque spans inside shell blocks

For a SHELL_BLOCK_CONTENT body, braces are ignored when they appear:

inside a single-quoted string '…' (no escapes; single-line or multi-line span — a POSIX single-quoted string spans newlines, and quote state MUST carry across body lines per CS-0154);
inside a double-quoted string "…", with \-escapes honoured (single-line or multi-line span, as above);
after a backslash-escape \{ or \} (the next byte is consumed as data; single-line span);
inside a POSIX heredoc body opened by <<TAG, <<-TAG, <<'TAG', or <<"TAG" and closed by a line whose trimmed text equals TAG (multi-line span).

Both string spans and heredoc openers participate on the opening-line remainder like on any other body line: a quote or heredoc opened after the block’s { on the opening line keeps its interior opaque on the following lines. App. A.5’s per-segment trim applies to every body line regardless of any open quote context: the interior lines of a multi-line quoted string are trimmed like any other body line before the commands are assembled.

The heredoc closer is matched against the trimmed body line: this is consistent with the SHELL_BLOCK_CONTENT line-normalisation rule above, which trims each interior line before assembling the shell script. Authors writing a heredoc inside an indented { … } body therefore do not need to dedent the closing delimiter to column 0 in the source; the trim-first match closes the heredoc as soon as the trimmed line matches the delimiter.

When a shell heredoc is opened on one source line and closed on a later one, every brace byte in the intervening lines is opaque to the brace counter. Multiple heredocs queued on a single opener line (cmd <<A <<B) MUST be consumed in opener order: the first delimiter line closes the first heredoc, the second closes the second, and so on. CS-0035 establishes this multi-line carry as a normative requirement; see App. D.

Example 3.9.1

recipe "build"
    cook "out.wasm" >{
        if mode == "release" then
            local t = {1, 2, 3}   -- a literal `}` in a comment: }
            local s = "a { brace in a string }"
            print(s)
        end
    }
    cook "out.js" {
        wasm-pack build
        cp out/main.js out.js
    }

Both blocks here are cook-step bodies (CS-0134: a >{ … } or { … } block appears only as a step body, never standalone in a recipe body). The first cook step’s exec_lua_block (>{ … }) body includes the if/end structure and the nested table {1, 2, 3}; the braces inside the string "a { brace in a string }" and after the -- comment are ignored by the brace counter. The second cook step’s { … } shell body, after line normalisation, is the two-element command list ["wasm-pack build", "cp out/main.js out.js"].

Example 3.9.2

recipe "build"
    cook "doc.txt" >{
        local doc = [==[
            multi-line long string with a } byte
            and a stray ]] that does NOT close it
        ]==]
        print(doc)
    }

The >{ … } here is the cook step’s exec_lua_block body (CS-0134: a >{ … } block appears only as a step body). The [==[ … ]==] long string spans four source lines. The } byte on the second body line and the ]] token on the third are interior to the long string and MUST be opaque to the brace counter. The >{ … } block closes only at the trailing } on the line after print(doc). The corresponding pin is fixture 041-lua-long-string-leveled in the conformance corpus.

Example 3.9.3

recipe emit
    cook "out.txt" {
        cat <<EOF > out.txt
        a heredoc with a } brace
        and a } here too
        EOF
        echo done
    }

The cat <<EOF opens a shell heredoc; its body extends through the next two lines, in which } bytes are heredoc data and MUST be opaque to the brace counter. The heredoc closes on the trimmed EOF line; brace counting resumes for echo done and the trailing } closes the { … } block. The corresponding pin is fixture 042-shell-heredoc-brace-in-body.

Example 3.9.4

probe platforms
    json { echo '[
        {"name": "web",     "level": "9"},
        {"name": "desktop", "level": "0"}
    ]' }

The producer’s shell block opens on the json line, and echo '[ — the opening-line remainder — is the block’s first body segment (CS-0154). The single-quoted string opened there spans four source lines; every { and } byte inside it, including those in the JSON records, is opaque to the brace counter. The block closes at the final }; the ]' before it on the closing line is body text (it carries the string’s closing quote). After App. A.5 normalisation the body is the four-segment command sequence ["echo '[", "{\"name\": \"web\", \"level\": \"9\"},", "{\"name\": \"desktop\", \"level\": \"0\"}", "]'"], which the assembled script joins with newlines — a single echo whose quoted argument spans them. The corresponding pin is fixture positive/shell-block-multiline-quote.

Example 3.9.5

probe rows
    >{ return {
        { name = "web" },
        { name = "desktop" },
    } }

return { is the opening-line remainder and the Lua block’s first body segment. The closing line } } first closes the Lua table (depth 2 → 1) and then the block (1 → 0); its prefix } (the table’s closer) is body text. The assembled LUA_BLOCK_CONTENT chunk is return { ⏎ { name = "web" }, ⏎ { name = "desktop" }, ⏎ }. Before CS-0154 both boundary segments were discarded, truncating the chunk to its interior lines — a Lua syntax error. The corresponding pin is fixture positive/lua-block-boundary-segments.

Note 3.9.1

The governing examples are the tests test_lua_block_nested_braces, test_lua_block_brace_in_string, test_lua_block_brace_in_comment, and the cs_0035_* family in cli/crates/cook-lang/src/tests.rs; the tests collects_three_lines, drops_blank_lines, respects_nested_braces_in_content, and the cs_0035_* / cs_0154_* families in cli/crates/cook-lang/src/shell_block.rs; and the unit tests in cli/crates/cook-lang/src/brace_scan.rs (informative).

3.10. Line classification cascade

For each source line, a conforming implementation MUST classify the line into exactly one token by applying the following tests in order, with the first matching test winning. The line’s trimmed form (leading and trailing whitespace removed) is the subject of each test except where noted.

A keyword separator in tests 7–11 below is a space (U+0020), a tab (U+0009), or a double quote (U+0022). The double quote is admitted so that authors can write recipe "foo" with no space between the keyword and the opening quote of a quoted name; the quote is not consumed by the keyword test and remains available to the STRING parser (§ 3.5). Test 11 (import) admits only space or tab: no quoted form exists for the import alias (App. A.2).

If the trimmed line is empty, the token is Blank.
If the trimmed line’s first character is #, the token is Comment (§ 3.6).
If the trimmed line begins with >>{, the line is recognised as a doubled-> prefix; per CS-0134 this prefix was REMOVED from the language, and a conforming implementation MUST reject the line with a migration diagnostic rather than emit a usable token (§ 3.7).
If the trimmed line begins with >> (but not >>{, by test 3), the line is recognised as a doubled-> prefix; per CS-0134 this prefix was REMOVED from the language, and a conforming implementation MUST reject the line with the same migration diagnostic as test 3 (§ 3.7).
If the trimmed line begins with >{, the token is LuaBlockOpen (§ 3.7, § 3.9, § 8.7).
If the trimmed line begins with > (but not >> by test 4 or >{ by test 5), the token is LuaLine (§ 3.7, § 8.7).
If the line begins at column 0 and the trimmed content begins with the keyword recipe followed by a keyword separator, the token is RecipeHeader (§ 6.1). An indented line whose trimmed content begins with recipe falls through to test 12.
If the line begins at column 0 and starts with chore followed by a separator (space, tab, or "), the token is ChoreHeader. The keyword separator rule of § 3.4 applies: chores_cleanup (no separator after chore) is not a ChoreHeader. (CS-0020 / v0.4.)
If the line begins at column 0 and the trimmed content is exactly config, or begins with config followed by a keyword separator, the token is ConfigHeader (§ 5.3). An indented line falls through to test 12.
If the line begins at column 0 and the trimmed content begins with use followed by a keyword separator, the token is UseDecl (§ 5.1). An indented line falls through to test 12.
If the line begins at column 0 and the trimmed content begins with import followed by a whitespace separator (space or tab only; not "), the token is ImportDecl (§ 5.2). An indented line falls through to test 12. 11.5. If the line begins at column 0 and the trimmed content is exactly register, or begins with register followed by a keyword separator (space or tab), the token is RegisterHeader (§ 5.4). An indented line falls through to test 12. A register line with non-whitespace content after the keyword on the same line is rejected at the syntactic layer per § 5.4; the lexer admits the RegisterHeader token regardless and lets the parser issue the diagnostic.
Otherwise, the token is Content. The lexer emits Content uniformly; its further classification is the concern of the step-dispatch cascade (App. A.4, § 8.1) and is made by the syntactic layer per enclosing container. The admissible dispatch results differ by container (CS-0134): inside a recipe body a Content token classifies to gather_step, seal_step, cook_step, test_step, or a bare module_call (register-phase, § 8.1 rule 7) — a loose shell_command is rejected, because a recipe body is purely declarative (§ 6.3). Inside a chore body a Content token classifies to a shell_command, which remains admitted (§ 7.2) and is interactive by default (§ 7.3). CS-0134 removed the @ interactive prefix and its interactive_command step kind from the language (Note 3.7.1), so an @ line is no longer a distinct classification in either container. (Top-level module_call is dispatched separately per § 4.4.)

The ordering is significant. In particular, the four Lua-prefix tests (3–6) MUST appear in longest-prefix order — >>{ before >> before >{ before > — so that a longer prefix is never consumed as a shorter one followed by a literal character, and so that a doubled-> line receives the correct migration diagnostic rather than falling through to a >{ / > classification (§ 3.7). Tests 3 and 4 never yield a usable token: they exist only to recognise the removed >>{ / >> prefixes precisely enough to reject them with the migration diagnostic of CS-0134 (Note 3.7.1) rather than let such a line fall through to test 12 as unclassified Content. Tests 7–11 apply only to lines beginning at column 0, and MUST require a separator so that identifiers whose first word merely starts with a keyword (for example, recipes_cleanup, chores_cleanup, configure, useful, important) fall through to test 12 and are not misclassified as declarations.

A trimmed line beginning with three or more consecutive > characters (>>>, >>>>, etc.) does not match any of tests 3–6 and is reserved (§ 3.7). A conforming implementation MUST reject such a line with a parse error rather than fall through to test 12.

Tests 1–6 and test 12 apply to lines whether or not they are indented. Tests 7–11 are sensitive to column-0 positioning: an indented line whose first non-whitespace content would otherwise match a top-level keyword test falls through to test 12 and is classified as Content.

Example 3.10.1

use "cpp"                 # test 10: UseDecl (column 0)
config debug              # test 9:  ConfigHeader (column 0)
    var.CC = "gcc"

recipe build: lib setup   # test 7:  RecipeHeader (column 0)
    # test 2: Comment
    gather "src/*.c" # test 12: Content (step-dispatched to gather_step)
    >>{                   # test 3:  recognised, then REJECTED (CS-0134 — doubled-`>` removed)
        var.CFLAGS = "-Wall"
    }
    >> local n = 4        # test 4:  recognised, then REJECTED (CS-0134 — doubled-`>` removed)
    >{                    # test 5:  LuaBlockOpen (lexes; recipe parser REJECTS a standalone >{ per CS-0134)
        print("hi")
    }
    > os.exit(0)          # test 6:  LuaLine (lexes; recipe parser REJECTS a standalone > per CS-0134)
    recipes_cleanup       # test 12: Content (falls through test 7 — no separator)

Example 3.10.2

register                  # test 11.5: RegisterHeader (bare form)
register foo              # test 11.5: RegisterHeader (parser rejects the trailing arg)
register	foo           # test 11.5: RegisterHeader (tab separator)
    register              # test 12:   Content (indented; falls through 11.5)
register_cleanup          # test 12:   Content (no separator after `register`)
registers_cleanup         # test 12:   Content (no separator; not a keyword match)

The bare register and register foo lines both produce a RegisterHeader token; per § 5.4, the parser rejects the second because a register_block admits no name. The tab-separator form is equivalent to the space-separator form at the lexical layer. The indented register line is a Content token because tests 7–11.5 are sensitive to column-0 positioning. register_cleanup and registers_cleanup fall through to test 12 because the byte after register is _ and s, not a keyword separator.