Tooling API

@rohal12/spindle/tooling is the entry point for editors, linters and build tools that work on Spindle stories from Node.js. It loads without Preact, zustand, immer or micromark: everything it exports is a plain string-to-data function.

Besides the macro registry (defineMacro, getMacroRegistry) and the checks (parseStoryVariables, validateMarkup), it exports the parsing rules Spindle itself reads passages with. Use them instead of copying the rules: the runtime calls the same functions, so tooling and runtime agree on where a string, template or regex literal ends, what a selector is, how a macro's arguments split, which closer closes which opener and which parts of a passage are code.

Stability

The exports below are part of the package's public API and follow semver: a change to a signature, or to what a rule accepts, comes with a major version (while the package is at 0.x, a minor version), and a new export with a minor one. Types are in types/tooling.d.ts. Offsets are UTF-16 code units, as in JavaScript strings.

The leaf rules take complete input and don't recover from half-typed text on their own. The tolerant functions (tokenizeMarkupTolerant, pairMarkup, passagePieces, parseDeclarations) never throw for malformed input: they return what they could read, and the errors.

JavaScript lexer

Export	What it does
`lexJs(src, handlers, goal?)`	Walks `src` as JavaScript, calling `code`, `literal` and `variable` handlers in source order, every character once. Lenient: an unterminated literal runs on.
`lexTemplate(src, start, handlers?, n)`	`lexJs` for the template literal opening at `start`; returns the index past its closing backtick.
`findCodeEnd(src, start, options?)`	Where the code starting at `start` ends: the `}` that closes the code (or, with `stop`, the first `{` for which it holds); `-1` if there is none.
`scanStringLiteral(src, start)`	The end of the `"…"` or `'…'` literal at `start`, and whether it is closed.

goal is 'expression' (the default, a macro argument) or 'statements' (a {do} body); it decides whether / starts a regex or divides. A Sigil is $, _, @ or %; variable handlers get the sigil and name of each reference such as $gold (not a property name, nor text in a literal).

Markup tokens and selectors

tokenizeMarkup(source, { text? }) returns the flat Token[] (text, link, macro, variable, expression and html tokens, each with start and end; a link token also has targetStart and targetEnd, where its target is written), and throws a MarkupError (with reason, offset, line, column) for a malformed tag.
tokenizeMarkupTolerant(source, { text? }) is tokenizeMarkup for half-typed markup. It never throws for malformed tags: it returns { tokens, errors }, where errors holds a MarkupError for each malformed tag. The tokens before a malformed tag are kept, the character that started it is read as text, and tokenizing resumes after it, so every character of source is in exactly one token and all offsets are in source. For well-formed markup the tokens are tokenizeMarkup's and errors is empty. A MarkupError's offset is where the problem is: the start of the unclosed { or [[, or the opening quote of an unclosed attribute value. It also has end (where the offending text ends), a stable code and data (see Diagnostics).

Every token says where its parts are written, so an editor needn't scan the token's text again (all offsets are UTF-16 into the input, and the same for tokenizeMarkup and tokenizeMarkupTolerant):

Token	Fields
`macro`	`nameStart`/`nameEnd` (after `{`, `/` and the selectors); `argsStart`/`argsEnd` for `rawArgs`, without the whitespace around it (empty, at `nameEnd`, with no arguments)
`link`	`displayStart`/`displayEnd` for the label (`targetStart`/`targetEnd` for the target)
`variable`	`nameStart`/`nameEnd`, without the sigil
`expression`	`expressionStart`/`expressionEnd`
`html`	`tagNameStart`/`tagNameEnd` (for closers too); `attributeSpans`, every attribute as written, in order, duplicates included: `{ name, nameStart, nameEnd, valueStart?, valueEnd?, quote? }` (a value is without its quotes; `quote` is `"` or `'`, absent for an unquoted value; `attributes` still has the first of equal names)
`macro`, `link`, `variable`, `expression`	`selectorsStart`/`selectorsEnd` for the `.class#id` selectors, without the space after them; absent without selectors

A text token that is a closed HTML comment has comment: true.

parseSelectors(source, at?) reads the .class#id selectors that start source at at, and returns { className?, id?, end }, where end is the index past them and the one space that may follow. Names may contain {$name} interpolations.
SIGIL_SCOPES maps each sigil to its scope (variable, temporary, local, transient), and isSigil(c) tests a character.

Macro arguments

splitArgs(raw) splits at top-level commas, or, with none, adjacent standalone values ("Label" "target").
splitTopLevel(src, isSeparator) splits at every separator character outside literals, comments and brackets.
readQuoted(src, start), unescapeQuoted(body) and stripLooseQuotes(src) read quoted arguments: \", \' and \\ are unescaped, other backslashes kept.
endsWithOperator(src) tells whether code ends in an operator that still needs an operand.
splitIncludeFlag(rawArgs) returns { inline, passage } for the arguments of {include}: the inline flag only counts as the first or last word outside quotes and brackets.

Code and passage names

transform(expr, goal?) is the sigil transform: it returns the JavaScript Spindle runs for expr, with $var, _var, @var and %var turned into __variables["var"], __temporary["var"], __locals["var"] and __transient["var"]. These names are sigil identifiers themselves (they start with _), which the lexer refuses to declare (const __variables is refused like const _x), so a helper's own variables or locals never captures a reference. References in string, template and regex literals and in comments are untouched. Pass 'statements' for a {do} body. It throws a SyntaxError for code that is not well-formed.
passageTarget(arg) reads the passage argument of {goto}, {include} or {link}: a string literal is a name ({ kind: 'name', name }, its value as JavaScript reads it: "\u0048all" is Hall), anything else (a literal JavaScript rejects too) an expression ({ kind: 'expression', expression }) whose value names the passage when it runs.
evaluatePassageName(expr, evaluate, { storyData, currentPassage }) is how the macros find the passage: String(evaluate(expr)), throwing what evaluate throws and an error naming the current passage if no passage has that name. Give it your own evaluate.
collectPassageReferences(source) returns the passages the markup names, in source order (it is passagePieces less what names no passage, and reads the labels and attribute values that hold markup too), as { macro, target, start, end }: [[…]] links (macro is link), the passage argument of {goto}, {include}, {link} and macros that declare one, the goto and dialog actions of {watch} (and the string and text arguments of macros that declare they hold a passage name) and the body of {dialog}, one for each. target is a passageTarget; start and end span the name as written (quotes included), the target of a link (not its label). It uses the macros registered with defineMacro, and skips malformed tags, so it reads half-typed markup.

Pairing tags

pairMarkup(tokens, { isBlock?, isRaw?, source? }) pairs the flat tokens into a tree, as the runtime does (it builds its AST from the same pairing): which closer closes which opener, which {else}, {case}, {default}, {elseif} or {next} belongs to which macro, and which macros take a body. It returns { nodes, errors } and never throws:

A PairedNode is { token, body?, start, end }. A token that stands alone (text, a link, a variable, an expression, a macro without a body, a self-closing element) has no body; the opening tag of a macro or element has body: { children, branches, close? }, where branches are { tag, children } and close is the closing tag (absent if it is never closed). start and end span the whole thing, from the opening tag to the end of the closer.
errors has a PairingError for each problem, in the order a parser reading from left to right notices them: { code, message, start, end, noticedAt, data }, with code one of unclosed-block ({if} with no {/if}, <div> with no </div>), mismatched-closer, stray-closer and misplaced-branch. start and end are the offending tag.
The tree is recovered around a problem. A closer closes the innermost open element, as the runtime reads it; when it names one further out, it closes that one, and what is open inside it stays unclosed. A closer with nothing open, and a branch outside its macro, are dropped from the tree.

Which macros take a body is isBlock(name): by default the built-in ones (isBlockMacro, also exported). A story that defines other block macros passes them (the block of defineMacro's macros, and the widgets it defines, which are blocks when their body uses {@children}). source lets a message say at which line and column an element was opened. tokenizeMarkupTolerant then pairMarkup reads half-typed markup:

Effective macro and widget metadata can replace a built-in block with an inline definition. Such calls need no closing tag; independent explicit block registrations still take a body.

const { tokens } = tokenizeMarkupTolerant(source);
const { nodes, errors } = pairMarkup(tokens, { source });

Macro ownership

macroCalls(source, macros?) classifies opening macro heads as { token, kind, owner? }, in source order. Pass the complete effective metadata (for example, builtinMacros plus discovered macros and widgets). Widgets take precedence over macros, and later definitions replace earlier ones within each group. The function keeps no registry state and tolerates unfinished markup.

invocation: an ordinary macro or widget call.
value: a direct clause consumed by a built-in parent, such as {option} inside {listbox} or {cycle}, even when an option widget exists.
branch: a declared companion such as {else} or a custom submacro.

token is the opening MacroToken, with original UTF-16 spans including nameStart and nameEnd. owner is the consuming parent's opening token; it is absent for invocations and stray branches. Closing tags are omitted. Labels and attributes that hold markup are classified independently, preserving their original source spans. A replacement macro or widget does not inherit a built-in control's value ownership. This identifies dispatch roles, not whether a conditional body executes; custom JavaScript can choose how to use its body. Editors can use kind === 'invocation' when finding or renaming widget calls, preserving parent-owned clauses that happen to share the widget's name.

macroCallAt(source, cursor, macros?) returns the opening head containing a UTF-16 cursor, including an unfinished head (#707), or undefined outside a head. It uses the same MacroCall shape and ownership rules as macroCalls. For example, at EOF in {option "A, an option widget is an invocation; in {cycle "$x"}{option "A or {listbox "$x"}{option "A, the call is a value with the control as owner. Labels and attributes have independent ownership scopes, and replacement widgets do not inherit built-in control ownership. Invalid cursor offsets throw RangeError.

Pieces of a passage

passagePieces(source, macros?) says which parts of a passage are code, which are passage names, and which are text with markup of its own, and where each is: the answer the story-start check and collectPassageReferences read, so an editor needn't reconstruct it for variable references (rename, find references, undeclared-variable diagnostics, semantic tokens, hover), syntax checking and passage-name references. It reads half-typed markup, and returns, in source order, with offset in UTF-16 units into source:

{ kind: 'code', code, goal, label, passage?, inString?, macro? }: {$expr} displays, {do} bodies, the conditions of {if}, {elseif} and {case}, macro arguments of an expression, statements or passage parameter, strings that hold code (goal is 'expression' or 'statements'), and the {…} of attributes that hold code (onclick). passage: true marks a passage argument that is an expression.
{ kind: 'passage', name, length, macro, label }: a passage name written out: [[…]] links (macro is link), quoted passage arguments, strings that hold a passage name, the body of {dialog}. length is how much of the markup it is, quotes included.
{ kind: 'text', text, where, tokens, errors }: text that holds markup of its own (labels, attribute values). Its tokens and errors (a MarkupError for each malformed tag and unpaired tag, the first being what the story-start check reports) have offsets into source. The pieces of that markup follow it, with nested: true and where (the text's description, as in an error message), also with offsets into source: no caller needs to tokenize again and add offsets.
{ kind: 'argument-error', message, length, macro, label }: macro arguments that don't have their parameters' forms.

macros are those the markup may use (builtinMacros and your own, as for validateStoryMarkup): their declared parameters tell which arguments are code. In a quoted string with escapes ("say \"hi\" {$x + }"; in the argument of a widget, any JavaScript escape: A, \x41, \n), the text is the unescaped value, so a character is not at offset + index: such a piece has sourceOffsets, and pieceOffset(piece, index) gives where character index of its code or text is in source (it is offset + index when there is no sourceOffsets). A nested piece's length and sourceOffsets count the source's characters.

Keep the source: 'builtin' provenance when passing entries from builtinMacros or getMacroRegistry(). An explicit macro entry with source: 'user', or without source, replaces that builtin. In particular, a replacement named do still receives its raw body as text, but that body is not classified as executable JavaScript: passagePieces, variable-reference checks and completion contexts do not invent code execution for a replacement that ignores its children. The genuine builtin do retains its statements body, and a widget named do retains its markup and forwarding semantics. When no do entry is supplied, the implicit builtin default remains available.

Stateless checks and the built-in macros

validateMarkup(passages, options) and collectPassageReferences(source) read the process-global registry, which only defineMacro can add to. A host that analyses several projects in one process (a language server with workspace folders) would leak macros between them. For that, use the functions that take the macros and keep no state:

validateStoryMarkup(passages, macros, options?) is validateMarkup against macros. The options are checkPassageNames and tolerant: the story-start check reads a passage up to its first malformed or unpaired tag, which is enough to refuse to start. An editor wants every problem at once, so with tolerant: true all the malformed tags and pairing errors of a passage are reported (with the same codes, messages and spans), and the unknown macros, syntax errors in code, argument errors and unknown passage names are checked in the markup that could be read around them: an unclosed {if} on line 3 doesn't hide the unknown macro on line 5, the broken link on line 7 and the bad expression on line 9. Without the option nothing changes.
collectStoryPassageReferences(source, macros) is collectPassageReferences against macros.
passagePieces(source, macros?) as above.
builtinMacros is the list of built-in macros (a frozen array of the metadata getMacroRegistry() starts with), loaded without fs: a bundler (esbuild, rollup) inlines it. A macro is { name, block, subMacros, parameters?, interpolate?, widget? } (ToolingMacro); the metadata getMacroRegistry() returns has all of it except widget. validateStoryMarkup reads the widgets from the passages it is given, so it needs no widget; to check the calls of a widget in passagePieces and the other single-passage functions, pass the widget from widgetDefinitions as a macro: { name, block, subMacros: [], widget }, where widget is { params, holds } of its definition (see What an Argument Holds).

import { builtinMacros, validateStoryMarkup } from '@rohal12/spindle/tooling';

const mine = { name: 'alert', block: true, subMacros: [] };
validateStoryMarkup(passages, [...builtinMacros, mine]);
validateStoryMarkup(otherProjectsPassages, builtinMacros); // no `alert` here

widgetDefinitions(passages, macros?) lists the widgets a story defines as the runtime registers them: those of StoryInit and of the passages tagged widget, in source order. Each is a WidgetDefinition:

name, params (the @ parameters, as {widget "Name" @a @b} declares them, separated by spaces or commas: @a, @b is the same) and block: whether its body renders {@children}. It is decided on tokens, so a {@children} in an HTML comment or a {do} body does not count, while one in an attribute value or a label does. In nested definitions it belongs to the innermost.
passage, and as [start, end) UTF-16 offsets into its content: the opening {widget …} tag (start, end), the name as written without quotes (nameStart, nameEnd) and the {/widget} closer (closeStart, absent while the definition is not closed).

It is tolerant: a half-typed definition is reported without closeStart, one whose arguments cannot be read is left out. macros is the list validateStoryMarkup takes, for the parameters that hold markup. parseWidgetDef(rawArgs) reads the arguments alone ({ name, params }). The runtime's startup, validateStoryMarkup and these exports read definitions with the same code.

Story-wide validation uses the last closed definition of each widget, with StoryInit definitions taking precedence over tagged widget passages regardless of passage order. Both markup and variable checks use that definition's parameter names and holds; unclosed definitions do not replace it.

import { widgetDefinitions, builtinMacros } from '@rohal12/spindle/tooling';

const [def] = widgetDefinitions(
  [
    {
      name: 'W',
      tags: ['widget'],
      content: '{widget "Box" @tone}<b>{@children}</b>{/widget}',
    },
  ],
  builtinMacros,
);
// { name: 'Box', params: ['@tone'], block: true, passage: 'W', start: 0, … }

Variable references

variableReferences(source, macros?) returns every $ and % variable reference a passage's markup evaluates, in source order, as { sigil, name, path, start, end }: $a.b.c is name: 'a', path: ['b', 'c'], and [start, end) are UTF-16 offsets into source. They are found where the story start looks for them: {$var} displays, expressions, conditions, {do} bodies and the code arguments of macros (by their declared parameters, so {set} targets count), the quoted names input macros bind ({textbox "$name"}; pass the macros with storeVar), and the markup in labels, selectors and HTML attributes. Strings, comments, property names and prose hold none; _ and @ locals are never variables. The path is the dotted part only: $a?.b and $a["b"] give $a. The target of {unset} and {computed} and a {$name} in the selectors of a link, display or expression are not in the list, as the story start does not check them. Malformed tags are skipped.

variableReferences(source, macros, { all: true }) returns the complete list, for rename and find-references: the checked references above, in source order, and also sigilless input bindings ({textbox "name"} and {textbox name}), the variable a macro names in a variable parameter (the target of {unset $x} and {computed $x = …}), and the {$name} in the selectors of a link ([[.c{$name} Go->T]]), display or expression. validateVariableReferences checks the first list only; sigilless bindings do not add startup checks.

Input binding references have sigil: '$' even when the author omitted it. Their binding object contains name, path, explicitSigil, start, end, nameStart and nameEnd, with UTF-16 offsets into the passage. The full span excludes quotes and includes $ when written; [nameStart, nameEnd) covers only the root spelling, so a rename can replace it without removing quotes, a sigil or dot members. Root and member names are decoded: {textbox "$\u0078.na\u006de"} refers to name: 'x', path: ['name'], while its editable root spelling is \u0078.

parseBoundVariable(rawArgs) exposes the same binding reader the runtime uses for built-in and custom storeVar macros. It returns that binding object with offsets into rawArgs, or null for missing arguments, and throws for rejected namespaces, invalid identifier escapes and reserved path names. It reads a raw identifier path, stripping quotes without decoding the argument as a JavaScript string: the argument "$\u0078" binds x, while a JavaScript string escape such as "$\x78" is rejected. Escaped roots require a literal $ sigil; dot members use JavaScript identifier escapes. Tooling skips rejected quoted bindings.

validateVariableReferences(passages, declarations, macros?) checks them, as the story start does, and returns VariableDiagnostics: { passage, code, name, path?, start, end, message }, with offsets into the passage's content and the story start's message.

declarations is { variables, transients? }, maps from name to schema: from parseDeclarations, new Map(declarations.map((d) => [d.name, d.schema])). A variable with no schema (a non-static initializer) is declared and its fields are not checked. transients is checked only when given.
undeclared-variable and undeclared-transient: the variable is not declared. primitive-field: a field a number, string or boolean has no member of ($hp.nope; $hp.toFixed, $name.length and a character of a string, $name.0, are fine). reserved-name: $__proto__.
Not errors, as at story start: an unknown field of an object (a registered class can add members), any field of an array or of a null default.
Passages tagged script or stylesheet, and the declaration passages, are skipped.
The widgets the passages define are read as validateStoryMarkup reads them: a call of one whose parameter holds expression, statements or markup has the references in that argument checked (variableReferences finds them too, given the widget metadata of the macro).

import {
  builtinMacros,
  parseDeclarations,
  validateVariableReferences,
} from '@rohal12/spindle/tooling';

const { declarations } = parseDeclarations('$hp = 10\n');
const variables = new Map(declarations.map((d) => [d.name, d.schema]));
validateVariableReferences(
  [{ name: 'Start', content: 'HP {$hp} of {$maxHp}' }],
  { variables },
  builtinMacros,
);
// [{ passage: 'Start', code: 'undeclared-variable', name: 'maxHp', start: 13, end: 19, message: 'Undeclared variable: $maxHp' }]

Macro definitions

discoverMacros(source) reads what the defineMacro({ … }) and Story.defineMacro({ … }) calls in JavaScript declare, without running it: a script passage, the body of a {do} (such as StoryInit's), a project file. It is for editors, which need the metadata of custom macros (the typed parameters are what lets passagePieces find code in a macro's arguments) before anything runs. It returns, in source order, a DiscoveredMacro for each call: a ToolingMacro (name, block, subMacros, parameters?, interpolate?, storeVar?, so it can go to validateStoryMarkup and passagePieces) with merged?, description? and where the name is written without its quotes, [nameStart, nameEnd) UTF-16 offsets into source, for go-to-definition.

It is static and tolerant, and throws for nothing. It skips what it does not read as JavaScript does: a regex literal, a template, a comment that ends at any line terminator. A call is found where it is written (also in a function), when its argument is an object literal, or a variable declared as one with const, let or var, and its name is a string. A variable is the declaration its call sees (the innermost in scope, and before the call unless the call is in another function; a later item of const a = 1, config = {…} is a declaration too, the scope of a let or const in a for header is the loop, and that of a var in a class static block is the block): a parameter, a pattern, a function, a value that is not an object literal, an assignment (also &&=, +=, ++ and a destructuring one) or a second declaration in the same scope makes it unknown, and so does a write to its name, block, subMacros, parameters, interpolate, storeVar, merged or description that can come before the call (config.block = true, delete config.name, config.parameters.push(…), Object.assign(config, …)); a write to another property (config.render = …) does not matter. Then the call declares no macro. Of that object it reads only what is written out, and a later member that replaces one is the value (a spread of an object literal is read, { name: 'a', ...{ name: 'b' } } is b; a method, an accessor, a shorthand member, a computed key or a spread of anything else leaves the property it can replace as not written): block, interpolate, storeVar and merged as true or false, description, the strings of subMacros (block defaults as defineMacro does: a macro with sub-macros is a block unless block: false) and parameters, when every parameter is an object literal with a string name and type (and, as written, required, description, holds and the parameters of an options one). If one is not, or defineMacro would refuse it (an unknown type), the macro declares no parameters. Calls in comments and strings, text that is not a call, and a name that is not a string are skipped.

import {
  discoverMacros,
  validateStoryMarkup,
  builtinMacros,
} from '@rohal12/spindle/tooling';

const macros = discoverMacros(`
  Story.defineMacro({
    name: 'alert',
    block: true,
    parameters: [{ name: 'level', type: 'expression' }],
    render: (props, ctx) => ctx.h('div', null, ctx.renderNodes(props.children)),
  });
`);
// [{ name: 'alert', block: true, subMacros: [], parameters: [...], nameStart: 29, nameEnd: 34 }]
validateStoryMarkup(passages, [...builtinMacros, ...macros]);

Formatting a passage

formatPassage(body, options?) formats the text of one passage and resolves to the formatted text. It is what an editor's Format command needs, without a language server:

The bodies of block macros are indented by two spaces per level; {else}, {elseif}, {next}, {case} and {default} snap back to their macro. Trailing whitespace goes, except Markdown hard breaks (two spaces before a line) and the indentation of lists and code fences, which is kept relative to the body.
What the runtime executes is not touched: a macro, a link, an expression or a {do} body keeps its text, line breaks and the whitespace after them included. These are the spans tokenizeMarkupTolerant reads, so the formatter agrees with the runtime on where a macro ends. A string or template literal that spans lines, in a {do} body or a <script>, keeps its value: only the lines of code around it are re-indented.
<script> blocks and multi-line HTML are formatted with Prettier, with the macros inside kept as written. A block whose macros Prettier would lose, move or rewrite (CSS in a <style> such as a{color:red}, which the runtime reads as a macro, or {$a} in a script) stays as written. <svg> blocks are left alone.
What a passage displays does not change. Prettier formats HTML with htmlWhitespaceSensitivity: 'css', so whitespace is added or removed only where it does not render (<b>a</b>, b keeps its comma against the </b>). HTML holding a <pre>, a <textarea> or a white-space style, whose whitespace renders as written, stays as written. HTML that a story stylesheet gives another display than its default is formatted for the default.
The result ends with one newline ('' formats to '\n'), keeps the more common of CRLF and LF, and keeps a byte order mark. Formatting the result again changes nothing.

The options:

kind: 'normal' (default) for story markup; 'script' or 'stylesheet' formats the whole text as JavaScript or CSS.
prettier: Prettier, as { format(code, options): Promise<string> }. It is injected, not imported, so the tooling bundle holds no copy of it: pass prettier, or prettier/standalone with the babel, estree, postcss and html plugins in a browser. Without it, or where it throws on a piece of code (a half-typed <script>), that code stays as written and the rest is formatted. prettierOptions (default { tabWidth: 2 }) are its options.
isBlock(name): which macros take a body. By default the built-in block macros and the names the passage itself closes with {/name}; pass your own to replace that default whole.
blockMacros: names of block macros to add to that default, such as the ones the story or an editor knows (['dialog', 'my-widget']). Ignored when isBlock is given.
isDedentingSubMacro(name): default else, elseif, next, case and default.
maxLineLength: wrap prose lines longer than this at word boundaries, never inside a macro, an expression or a link, and never in a {do} body. Lines belonging to Markdown headings, tables, lists, blockquotes, reference definitions, links, images or code stay intact to preserve their structure, even when longer than the limit.

import * as prettier from 'prettier';
import { formatPassage } from '@rohal12/spindle/tooling';

await formatPassage('{if $a}\nyes\n{/if}\n', { prettier });
// '{if $a}\n  yes\n{/if}\n'
await formatPassage('const   x=1', { kind: 'script', prettier });
// 'const x = 1;\n'

In a browser, prettier/standalone has no parsers of its own: it takes them from the plugins option of each call. prettierOptions replaces the default { tabWidth: 2 } rather than adding to it, and a call made without plugins fails (that code then stays as written). Wrap it so every call carries the plugins, whatever options the formatter passes (this is what Spindle's Twine editor extensions do):

import * as prettierStandalone from 'prettier/standalone';
import * as babel from 'prettier/plugins/babel';
import * as estree from 'prettier/plugins/estree';
import * as postcss from 'prettier/plugins/postcss';
import * as html from 'prettier/plugins/html';
import { formatPassage } from '@rohal12/spindle/tooling';

const prettier = {
  format: (code, options) =>
    prettierStandalone.format(code, {
      ...options,
      plugins: [babel, estree, postcss, html],
    }),
};

await formatPassage(text, { prettier, prettierOptions: { tabWidth: 4 } });

Diagnostics

isSpecialPassage(name) exposes the runtime's case-sensitive navigation restriction without booting a story. Filter navigation completions with !isSpecialPassage(name); keep these passages available for include and dialog content. Literal links, built-in goto and link passage arguments, and watch's goto option receive a special-passage diagnostic for forbidden targets. Custom macro and widget passage parameters keep their own contracts. Dynamic destination expressions are checked when they run.

Markup inside labels and attributes has a shared limit of 64 nested texts and a total rescanning budget of eight times the passage length (at least 65,536 UTF-16 units). Exhausting either produces a positioned nesting-limit diagnostic; variable-reference collection stops expanding that text. Ordinary HTML and block nesting retains its separate limit of 256 elements.

A MarkupDiagnostic from validateMarkup and validateStoryMarkup has, besides passage, line, column, message and the file and fileLine of its passage:

code, the kind of error. It stays the same where the wording of message changes, and is covered by semver with the data it carries, so editors map codes to severities and quick-fixes without parsing messages.
start and end, UTF-16 offsets into passage.content: the offending text. line and column are those of start.
data, what the diagnostic names.

Name suggestions are optional and use a bounded comparison budget. Diagnostics still report unknown names when an oversized name or a large candidate set exhausts that budget; suggestions may then be empty.

`code`	What	`data`
`unclosed-link`, `unclosed-expression`, `unclosed-macro`, `unclosed-tag`, `unclosed-attribute`	`[[` with no `]]`; `{$…` with no `}`; `{name…` with no `}`; `<div` with no `>`; a value with no closing quote	none
`invalid-closer`, `closer-with-selectors`, `closer-with-arguments`, `unexpected-character`, `syntax`	`{/` and no name; a closer with selectors or with arguments; a character a tag can't have; any other	none
`unclosed-block`	`{if}` with no `{/if}`; `<div>` with no `</div>`	`name`
`mismatched-closer`	`{/if}` where `{/for}` should close the `{for}`	`name` (the opener), `closer`
`stray-closer`	a closer with nothing to close	`name`
`misplaced-branch`	`{else}` outside `{if}`	`name`, `parent`, `inside` (for a branch inside another element)
`unknown-macro`	a macro that is not defined	`name`, `suggestions` (the closest known macro, if one is close)
`argument-error`	arguments that don't have their parameters' forms	`macro`
`code-syntax`	a syntax error in code	`macro` for a macro argument
`unknown-passage`	a passage name that names no passage	`name`, `macro` (`link` for `[[…]]`), `suggestions` (the closest passage name, if one is close)
`special-passage`	a literal navigation target reserved for runtime use	`name`, `macro`
`nesting-limit`	markup nesting or recursive label/attribute expansion exceeds its depth or work budget	`limit`, and `name` for element nesting
`unquoted-passage-name`	`{goto Kitchen}`	`name`, `macro`

message and formatDiagnostic are unchanged. The codes of malformed markup are also on the MarkupErrors of the tolerant functions, with the same data.

Declarations

parseDeclarations(content, sigil?) reads a StoryVariables (or, with '%', StoryTransients) passage without evaluating the initializers and without throwing: { declarations, errors }. parseStoryVariables reads with the same grammar, and then evaluates.

A Declaration is { name, nameStart, nameEnd, valueStart, valueEnd, schema? }: the name (without the sigil) and the initializer expression (without the space around it) as [start, end) UTF-16 offsets into content. One bad line doesn't hide the rest.
schema ({ type, fields? }) is there when the initializer is static: a number (also signed, hex, NaN and Infinity, with or without a sign), string (also a template without ${}), boolean or null literal, an array literal ({ type: 'array' }, as parseStoryVariables makes it), or an object literal, whose fields are its members with an identifier, string or number key (a whole number literal, named as the runtime names it: 0x10 is 16, 1e+2 is 100, 1.5 is 1.5) and a static value. A member that is not static (a call, a reference) has no field, and neither has a name that a later member can replace: a member that is not static, a shorthand member, a computed key, an accessor or a spread takes away the fields written before it ({ a: 1, ...x } has none; { ...x, a: 1 } has a), and of equal keys the last counts. Anything else (Math.PI, new Date(), 1 + 2, a reference) has no schema, and is no error: its value is only known by running it.
An error is { code, offset, end, message }: invalid-declaration (a line that is not $name = value), invalid-name (a name no variable can have), duplicate-declaration (both declarations are returned; the later wins when evaluated), unsupported-value and syntax.
- unsupported-value is an initializer that is a literal no variable can hold, wherever it is in an object that is static: a function (function, =>, class, a method), undefined (also void 0) or a BigInt (10n), and a literal whose evaluation throws (+1n). Its message is what parseStoryVariables throws after the passage name: it names the value the runtime finds (Unsupported type "undefined" for value undefined. for { a: { b: undefined } }), not the initializer. Only the final value of a key counts ({ a: undefined, a: 1 } is fine), and a call (function () { return 1 }()) is a value that is only known by running it.
- syntax is an initializer that the runtime cannot compile ((1, 1 2, { a: 1 b: 2 }). offset is the problem, end the next character (for an unexpected end, the last character), and the message that of the parser (the runtime's wording differs). The declaration is returned, without a schema.

import { parseDeclarations } from '@rohal12/spindle/tooling';

const { declarations, errors } = parseDeclarations('$hp = 100\n$f = () => 1\n');
// declarations: hp (schema: number), f (no schema)
// errors: one `unsupported-value`, at `() => 1`

import {
  tokenizeMarkup,
  splitArgs,
  parseSelectors,
} from '@rohal12/spindle/tooling';

for (const token of tokenizeMarkup('{.hint link "Go, now" "Hall"}')) {
  if (token.type === 'macro') console.log(token.name, splitArgs(token.rawArgs));
}

widgetDefinitions(passages, macros?) returns every declaration, including incomplete ones. effectiveWidgetDefinitions(definitions) selects the closed definitions the runtime uses: tagged widget passages register first, then StoryInit, with later names replacing earlier ones case-insensitively. An unclosed definition never replaces a closed one.

The returned definitions are the same objects as in the input. For documents with duplicate passage names, call widgetDefinitions for each passage separately and map each returned definition object to its source document before combining the lists. Pass that combined list to effectiveWidgetDefinitions; use the identity map to recover each selected definition’s document and interpret its UTF-16 offsets there. WidgetDef.holds and ToolingMacro.widget carry the widget argument contracts.

Executable read and write roles

parseCode(source, goal = 'expression') exposes the runtime’s JavaScript analysis. Use 'statements' for a {do} body. It returns { refs, strings }; each reference has sigil, name, start, end, shorthand, read and write. Offsets are UTF-16 offsets into the code string.

A direct assignment, deletion or destructuring target writes its binding. Compound assignments and updates read and write. A member assignment reads its root binding, and computed keys and default initializers are reads. For example, parseCode('[_x = _x + 1] = []') reports the first _x as write-only and the second as read-only. It throws for malformed code and invalid sigil declarations, just as the runtime does; use lexJs for tolerant lexical scanning when parsing fails.

Completion contexts

completionContext(source, cursor, macros?) identifies whether a UTF-16 cursor is in executable markup, including unfinished tags, expressions, labels, attributes and code-holding strings (#632). Pass builtinMacros plus your project's macro and widget metadata. It inspects the prefix before the cursor with the runtime grammar and argument rules; it does not execute story code. Invalid cursor offsets throw RangeError.

The result is { kind: 'literal' } in prose, ordinary strings, comments, regex literals and template text. In executable positions, including template interpolations and a bare variable sigil, it returns { kind: 'code', code, goal, cursor, offset, sourceOffsets? }. cursor is the position in code; pieceOffset(context, index) maps code positions back to the original passage, including escapes inside nested quoted labels. lexJs(context.code, handlers, context.goal) can then inspect variable and field references without reconstructing markup parsing.

import {
  builtinMacros,
  completionContext,
  pieceOffset,
} from '@rohal12/spindle/tooling';

const source = '<span title="{$inventory.';
const context = completionContext(source, source.length, builtinMacros);
if (context.kind === 'code') {
  console.log(context.code); // $inventory.
  console.log(pieceOffset(context, context.cursor)); // source.length
}

When code is nested in text, containers gives its ownership chain from outermost to innermost, including unfinished containers (#651):

{ kind: 'string', quote, escapes } describes a quoted macro argument (escapes: 'quotes') or widget argument (escapes: 'javascript'). The quote is " or '.
{ kind: 'attribute', name, quote, code } describes an HTML attribute. quote is null for an unquoted value; code identifies event/code attributes, where only sigil interpolations are markup.
{ kind: 'link-label' } describes the display text of a bracket link.

To insert code, encode replacement text from the innermost container outwards. For a quotes string, prefix each backslash and either quote character with a backslash; for a javascript string, use JavaScript string escaping for its delimiter (including line endings). Attribute and link containers preserve code inside markup braces as written. Do not HTML-encode quotes inside an expression. containers is absent at the passage level, and literal contexts remain { kind: 'literal' }.

For example, completionContext(source, source.length, builtinMacros) for source = '{button "{print $obj.' reports a string container with quote: '"' and escapes: 'quotes'. Replacing the dot with ["first-name"] requires escaping the quotes in the inserted text to produce \" in the passage. Use pieceOffset for the replacement range; it retains mappings through decoded strings and CRLF independently of the container chain.

completionTarget(source, cursor, macros?) supplies the complete code region around a cursor, including the suffix after it (#706). It returns the same literal/code distinction, containers and source mappings as completionContext, with cursor in the full decoded code. identifier, when present, gives { kind: 'root' | 'member', name, start, end, dotStart? }. name is decoded and excludes a root's sigil; [start, end) includes the sigil for roots and the entire existing identifier, including Unicode escapes after the cursor. A member's dotStart includes the preceding member operator when replacing dot access with bracket access. Empty sigils and empty members support edits while typing. Code positions that have no root/member target omit identifier.

const source = String.raw`{print $fo\u006f}`;
const target = completionTarget(source, 10, builtinMacros);
if (target.kind === 'code' && target.identifier) {
  const start = pieceOffset(target, target.identifier.start);
  const end = pieceOffset(target, target.identifier.end);
  const edited = source.slice(0, start) + '$foobar' + source.slice(end);
  // {print $foobar}
}

For $obj.$fo\u006f, a field edit uses identifier.start through identifier.end; choosing a field that needs ["first-name"] instead starts at identifier.dotStart. Map both boundaries with pieceOffset and encode the inserted text through containers. These mappings include escaped suffixes in complete and unfinished nested labels and attributes. Consumers need no identifier parser or guessed closing markup. completionContext continues to return only the code prefix for existing callers.

Macro argument spans

macroArgumentSpans(rawArgs, parameters) groups arguments using the runtime’s parameter rules without evaluating expressions. Each result contains the original parameter object, its top-level parameterIndex, its parameterPath through nested options, and UTF-16 start/end offsets into rawArgs. Quotes, flag words, separators and option keywords are included. Options have both an enclosing span and individual keyword spans; choose the most specific overlapping span for nested option help.

For parameters label: string and value: expression, "label" 1 + 2 maps 1 + 2 to value, even though it has several whitespace-separated terms. Names and rest text similarly map to their single declared parameter. Results survive missing or unclosed strings so an editor can offer help while typing. Missing arguments have no span; on whitespace after the last argument, an editor may keep that argument active or select the next parameter according to its UI policy.

Metadata-aware tokenization

tokenizeMarkup(source, { macros, text? }) and tokenizeMarkupTolerant(source, { macros, text? }) accept effective ToolingMacro metadata, including widget definitions (#652). Supply builtinMacros followed by discovered macros and widgets. A widget named do then has an ordinary markup body, so nested invocations appear as macro tokens for navigation and rename. Without metadata, or with the builtin do, its raw JavaScript body stays a text token.

const macros = [
  ...builtinMacros,
  { name: 'do', block: true, subMacros: [], widget: { params: [] } },
];
const result = tokenizeMarkupTolerant('{do}{caption "x"}{/do}', { macros });
// result.tokens: do opener, caption invocation, do closer

Both functions keep UTF-16 offsets in the original source. The tolerant function also retains its recovery errors. Use text: true when tokenizing a label or attribute value; HTML and bracket links then remain text. Metadata is local to the call and does not change later tokenization.

Macros

Special Passages

Widgets

Custom Macros

Tooling API

Stability

JavaScript lexer

Markup tokens and selectors

Macro arguments

Code and passage names

Pairing tags

Macro ownership

Pieces of a passage

Stateless checks and the built-in macros

Widget definitions

Variable references

Macro definitions

Formatting a passage

Diagnostics

Declarations

Effective widget definitions

Executable read and write roles

Completion contexts

Macro argument spans

Metadata-aware tokenization

Tooling API ​

Stability ​

JavaScript lexer ​

Markup tokens and selectors ​

Macro arguments ​

Code and passage names ​

Pairing tags ​

Macro ownership ​

Pieces of a passage ​

Stateless checks and the built-in macros ​

Widget definitions ​

Variable references ​

Macro definitions ​

Formatting a passage ​

Diagnostics ​

Declarations ​

Effective widget definitions ​

Executable read and write roles ​

Completion contexts ​

Macro argument spans ​

Metadata-aware tokenization ​

Tooling API

Stability

JavaScript lexer

Markup tokens and selectors

Macro arguments

Code and passage names

Pairing tags

Macro ownership

Pieces of a passage

Stateless checks and the built-in macros

Widget definitions

Variable references

Macro definitions

Formatting a passage

Diagnostics

Declarations

Effective widget definitions

Executable read and write roles

Completion contexts

Macro argument spans

Metadata-aware tokenization