Template Sandbox

A method's templates are code. PipeLLM prompts, PipeCompose templates and construct templates, PipeImgGen and PipeSearch prompts, PipeJudge prompts and questions, and PipeCondition expressions are all Jinja2 templates, and they render inside the process that runs the method. When that process is shared, as it is on a hosted runner, a template must not be able to reach anything but the data it was given.

Pipelex therefore renders every template, its own included, under one policy: a template reads data and calls methods of plain values, and nothing else. And because a template that only reads data can still ask for a billion-character string, every render also spends from a budget of work, and a render that would overdraw it is refused before it allocates.

What a template may do

What a template may not do

A method template renders without a template loader, so {% include %}, {% extends %} and {% import %} find nothing to load.

How a refusal shows up

Many refusals are visible in the template itself, and validation reports them before anything runs: a template that reads an undeclared underscore name with a dot fails with the template_private_name validation error, which names the pipe and the template (prompt, system_prompt, template, expression, or construct field '<path>').

What validation cannot see, because it depends on the values at run time (a method call, a name computed while rendering, or a bracketed key, which is legitimate on a plain dict), is refused where it happens: the render raises Jinja2TemplateSecurityError, with the code security.template_refused, whose message names the type the refused name was reached on, and the attribute or callable only when the template spells it, as in {{ record._secret }}, and which is located on the pipe and on the field its template is written in, as the bundle writes it: template, or template.template for a compose template written as a table of its own, prompt, system_prompt, negative_prompt, a condition's expression or expression_template, a judgment's question or questions.<name>.question, a document's filename, template or template_file, or a construct's construct.<field>, which each construct field locates itself, since a construct has no template field. The sandbox raises SandboxRefusalError, which words the refusal with the names it quotes and without them and carries the bare name it refused, and the render keeps the wording with the names, and passes the refused name on as the problem's name, only when the parsed template spells every name the refusal quotes: as a variable, an attribute or a literal key, or in a replacement field of the format string a .format or .format_map call formats, such as the _x that '{0._x}'.format(v) reads. That scan runs while the refusal is built, so it is bounded: it follows fields nested in format specs only as deep as the formatter reads them, and a format string nested deeper, or one that does not parse, counts nothing, so the refusal always comes out as itself. No other text of the template counts, neither a word another string literal holds, as mode == 'debug _token' does, nor the plain text outside its expressions, so a name the run's data chose stays withheld even when the template happens to write the same word; the render budget names the call it refused by the same rule. A template whose own source names what it reached for, such as {{ topic['_owner'] }}, gives a problem naming the pipe and the name, A template of pipe 'say' reached for '_owner', which the template sandbox forbids, with the step Remove '_owner' from the templates of pipe 'say'. A template that looks the name up from the run's data, such as {{ topic[topic.text] }}, would carry the run's input in that name, so its message says A template may not read an unsafe name on a 'StuffArtefact' value, its problem carries no name, and its sentences say what happened without it: A template of pipe 'say' looked a name up from the run's data, and the template sandbox refused that name, with the step to change the templates so that the run's data no longer chooses the names they look up. security.template_refused requires name, which this case withholds by rule until the standard words it, so no raise site is at fault and the problem logs no missing parameter. A refusal never renders as an empty string, so a prompt is never sent with a hole in it.

Jinja2TemplateSecurityError is a SecurityError, like UriReadRefusedError, SsrfBlockedError and UnsafeSchemaError, the other policy refusals that are never absorbed into a domain answer. A host that treats that family alike, never retrying any of it, catches that base class, and an except ToolError around a render does not catch this refusal. The sandbox's budget refusal is a separate error, Jinja2TemplateBudgetError, which is a ToolError (see below). Jinja2TemplateSecurityError stays a type of its own, apart from Jinja2TemplateRenderError, which a template that fails for any other reason raises.

A render that would overdraw its budget raises Jinja2TemplateBudgetError, with the code pipe.template_budget_exceeded, whose message names the operation that was refused and what it would have spent, never the template or a value: for example, the operator '*' needs about 1,000,000,000, and 134,217,062 are left. A refused call names the method it called only when the template spells that name, as in {{ note.center(width) }}; a method the template reaches by a value, as in {{ (note | attr(key))(width) }}, is described without its name, calling a method of a 'str' value. A template whose macros or includes nest deeper than Python allows raises the same error. It is a ToolError, not a SecurityError: a budget is a limit on what a render may spend rather than a policy refusal, so it is handled by its code like any other fault of the method.

A template that names a filter or a test Jinja2 does not know inside a conditional branch compiles, and only the render of that branch finds the name. The render raises Jinja2TemplateInvalidError, with the code method.template_invalid, whose message names the unknown name and never quotes the template. A PipeCondition expression and a PipeLLM prompt answer it as a fault of the method, located on the pipe and the field, with the code's step for the method's author, which the caller view carries as well.

These codes describe a template of the method, which is the caller's to fix and which an API answers with HTTP 422. A template Pipelex itself renders, such as one of its own prompts, is no fault of the caller's: the same refusals take runtime.unexpected there, and a budget overdrawn on the caller's data takes input.too_large.

How it is built

Every environment comes from one factory, make_jinja2_env_from_loader in pipelex/tools/jinja2/jinja2_environment.py, and that factory builds a PipelexTemplateEnvironment (pipelex/tools/jinja2/jinja2_sandbox.py), a subclass of Jinja's ImmutableSandboxedEnvironment. Rendering, parsing and compiling all go through it, including the condition pipe's compile at load and the load-time template walkers.

Jinja's stock sandbox refuses internals, but not public methods, and every pydantic model has public constructors. Under the stock sandbox a template could call model_validate on a content object and forge an image pointing at a storage key of its choosing. The Pipelex policy closes that by allowing calls by kind rather than by name:

• Jinja's own runtime: macros, caller, loop, cycler, joiner, block references, and the environment's globals.
• Methods a plain value type defines, bound to an instance of that type. A subclass of a plain type (a StrEnum member is a str) cannot add or override a callable method, and a class method is refused because it is bound to the class. The methods allowed are an audited list, PLAIN_VALUE_METHOD_COSTS in pipelex/tools/jinja2/jinja2_render_costs.py, which also gives each method's cost: a method that changes a list, a dict or a set in place, str.encode and int.to_bytes are left out, each with its reason in REFUSED_PLAIN_VALUE_METHODS, and a method a new Python release adds to a plain type is refused until it is classified.
• The sandbox's own str.format wrapper, which resolves replacement fields through the same attribute and item checks. The raw str.format stays refused.
• The methods a type declares as its template surface (pipelex/tools/jinja2/template_surface.py).

A type declares its template surface as a __template_surface__ class attribute: the methods a template may call on an instance and the underscore names it may read. The sandbox reads the declaration from the type, never from the instance. StuffArtefact, the adapter every input is wrapped in, declares get, iter_keys, iter_items and iter_values, and its four metadata fields. The Stuff it wraps is not one of its attributes at all: Python code reaches it through unwrap_stuff_artefact, and its bracket access and get resolve a key to a content field or a metadata field only.

The sandbox refuses a bracketed name before trying obj[name]. Jinja only checks a bracketed name when the item lookup fails, so an object whose __getitem__ answered any key would otherwise hand over what the dot spelling could not. A plain dict is the exception, because its items are data: a key it holds is returned, and a key it does not hold falls back to an attribute read, which the underscore rule refuses.

Filters are not calls in this sense: they are Pipelex's or Jinja's own code, registered by the environment. That makes each filter trusted code with one obligation: a filter never calls a callable it was handed, since that callable came from the template's values and the policy never vetted it. So with_images, format and tag call a value's rendering method only when the value's class defines it, and they check that on the type (pipelex/tools/jinja2/renderable_dispatch.py). A runtime-checkable Protocol alone would not do: on Python 3.11 it accepts a namespace() holding a callable of the template's choosing.

Jinja's own filters and markupsafe keep that obligation only for values whose attributes the template cannot choose: they look __html__ up on a value when escaping, and items in dictsort and xmlattr, and call what they find. A namespace() answers any name with whatever the template stored under it, so the environment's namespace holds data only, and storing a callable in one, at construction or with {% set ns.x = … %}, is refused.

The render budget

The sandbox limits what a template can reach, not what it can spend, and a template needs no forbidden call to be expensive: {{ 'x' * 10 ** 9 }}, a width in '%1000000000s' % x, the center, batch or format filters, a string doubled with ~ in a loop, two empty nested loops and a recursive macro each spend what the template asks. Rendering runs in the process that runs the method, on its event loop and sometimes inside a Temporal workflow, so on a shared runner one method's template would stall or exhaust a process that serves others. A timeout cannot help: a render never yields to the event loop, so nothing can interrupt it.

So every render gets a budget of work units, roughly one byte read or allocated each, and every operation spends from it. The default, DEFAULT_RENDER_BUDGET_UNITS in pipelex/tools/jinja2/jinja2_render_budget.py, is 128 Mi units: it prints about a hundred megabytes of text, and in the measurements behind it the slowest refusal of a hostile template took about half a second. A template included by another spends from its includer's budget, and a macro from its caller's, unless it was imported without with context (see below).

Each operation is charged for its inputs before it runs and for its result after. An operation whose result can outgrow its inputs, such as *, **, %, a format width, center, ljust, replace, join, batch, wordwrap or lipsum, is estimated first and refused before it allocates anything. An operation that reads a fixed part of its value, such as length, first, default or dict.get, is charged a step and its arguments, whatever the size of the value. An object of the run's data, such as a text content, is converted to count its text when an estimate needs it, so repeating it (([doc] * 20000) | join) is refused like repeating a string. str.format charges each replacement field as it formats it, with its spec resolved, so a width passed as an argument or through format_map counts. A filter that orders or deduplicates its input, sort, dictsort, groupby, min, max or unique, is charged for its comparisons, each weighing the heaviest key, read the way the filter reads it, and so are a list's or a tuple's count and index. Work that grows faster than an operation's result, such as striptags, wordwrap on long words or sum over lists, is spent at every call, not only checked. Integer arithmetic stops at 64 bits, which also keeps a template from building the huge integers whose hashing and printing cost the square of their size. Jinja's own cap on range still applies: a range of more than a hundred thousand items fails the render with Jinja2TemplateRenderError.

The environment charges through the sandbox's hooks and one rewrite, all in PipelexTemplateEnvironment. They rely on internals of Jinja 3.1, which is why Pipelex requires a Jinja below 3.2.

• Calls and operators go through the sandbox's call and call_binop hooks, and every arithmetic operator is intercepted.
• Filters and tests are wrapped before a template compiles. A filter or a test registered on the environment must have a cost in FILTER_COSTS or TEST_COSTS (pipelex/tools/jinja2/jinja2_render_costs.py), or the first compile fails with a TypeError naming it.
• Printed values go through the environment's finalize, which charges the text it produces. A caller's own finalize is passed to the constructor and runs first.
• What no hook reaches, which is loop iteration, ~, comparisons, slices, list, tuple and dict literals, the template's static text and a {% filter %} block's result, is routed through internal filters by a rewrite of the parsed template (pipelex/tools/jinja2/jinja2_render_rewrite.py). The rewrite changes what is charged, never what a template that renders without it renders. A tuple used as a key, in a dict literal or an item read, is also charged for hashing, since a tuple hashes every element again each time.
• A filter block's result is printed as a {{ }} tag prints a value. Jinja writes it out as it is, with neither finalize nor a conversion to text, so a filter returning something other than text, such as {% filter length %}, would otherwise fail the render. The rewrite hands the result to an internal filter that returns text as it is, unescaped under autoescape as Jinja leaves it, and passes any other result through finalize, the caller's own included, then converts it as a printed value is. Either way the result is charged as a print.
• Converting Markdown to HTML is charged where it happens, in pipelex/tools/markdown/markdown_parser.py, since markupsafe converts a Markdown value through its __html__ wherever an HTML template prints, escapes, joins or formats one, and no hook sees that call. The template makes its budget the active one while it renders, and the conversion charges it 2,048 units for every character of its source and every cell of its tables before it parses, then checks that its output, bounded from the parsed tokens, fits what is left before it renders, since a reference link is printed at every use. A conversion costs microseconds a character, so a render converts at most about 65,000 characters of Markdown, a document printed twice counting twice. Formatting Markdown for a document engine, which the markdown filter of plain-data templates does, is charged the same way, in pipelex/tools/markdown/markdown_formatting.py: both conversions parse through charged_parse, which holds the order of the charges.

The budget is the same for every template category and is not a setting in pipelex.toml: it is a safety bound, not a tuning knob. PipelexTemplateEnvironment takes it as the render_budget_units constructor argument, which the tests use to exercise small budgets.

Paths outside templates

A method also writes dotted paths that are not templates: a PipeCompose construct field's from and its list_to_dict_keyed_by, a PipeBatch's list, and the image and document references of a prompt. The runtime walks these with getattr, so the same rule applies to them: a segment starting with an underscore is refused. A construct's from path and list_to_dict_keyed_by are refused when the bundle is validated, and every path is refused again when it is walked.

What the sandbox does not cover

• What a whole run spends. The budget is per render. A method that renders many templates, or a batch that renders one per item, spends one budget per render.
• Macros imported without context. A macro imported with {% import %} or {% from … import %} and no with context spends from the budget of the module Jinja builds for the import, which Jinja caches on the imported template. Pipelex builds a new environment for every render, so that budget never outlives one render, and a customer template has no loader to import from.
• The URLs values carry. A template cannot forge content under this policy, but a value can still carry a URL, and reading it is the storage layer's decision, not the template's.

Related

• StuffArtefact & Image Rendering: how inputs reach templates, and how images are extracted from them.
• Jinja2TemplateSecurityError: the error a refusal raises at render time.
• Jinja2TemplateBudgetError: the error a render raises when it would overdraw its budget.
• Jinja2TemplateInvalidError: the error a render raises when a template names a filter or a test that does not exist, or cannot otherwise render as written.