PipeLLM

PipeLLM is the core operator in Pipelex for leveraging Large Language Models (LLMs). It can be used for a wide range of tasks, including text generation, summarization, classification, and structured data extraction.

How it works

At its core, PipeLLM constructs a detailed prompt from various inputs and templates, sends it to a specified LLM, and processes the output. It can produce simple text or complex structured data (in the form of Pydantic models).

For structured outputs, you have two options:

1. Direct (default) — PipeLLM asks the LLM to fill the target Pydantic schema in a single call. Fast and cheap; relies on the model's structured-output quality.
2. Preliminary text — set structuring_method = "preliminary_text". Pipelex rewrites the pipe at load time into a PipeSequence of two pipes: a PipeLLM that produces free-form text, then a PipeStructure that turns that text into the target concept. Two LLM calls per invocation, but the prose stage is unconstrained by JSON. See Structuring Method (preliminary_text) below.

If you already have text from elsewhere (a PDF extraction, a search result, an upstream pipe), call PipeStructure directly — there's no need to wrap a PipeLLM around it.

Every input the pipe declares must be read by prompt or system_prompt, and every variable they read must be declared in inputs. Validation refuses an input neither prompt reads as extraneous_input_variable, naming the input, so you either reference it in a prompt or remove it from inputs; an undeclared variable is refused as missing_input_variable. A name a prompt sets with {% set %} is its own, not an input, from that statement on: after an {% if %} it stays set only when every branch sets it, the {% else %} included, so a name only some branches set is read from the input of that name on the other paths; and a loop, a macro or a block keeps what it sets to its own body.

Working with Images (Vision Language Models)

PipeLLM supports Vision Language Models (VLMs) that can process both text and images. To use images in your prompts:

Basic Image Input

Images must be declared in the inputs section of your pipe definition. The image will be automatically passed to the VLM along with your text prompt.

[pipe.describe_image]
type = "PipeLLM"
description = "Describe an image"
inputs = { image = "Image" }
output = "VisualDescription"
prompt = """
Describe the provided image in great detail: $image
"""

Flexible Image Inputs

You can use any concept that refines Image as an input, and choose descriptive variable names that fit your use case:

[pipe.analyze_wedding]
type = "PipeLLM"
description = "Analyze wedding photo"
inputs = { wedding_photo = "Photo" }
output = "PhotoAnalysis"
prompt = """
Analyze this wedding photo and describe the key moments captured: $wedding_photo
"""

Images as Sub-attributes of Structured Content

When working with structured content that contains image fields (like Page, which has a page_view field), declare the structured value with its whole concept in inputs, and read the image field through it in the prompt:

[pipe.analyze_page_view]
type = "PipeLLM"
description = "Analyze the visual layout of a page"
inputs = { page = "Page" }
output = "LayoutAnalysis"
prompt = """
Analyze the visual layout and design elements of this page:
@page.page_view

Focus on typography, spacing, and overall composition.
"""

In this example:

• page is the input variable, declared with its whole concept, Page
• page_view is the Image field within the Page structure
• The path page.page_view reads that field through page, and because the concept the path reaches is Image, Pipelex attaches it to the model call as an image

An input name is always a plain name: a key such as "page.page_view" = "Image" is refused as invalid_input_name. When a pipe should receive only the field, under a name of its own, the calling sequence binds the field with a binding step, { from = "page.page_view", result = "page_view" }, and the pipe declares page_view = "Image".

Multiple Images

You can include multiple images in a single prompt by listing them in the inputs:

[pipe.compare_images]
type = "PipeLLM"
description = "Compare two images"
inputs = { 
    first_image = "Image",
    second_image = "Image"
}
output = "ImageComparison"
prompt = """
Compare these two images and describe their similarities and differences: $first_image and $second_image
"""

Combining Text and Image Inputs

You can mix any stuff and image inputs in the same pipe:

[pipe.analyze_document_with_context]
type = "PipeLLM"
description = "Analyze a document page with additional context"
inputs = { 
    context = "Text",
    document.page_view = "Image"
}
output = "DocumentAnalysis"
prompt = """
Given this context: $context

Analyze the document page shown in the image and explain how it relates to the provided context: $document.page_view
"""

Working with Documents

PipeLLM supports including documents (PDFs, etc.) in prompts. Documents are passed to the LLM along with your text prompt, allowing the model to analyze their content.

Basic Document Input

Documents must be declared in the inputs section of your pipe definition. The document will be automatically passed to the LLM along with your text prompt.

[pipe.summarize_document]
type = "PipeLLM"
description = "Summarize a document"
inputs = { document = "Document" }
output = "DocumentSummary"
prompt = """
Summarize the key points from this document:

@document
"""

Flexible Document Inputs

You can use any concept that refines Document as an input, and choose descriptive variable names that fit your use case:

[concept.FinancialReport]
description = "A financial report"
refines = "Document"

[pipe.analyze_report]
type = "PipeLLM"
description = "Analyze financial report"
inputs = { financial_report = "FinancialReport" }
output = "ReportAnalysis"
prompt = """
Analyze this financial report and extract the key metrics: $financial_report
"""

Multiple Documents

You can include multiple documents in a single prompt by listing them in the inputs:

[pipe.compare_documents]
type = "PipeLLM"
description = "Compare two documents"
inputs = { first_doc = "Document", second_doc = "Document" }
output = "DocumentComparison"
prompt = """
Compare these two documents and describe their similarities and differences: $first_doc and $second_doc
"""

For a variable number of documents, use the list syntax:

[pipe.merge_documents]
type = "PipeLLM"
description = "Merge multiple documents"
inputs = { documents = "Document[]" }
output = "MergedContent"
prompt = """
Combine the information from all these documents into a single coherent summary: $documents
"""

Combining Text, Images, and Documents

You can mix text, image, and document inputs in the same pipe:

[pipe.analyze_with_context]
type = "PipeLLM"
description = "Analyze a document with context"
inputs = {
    context = "Text",
    screenshot = "Image",
    reference_doc = "Document"
}
output = "ContextualAnalysis"
prompt = """
Given this context: $context

And this screenshot for reference: $screenshot

Analyze the document and explain how it relates to the context: $reference_doc
"""

Dynamic Inputs Render as Text

A native.Dynamic input referenced in a prompt renders as text through the template, exactly as it does in a PipeCompose template. Dynamic is compatible with every concept by design, but compatibility is not identity: the prompt analyzers only attach a variable as an image or document when it is definitely one, and a dynamic concept's shape is unknowable statically. If you want image or document attachment, declare Image / Document (or a concept refining them) instead. Consequently, | with_images on a Dynamic variable is refused — statically unknowable nested images are refused, not guessed.

This holds even when the value you supply at run time is an image. A Dynamic slot does not force DynamicContent: the input shaper hands the raw value to the bottom-up factory, so passing an ImageContent (or an explicit {"concept": "native.Image", ...} form) into a Dynamic slot produces a native.Image stuff. That image is still not attached to the prompt, because attachment is decided from the declared concept, not from what the caller happened to pass. Declare the slot Image when you want the image to reach the model.

The same rule governs PipeImgGen prompts (positive and negative), which use the same prompt analyzer.

Configuration

PipeLLM is configured in your pipeline's .mthds file.

MTHDS Parameters

LLM Setting Fields

When model is specified as a table (inline LLM setting), it accepts the following fields:

A setting the model it resolves to cannot take is refused when the method loads, before a run spends anything: validation runs the check the model's worker runs before every call, so a reasoning setting on a model that does not reason, a reasoning budget on a model taking only an effort, or a budget its max_tokens cannot hold is an llm_setting_refused_by_model error on the model or model_to_structure field holding the setting. The setting checked is the one the output is generated with: model for a single text, the structuring setting for anything else.

A text the model cannot finish within max_tokens, or within its context window, fails the pipe with LLMCompletionTruncatedError rather than passing the partial text on as the pipe's output, and so does a structured output cut at max_tokens, which the error names as a structured output; a text the model refuses or a safety filter stops fails it with LLMCompletionRefusedError. The first names the output tokens used and the limit sent, so raising max_tokens or shortening the input is the fix; a model that reasons spends its reasoning from the same limit. On an Anthropic model, a structured output's max_tokens is held to what the structured-output timeout allows, so when that lowered limit cut it the error says so and its fix is a lower reasoning effort or a smaller output, never a higher max_tokens. A request the provider refuses outright because the prompt plus the output it requests with max_tokens exceed the model's context window fails with the code model.context_exceeded: shortening the input or lowering max_tokens, which may be the model's default when the pipe sets none, or choosing a model with a larger context window, fixes it.

A prompt or system_prompt that names a filter or a test Jinja2 does not know inside a conditional block loads, because Jinja2 resolves such a name only when the block renders. The run then fails at the pipe, before any model is called, with the code method.template_invalid: the message names the field and the unknown name, quotes none of the prompt, and comes with a next step, and a hosted run answers it with HTTP 422.

Output Multiplicity

Specify output multiplicity using bracket notation in the output field:

• Single output (default): output = "Text" - generates exactly one item
• Variable output: output = "Text[]" - lets the LLM decide how many items to generate
• Fixed output: output = "Text[5]" - generates exactly 5 items

Examples:

# Single output (default)
output = "Summary"

# Variable - extract all keywords found
output = "Keyword[]"

# Fixed - generate exactly 3 alternatives
output = "Headline[3]"

Structuring Method (preliminary_text)

By default, PipeLLM asks the LLM to produce the target structured output in a single call. For some tasks — especially complex extractions from images or documents where the model thinks better in prose than in JSON — you can opt into a two-stage approach:

[pipe.review_restaurant]
type = "PipeLLM"
description = "Write a structured review of a restaurant from a transcript"
inputs = { transcript = "Text" }
output = "RestaurantReview"
structuring_method = "preliminary_text"
prompt = """
Write a thorough review of this restaurant based on the transcript:

@transcript
"""

What happens at load time

When Pipelex parses a .mthds file with structuring_method = "preliminary_text", it rewrites the pipe before any execution into a PipeSequence:

1. Step 1 — <pipe_code>__draft_text: a PipeLLM that inherits your original inputs, system_prompt, prompt, and model. Its output is always a single Text. This is the "write the prose" call.
2. Step 2 — <pipe_code>__structure: a PipeStructure that takes the draft text and produces your declared output (preserving multiplicity — Foo, Foo[], or Foo[N]). It uses model_to_structure if set, otherwise the default LLM choice for object generation.

The wrapping PipeSequence keeps your original pipe_code, so anything that calls or references your pipe (other pipes, main_pipe, the run API) keeps working unchanged.

Trade-offs

• Two LLM calls per invocation instead of one — slower and more expensive.
• Better quality on complex schemas, especially with vision models: the prose stage isn't constrained by the JSON schema, then a focused structuring call extracts the typed result.
• Independent model choice for the structuring step via model_to_structure — pair an expensive vision model for step 1 with a cheap fast model for step 2.
• Output cannot be Text — there's nothing to structure. Pipelex rejects this at load time.

When to choose what

• Stick with direct for most structured outputs — modern models handle them well and one call is always cheaper.
• Switch to preliminary_text when you're extracting from images or PDFs and a single-call structured output keeps missing fields, or when your schema is deep and the model produces better content in prose first.
• Use PipeStructure directly when the text already exists (extracted, searched, generated upstream).

Examples

Simple Text Generation Example

This pipe takes no input and writes a poem.

[pipe.write_poem]
type = "PipeLLM"
description = "Write a short poem"
output = "Text"
model = "$writing-creative"
prompt = """
Write a four-line poem about pipes.
"""

Text-to-Text Example

This pipe summarizes an input text, using a prompt to inject the input.

[pipe.summarize_text]
type = "PipeLLM"
description = "Summarize a text"
inputs = { text = "TextToSummarize" }
output = "TextSummary"
prompt = """
Please provide a concise summary of the following text:

@text

The summary should be no longer than 3 sentences.
"""

Vision (VLM) Example

This pipe takes an image of a table and uses a VLM to extract the content as an HTML table.

[pipe.extract_table_from_image]
type = "PipeLLM"
description = "Extract table data from an image"
inputs = { image = "TableScreenshot" }
output = "TableData"
prompt = """
Extract the table data from this image and format it as a structured table: $image
"""

Structured Data Extraction Example

This pipe extracts a list of Expense items from a block of text.

[concept.Expense]
structure = "Expense" # Assumes a Pydantic model 'Expense' is defined

[pipe.process_expense_report]
type = "PipeLLM"
description = "Process an expense report"
inputs = { report = "ExpenseReport" }
output = "ProcessedExpenseReport"
prompt = """
Analyze this expense report and extract the following information:
- Total amount
- Date
- Vendor
- Category
- Line items

@report
"""

In this example, Pipelex will instruct the LLM to return a list of objects that conform to the Expense structure.

Reasoning Example

This pipe uses extended reasoning to analyze a complex topic in depth:

[pipe.deep_analysis]
type = "PipeLLM"
description = "Analyze a complex topic with extended reasoning"
inputs = { topic = "Text" }
output = "Analysis"
model = { model = "claude-4.5-sonnet", temperature = 0.1, reasoning_effort = "high" }
prompt = """
Analyze the following topic in depth, considering multiple perspectives:

@topic
"""

You can also reference a reasoning preset:

[pipe.deep_analysis]
type = "PipeLLM"
description = "Analyze a complex topic with extended reasoning"
inputs = { topic = "Text" }
output = "Analysis"
model = "$deep-analysis"
prompt = """
Analyze the following topic in depth, considering multiple perspectives:

@topic
"""

Related Documentation

• The MTHDS Language Tutorial - Starts from a single PipeLLM and builds a method step by step
• Invoice extraction - A method in the method library that extracts structured invoice data using both the OCR text and the page view
• Tweet optimizer - A method in the method library that scores a draft tweet and rewrites it in your style
• Table extraction - A method in the method library that extracts a table from a screenshot into HTML, then reviews it against the image
• Gantt chart extraction - A cookbook method that returns every task and milestone of a Gantt chart image with their dates