# Pipelex Documentation

> Official documentation for Pipelex, an open-source library/CLI for building and running executable AI methods.

# Get Started

## Build & Run AI Methods

![Pipelex Banner](https://d2cinlfp2qnig1.cloudfront.net/banners/pipelex_banner_docs_v2.png)

# Build & Run AI Methods

A method is a reusable, typed AI procedure — declared in a `.mthds` file and executed by Pipelex. Each step is explicit, each output is structured, and every run is repeatable.

[:material-robot-outline: Claude Code](./get-started/build-with-claude-code.md){ .md-button .md-button--primary }
[:material-school: MTHDS Tutorial](./get-started/mthds-language-tutorial.md){ .md-button }
[:material-book-open-variant: Cookbook](./cookbook/index.md){ .md-button }

---

## Why Methods?

<div class="grid cards" markdown>

-   :material-file-document-check: **Declarative**

    Express business logic at a high level of abstraction, in human-readable `.mthds` files that work across models.

-   :material-shape-outline: **Typed**

    Concepts are semantic types: AI understands what you mean, and every input and output connects with purpose.

-   :material-refresh: **Repeatable**

    Deterministic orchestration that leaves exactly the room you want for AI to express its intelligence and creativity.

-   :material-puzzle: **Composable**

    Chain pipes into sequences, nest methods inside methods, and share them with the community.

</div>

---

## What a Method Looks Like

A single pipe in MTHDS — five lines that call an LLM with typed inputs and output:

```toml
[pipe.summarize_article]
type    = "PipeLLM"
inputs  = { article = "Text", audience = "Text" }
output  = "Text"
prompt  = "Summarize $article in three bullet points for $audience."
```

From here, Pipelex handles model routing across 60+ models, structured output parsing, and pipeline orchestration.

---

## Capabilities

<div class="grid cards" markdown>

-   :material-shape-outline: **[Typed Concepts](./features/concepts.md)**

    Semantic types that give meaning to every input and output — native, inline, or backed by Python classes.

-   :material-pipe: **[Pipe Operators](./features/pipe-operators.md)**

    Operators that do the work: LLM calls, text structuring, document extraction, image generation, web search, composition, and custom functions.

-   :material-sitemap: **[Pipeline Orchestration](./features/pipeline-orchestration.md)**

    Sequence, parallel, batch, and conditional controllers that wire pipes into full methods with shared working memory.

-   :material-cloud-check: **[60+ AI Models](./features/llm-integration.md)**

    One gateway key or bring-your-own: OpenAI, Anthropic, Mistral, Google, Deepseek, Hugging Face, and more.

-   :material-check-decagram: **[Validation and Dry Run](./features/validation-dry-run.md)**

    Validate pipelines before execution and dry-run with mocked responses — catch errors without spending tokens.

-   :material-console: **[CLI and Tooling](./features/cli.md)**

    Full CLI for init, build, validate, run, and graph visualization. Plus `plxt` for formatting and linting `.mthds` files.

</div>

---

## The MTHDS Ecosystem

MTHDS is the open standard behind Pipelex methods. It defines the language, the file format, and the ecosystem for sharing methods.

!!! info "Explore the ecosystem"

    - **[mthds.ai](https://mthds.ai/latest/)** — The MTHDS language specification
    - **[mthds.sh](https://mthds.sh)** — The Methods Hub for discovering and sharing methods
    - **[MTHDS Plugins](https://github.com/mthds-ai/mthds-plugins)** — Claude Code plugin for building, running, and validating methods

---

## Get Started

<div class="grid cards" markdown>

-   :material-robot-outline: **[Build with Claude Code](./get-started/build-with-claude-code.md)**

    Describe what you want in natural language — Claude writes, runs, and iterates on your method for you.

-   :material-school: **[MTHDS Language Tutorial](./get-started/mthds-language-tutorial.md)**

    Learn the declarative language step by step: concepts, pipes, sequences, inputs, and structured outputs.

-   :material-book-open-variant: **[Cookbook Examples](./cookbook/index.md)**

    Production-ready recipes — from Hello World to document extraction, synthetic data, and image generation.

</div>



## Build with Claude Code

# Build with Claude Code

Install the `mthds` CLI:

```bash
npm install -g mthds
```

Launch Claude Code and tell it to install the MTHDS plugin:

```
/plugin marketplace add mthds-ai/mthds-plugins
```

```
/plugin install mthds@mthds-plugins
```

then reload plugins:

```
/reload-plugins
```

If that doesn't work, exit Claude Code and reopen it:

```
/exit
```

Build your first method:

```
/mthds-build A method to analyze a Job offer to build a scorecard, then batch process CVs to score them
```

Run it:

```
/mthds-run
```

That's it — Claude writes the `.mthds` file, creates inputs, and runs the method for you. See the [Claude Code Skills Plugin](../features/claude-code-skills-plugin.md) documentation for the full list of commands (`/mthds-edit`, `/mthds-check`, `/mthds-fix`, `/mthds-explain`, and more).

See [Configure AI Providers](./configure-ai-providers.md) for other options: bring your own keys, local AI, etc.

![Claude Code + Pipelex + MTHDS](https://raw.githubusercontent.com/Pipelex/pipelex/main/.github/assets/Claude-Code-Pipelex-MTHDS-Cursor.png)


## The MTHDS Language Tutorial

# The MTHDS Language Tutorial

This tutorial walks you through writing `.mthds` files manually, step by step.

Set up your project and get free AI access:

```bash
uv tool install pipelex
pipelex init
pipelex login
```

`pipelex init` creates your project configuration. `pipelex login` opens your browser to authenticate with Pipelex Gateway (free). See [Configure AI Providers](./configure-ai-providers.md) for other options: bring your own keys, local AI, etc.

!!! tip "VS Code Extension"
    We **highly** recommend installing the Pipelex extension for `.mthds` syntax highlighting and flowchart visualization:

    - **VS Code**: Install from the [VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=pipelex.pipelex)
    - **Cursor, Windsurf, and other VS Code forks**: Install from the [Open VSX Registry](https://open-vsx.org/extension/Pipelex/pipelex)

    Running `pipelex init` will also offer to install the extension automatically if it detects your IDE.

## Step 1: Hello World

Create a file called `hello_world.mthds`:

```toml
domain      = "tutorial_hello_world"
description = "Your first Pipelex pipeline"
main_pipe   = "tutorial_hello_world"

[pipe]

[pipe.tutorial_hello_world]
type = "PipeLLM"
description = "Generate a creative story idea"
output = "Text"
prompt = """
Generate a one-paragraph creative story idea about a robot learning to paint.
"""
```

Run it:

```bash
pipelex run bundle hello_world.mthds
```

Here's what each part means:

- **`domain`** — a namespace for this bundle (like a Python package name)
- **`main_pipe`** — the entry point pipe that runs when you execute the bundle
- **`PipeLLM`** — a pipe type that makes an LLM call
- **`output = "Text"`** — the pipe returns plain text (a native concept)
- **`prompt`** — the prompt sent to the LLM

## Step 2: Chaining LLM Calls

Most real methods need multiple steps. `PipeSequence` chains pipes together, passing data between them.

Create `chaining_llm_calls.mthds`:

```toml
domain      = "chaining_llm_calls"
description = "Chain multiple LLM calls together"
main_pipe   = "generate_and_expand"

[pipe]

# First pipe: Generate a story idea
[pipe.chain_generate_idea]
type = "PipeLLM"
description = "Generate a creative story idea"
output = "Text"
prompt = """
Generate a one-paragraph creative story idea about a robot learning to paint.
"""

# Second pipe: Expand the story idea (uses result from first pipe)
[pipe.chain_expand_idea]
type = "PipeLLM"
description = "Expand a story idea into a detailed outline"
inputs = { story_idea = "Text" }
output = "Text"
prompt = """
Take this story idea and expand it into a 3-act outline:

@story_idea

Provide a brief description for each act.
"""

# PipeSequence: Chain the two pipes together
[pipe.generate_and_expand]
type = "PipeSequence"
description = "Generate a story idea then expand it"
output = "Text"
steps = [
  { pipe = "chain_generate_idea", result = "story_idea" },
  { pipe = "chain_expand_idea", result = "story_outline" },
]
```

Run it:

```bash
pipelex run bundle chaining_llm_calls.mthds
```

Key concepts:

- **`PipeSequence`** — runs pipes in order, passing results between them
- **`steps`** — each step names a pipe and stores its output in a `result` variable
- **`@story_idea`** — references data from a previous step's result in the prompt

## Step 3: Using Inputs

Instead of hardcoding everything in the prompt, you can pass data in at runtime.

Create `using_inputs.mthds`:

```toml
domain      = "using_inputs"
description = "Learn how to pass inputs to your pipeline"
main_pipe   = "write_about_topic"

[pipe]

# A pipe that takes an input and uses it
[pipe.write_about_topic]
type = "PipeLLM"
description = "Write a short paragraph about a given topic"
inputs = { topic = "Text" }
output = "Text"
prompt = """
Write a short, engaging paragraph about the following topic:

$topic

Keep it under 100 words.
"""
```

Create an `inputs.json` file:

```json
{
  "topic": "Photosynthesis"
}
```

Run it with the `-i` flag:

```bash
pipelex run bundle using_inputs.mthds -i inputs.json
```

Key concepts:

- **`inputs = { topic = "Text" }`** — declares what the pipe expects as input
- **`$topic`** — inserts the variable inline, right inside a sentence. The difference from `@variable` is rendering shape, not where the data comes from: `@variable` is the block form — it must sit alone on its own line and inserts the content as a tagged block (which is why `@story_idea` stood on its own line in Step 2) — while `$variable` is the inline form for weaving a value into surrounding text. Both resolve against the pipe's declared inputs
- **`@@` and `$$`** — write a literal `@` or `$` without triggering interpolation. Use `@@font-face` or `@@media` inside `<style>` blocks, `$$10` for a literal dollar amount before a word, and so on. The escape is per-occurrence.
- **`-i inputs.json`** — provides input data at runtime

## Step 4: Structured Outputs

LLMs can return structured data instead of plain text. Define a **concept** with a **structure**, and the LLM output will be validated against it.

Create `structured_output.mthds`:

```toml
domain      = "structured_output"
description = "Get structured data from LLMs"
main_pipe   = "struct_generate_book_idea"

[concept]

# Define the structure of a book idea
[concept.BookIdea]
description = "A book idea with title, genre, and synopsis"

[concept.BookIdea.structure]
title           = { type = "text", description = "The book title", required = true }
genre           = { type = "text", description = "The genre of the book", required = true }
synopsis        = { type = "text", description = "A brief synopsis of the book" }
target_audience = { type = "text", description = "Who this book is for" }

[pipe]

# Generate a structured book idea
[pipe.struct_generate_book_idea]
type = "PipeLLM"
description = "Generate a structured book idea"
output = "BookIdea"
prompt = """
Generate a creative book idea. Provide a compelling title, genre, synopsis, and target audience.
"""
```

Run it:

```bash
pipelex run bundle structured_output.mthds
```

Key concepts:

- **`[concept.BookIdea]`** — declares a typed concept with a description
- **`[concept.BookIdea.structure]`** — defines the fields and their types
- **`output = "BookIdea"`** — the pipe returns a structured `BookIdea` object instead of plain text

Learn more about concepts in [Define Your Concepts](../building-methods/concepts/define_your_concepts.md).

---

## Next Steps

You now understand the building blocks: pipes, sequences, inputs, and structured outputs.

- **Tutorial** — more lessons in the [Cookbook](../cookbook/index.md) (document extraction, batch processing, and more)
- **Build Reliable AI Methods** — deep dive into [method project design](../building-methods/kick-off-a-methods-project.md)
- **Pipe Operators & Controllers** — reference for [pipe operators](../building-methods/pipes/pipe-operators/index.md) (PipeLLM, PipeExtract, PipeCompose, ...) and [pipe controllers](../building-methods/pipes/pipe-controllers/index.md) (PipeSequence, PipeParallel, PipeBatch, PipeCondition)
- **Cookbook Examples** — real-world [examples and patterns](../cookbook/index.md)
- **CLI Reference** — full [command-line reference](../tools/cli/index.md)


## Configure AI Providers

# Configure AI Providers

## Configure API Access

To run pipelines with LLMs, you need to configure API access. **You have three options** - choose what works best for you:

### Option 1: Pipelex Gateway — Easiest and most Powerful for Getting Started

Get **free credits** for testing and development with a single API key for LLMs, document extraction, and image generation across all major providers (OpenAI, Anthropic, Google, Azure, open-source, and more). New models added constantly.

**Benefits:**

- No credit card required
- Access to OpenAI, Anthropic Claude, Google Gemini, xAI Grok, and more
- New models added constantly
- Perfect for development and testing
- Single API key for all models

**Setup:**

1. Get your API key at [app.pipelex.com](https://app.pipelex.com/)

2. Create a `.env` file in your project root:

    ```env
    PIPELEX_GATEWAY_API_KEY=your-key-here
    ```

3. Run `pipelex init` and accept the Gateway terms of service when prompted.

That's it! Your pipelines can now access any supported LLM. See [Gateway Available Models](../setup/gateway-models.md) for the full list.

!!! info "Terms of Service & Telemetry"
    When using Pipelex Gateway, you'll be prompted to accept our terms of service. By using the Gateway, identified telemetry is automatically enabled (tied to your hashed API key) to help us monitor service quality and enforce fair usage.

    **We collect only technical data** (model names, token counts, latency, error rates). We do **NOT** collect your prompts, completions, or business data. See [Telemetry](../setup/telemetry.md) for details and trade-offs, and our [Privacy Policy](https://go.pipelex.com/privacy-policy) for more.

### Option 2: Bring Your Own API Keys

Use your existing API keys from LLM providers. This is ideal if you:

- Already have API keys from providers
- Need to use specific accounts for billing
- Have negotiated rates or enterprise agreements
- Prefer not to send any telemetry to Pipelex servers

**Setup:**

Create a `.env` file in your project root with your provider keys:

```bash
# OpenAI
OPENAI_API_KEY=sk-...

# Anthropic
ANTHROPIC_API_KEY=sk-ant-...

# Google
GOOGLE_API_KEY=...

# Mistral
MISTRAL_API_KEY=...

# FAL (for image generation)
FAL_API_KEY=...

# XAI
XAI_API_KEY=...

# Azure OpenAI
AZURE_API_KEY=...
AZURE_API_BASE=...
AZURE_API_VERSION=...

# Amazon Bedrock
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=...
```

You only need to add keys for the providers you plan to use.

**Enable Your Providers:**

When using your own keys, enable the corresponding backends:

1. Initialize configuration:

    ```bash
    pipelex init
    ```

2. Edit `~/.pipelex/inference/backends.toml` (or `.pipelex/inference/backends.toml` in your project root if you have a project-local config — the project file fully overrides the global one):

    ```toml
    [google]
    enabled = true

    [openai]
    enabled = true

    # Enable any providers you have keys for
    ```

See [Inference Backend Configuration](../configuration/config-technical/inference-backend-config.md) for all options.

### Option 3: Local AI (No API Keys Required)

Run AI models locally without any API keys. This is perfect if you:

- Want complete privacy and control
- Have capable hardware (GPU recommended)
- Need offline capabilities
- Want to avoid API costs

**Supported Local Options:**

**Ollama** (Recommended):

1. Install [Ollama](https://ollama.ai/)
2. Pull a model from Pipelex's Ollama catalog: `ollama pull gemma3:4b`
3. No API key needed! Configure Ollama backend in `~/.pipelex/inference/backends.toml`

**Other Local Providers:**

- **vLLM**: High-performance inference server
- **LM Studio**: User-friendly local model interface
- **llama.cpp**: Lightweight C++ inference

Configure these in `~/.pipelex/inference/backends.toml`. See our [Inference Backend Configuration](../configuration/config-technical/inference-backend-config.md) for details.

---

## Backend Configuration Files

To set up Pipelex configuration files, run:

```bash
pipelex init
```

By default, this creates the global `~/.pipelex/` directory with:

```
~/.pipelex/
├── pipelex.toml              # Feature flags, logging, cost reporting
├── plxt.toml                 # MTHDS/TOML formatting and linting configuration
├── telemetry.toml            # AI trace destinations (PostHog, Langfuse, OTLP)
└── inference/                # LLM configuration and model presets
    ├── backends.toml         # Enable/disable model providers
    ├── backends/             # Per-provider model catalogs (anthropic.toml, openai.toml, ...)
    ├── deck/
    │   ├── 1_llm_deck.toml            # LLM presets and aliases
    │   ├── 2_img_gen_deck.toml        # Image generation config
    │   ├── 3_extract_deck.toml        # Document extraction config
    │   ├── 4_search_deck.toml         # Search config
    │   ├── x_custom_llm_deck.toml     # Custom LLM configurations
    │   └── x_custom_extract_deck.toml # Custom extract configurations
    └── routing_profiles.toml # Model routing configuration
```

To keep the configuration inside a project instead, run `pipelex init --local`: it creates the same structure in a `.pipelex/` directory at your project root, which takes precedence over the global `~/.pipelex/`.

Learn more in our [Inference Backend Configuration](../configuration/config-technical/inference-backend-config.md) guide.

---

## Next Steps

Now that you have your backend configured:

1. **Learn the concepts**: [MTHDS Language Tutorial](./mthds-language-tutorial.md)
2. **Explore examples**: [Cookbook Repository](https://github.com/Pipelex/pipelex-cookbook/tree/main)
3. **Deep dive**: [Build Reliable AI Methods](../building-methods/kick-off-a-methods-project.md)

!!! tip "Advanced Configuration"
    For detailed backend configuration options, see [Inference Backend Configuration](../configuration/config-technical/inference-backend-config.md).


# Cookbook Examples

## Overview

# Pipelex Cookbook Examples

Welcome to the Pipelex Cookbook!

[![GitHub](https://img.shields.io/badge/Cookbook_Repository-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/tree/main)

This is your go-to resource for practical examples and ready-to-use recipes to build powerful and reliable AI methods with Pipelex. Whether you're a beginner looking to get started or an experienced user searching for advanced patterns, you'll find something useful here.

We strongly encourage you to clone the [Pipelex Cookbook repository](https://github.com/Pipelex/pipelex-cookbook/tree/main), play with the examples, and tailor them to your specific needs.

## Quick Start

- [**Hello World**](./hello-world.md) — The simplest Pipelex pipeline: a single LLM call that generates a haiku.
- [**Summarize**](./summarize.md) — Multi-step text summarization with topic extraction and batching.

## Document Extraction

- [**Generic Document**](./extract-generic.md) — Extract markdown from complex PDFs using vision.
- [**Invoice**](./extract-invoice.md) — Extract structured invoice data with classification and vision analysis.
- [**Proof of Purchase**](./extract-proof-of-purchase.md) — Extract receipt data including product lists.
- [**DPE (Energy Diagnostic)**](./extract-dpe.md) — Specialized extraction for French energy performance certificates.
- [**Table from Image**](./extract-table.md) — Convert table images to HTML with self-correction review.
- [**Gantt Chart**](./extract-gantt.md) — Extract tasks and dates from Gantt chart images using divide-and-conquer.
- [**Markdown from Document**](./extract-markdown.md) — Simple document-to-markdown using shared method packages.
- [**Slides from Presentation**](./extract-slides.md) — Extract slide content and descriptions with vision analysis.

## Visual Generation

- [**Design Slides**](./design-slides.md) — Generate themed slide design proposals with visual mockups.
- [**Generate Image**](./generate-image.md) — Creative image generation through a three-step pipeline.

## Advanced

- [**Synthetic Data**](./gen-synthetic-data.md) — Generate diverse synthetic data samples from a description.
- [**Expense Data with Receipts**](./gen-expense-data.md) — Generate complete expense datasets with receipt images and fraud scenarios.
- [**Using Inference Plugins**](./using-inference-plugins.md) — Install an inference-backend plugin and run a method against the model it serves, with zero API keys.

## Work in Progress

These examples are under active development and may change:

- [**Tweet Optimizer**](./write-tweet.md) — Analyze and optimize tech tweets with style transfer.
- [**Blog Article Generator**](./blog-article-generator.md) — Generate SEO-optimized articles from structured requests.
- [**Advisory Board**](./advisory-board.md) — Multi-board business consultation with consensus analysis.
- [**Discord Newsletter**](./discord-newsletter.md) — Generate HTML newsletters from Discord channel data.
- [**Expense Validation**](./validate-expense-data.md) — Validate expense reports against company policies.

## Contribute

Have you built a cool pipeline with Pipelex? Share it with the community! We welcome contributions. Check out our [contribution guidelines](../contributing.md) and help us grow this collection of examples.


## Hello World

# Example: Hello World

This is the "Hello World" of Pipelex, a simple pipeline that demonstrates the basic concepts of Pipelex.

It's the perfect starting point to verify your installation and get a first taste of how Pipelex works.

## Get the code

[![GitHub](https://img.shields.io/badge/View_on_GitHub-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/a_quick_start/hello_world.mthds)

## What it demonstrates

- Minimal Pipelex setup with a single `PipeLLM` pipe
- The `main_pipe` field to designate the entry point
- Running a method bundle from the CLI

## The Method: `hello_world.mthds`

The method definition is extremely simple — a single LLM call that generates a haiku:

```toml
domain    = "quick_start"
main_pipe = "hello_world"

[pipe]
[pipe.hello_world]
type = "PipeLLM"
description = "Write text about Hello World."
output = "Text"
prompt = """
Write a haiku about Hello World.
"""
```

## How to run

1. Clone the cookbook repository and install dependencies:

    ```bash
    git clone https://github.com/Pipelex/pipelex-cookbook.git
    cd pipelex-cookbook
    pip install .
    ```

2. Set up your environment variables by copying `.env.example` to `.env` and adding your API keys.

3. Run the example:

    ```bash
    pipelex run bundle examples/a_quick_start/hello_world.mthds
    ```

Expected output: A haiku about "Hello World" displayed with pretty formatting.

## Related Documentation

- [PipeLLM Operator](../building-methods/pipes/pipe-operators/PipeLLM.md) - The core operator for LLM interactions
- [Define Your Concepts](../building-methods/concepts/define_your_concepts.md) - How to define custom data types for your methods


## Summarize

# Example: Summarize

This example demonstrates two approaches to summarizing text: a direct structured summary and a multi-step topic-based summarization pipeline.

## Get the code

[![GitHub](https://img.shields.io/badge/View_on_GitHub-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/a_quick_start/summarize.mthds)

## What it demonstrates

- Defining custom concepts with inline structures (`StructuredSummary`)
- Multi-step `PipeSequence` with intermediate results
- Batching over a list of topics (`batch_over` / `batch_as`)
- Two pipes producing different outputs from the same input

## The Method: `summarize.mthds`

### Concepts

The method defines a `StructuredSummary` concept with fields for the summary, an abstract, and a list of main concepts:

```toml
domain        = "summarize"
description   = "Example of summarizing text by topics."
system_prompt = "You are an expert at summarizing text."

[concept]
Topic   = "A topic a text deals with."
Summary = "A concise rewriting of a dense text."

[concept.StructuredSummary]
description = "A summary with a structured output."

[concept.StructuredSummary.structure]
summary       = { type = "text", description = "Summary of the text, super clear and factual", required = true }
abstract      = { type = "text", description = "Abstract of the summary 1 short sentences", required = true }
main_concepts = { type = "list", item_type = "text", description = "Array of main concepts tackled in the text" }
```

### Multi-step pipeline

The `summarize_by_steps` pipeline breaks summarization into three stages: extract topics, summarize each topic individually (batched), then combine all summaries:

```toml
[pipe.summarize_by_steps]
type = "PipeSequence"
description = "Summarize text by steps: extract topics, summarize for each topic, summarize from summaries."
inputs = { text = "Text" }
output = "Summary"
steps = [
  { pipe = "extract_topics", result = "topics" },
  { pipe = "summarize_topic", batch_over = "topics", batch_as = "topic", result = "summarized_topics" },
  { pipe = "summarize_from_summaries", result = "summary" },
]
```

Each topic is summarized independently using `batch_over`, and the final step merges everything into a single cohesive summary.

## How to run

**Direct structured summary:**

```bash
pipelex run bundle examples/a_quick_start/summarize.mthds \
  --pipe summarize_with_structure \
  -i examples/a_quick_start/inputs.json
```

**Multi-step topic-based summary:**

```bash
pipelex run bundle examples/a_quick_start/summarize.mthds \
  --pipe summarize_by_steps \
  -i examples/a_quick_start/inputs.json
```

## Related Documentation

- [PipeSequence Controller](../building-methods/pipes/pipe-controllers/PipeSequence.md) - Chain pipes into sequential workflows
- [PipeLLM Operator](../building-methods/pipes/pipe-operators/PipeLLM.md) - The core operator for LLM interactions
- [Understanding Multiplicity](../building-methods/pipes/understanding-multiplicity.md) - How batching works


## Generic Document

# Example: Generic Document Extraction

This example demonstrates a generic pipeline for extracting content from complex PDF documents. It extracts pages with views, then converts each page to markdown using a vision-capable LLM that combines OCR text with the page image.

## Get the code

[![GitHub](https://img.shields.io/badge/View_on_GitHub-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/b_basics/document_extract/extract_generic/bundle.mthds)

## What it demonstrates

- Vision-based page-to-markdown conversion using `PipeLLM` with `$vision` model
- Using shared method packages for page extraction
- Batching over pages to process each one independently
- Combining OCR text with page images for accurate extraction

## The Method: `bundle.mthds`

```toml
domain    = "extract_generic"
main_pipe = "power_extractor"

[pipe.power_extractor]
type = "PipeSequence"
description = "Update page content with markdown"
inputs = { document = "Document" }
output = "Text[]"
steps = [
  { pipe = "github.com/Pipelex/methods/documents->documents.extract_page_contents_and_views",
    result = "page_contents" },
  { pipe = "write_markdown_from_page_content",
    batch_over = "page_contents", batch_as = "page_content",
    result = "markdowns" },
]

[pipe.write_markdown_from_page_content]
type = "PipeLLM"
description = "Write markdown from page content"
inputs = { "page_content.page_view" = "Image", page_content = "Page" }
output = "Text"
model = "$vision"
system_prompt = "You are a multimodal LLM, expert at converting images into perfect markdown."
prompt = """
You are given an image which is a view of a document page: $page_content.page_view
You are also given the text extracted from the page by an OCR model.
Your task is to output the perfect markdown of the page.

Here is the text extracted from the page:
{{ page_content.text_and_images.text.text|tag("ocr_text") }}

- Ensure you do not miss any information from the page.
- Output only the markdown, nothing else.
"""
```

## How to run

```bash
pipelex run bundle examples/b_basics/document_extract/extract_generic/bundle.mthds \
  -i examples/b_basics/document_extract/extract_generic/inputs.json
```

## Related Documentation

- [PipeExtract Operator](../building-methods/pipes/pipe-operators/PipeExtract.md) - Extract text and images from documents
- [PipeLLM Operator](../building-methods/pipes/pipe-operators/PipeLLM.md) - The core operator for LLM interactions
- [Document Extraction](../features/document-extraction.md) - Overview of document extraction capabilities


## Invoice

# Example: Invoice Extraction

This example processes PDF invoices through a multi-step pipeline: extract pages with views, classify the invoice type, then extract structured data using vision.

## Get the code

[![GitHub](https://img.shields.io/badge/View_on_GitHub-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/b_basics/document_extract/extract_invoice/bundle.mthds)

## What it demonstrates

- Rich structured concept (`Invoice` with 14+ fields including nested `InvoiceDetails`)
- Nested `PipeSequence` (main pipeline batches over pages, sub-pipeline does analyze then extract)
- Vision-based extraction using page views and OCR text
- Using shared method packages for page extraction

## The Method: `bundle.mthds`

### Invoice concept

```toml
[concept.Invoice]
description = "Invoice information extracted from text, supporting both formal bills and receipts"

[concept.Invoice.structure]
invoice_id        = { type = "text", description = "Unique identifier for the invoice" }
invoice_number    = { type = "text", description = "Invoice number as shown on the document" }
date              = { type = "date", description = "Date when the invoice was issued" }
amount_incl_tax   = { type = "number", description = "Total amount including taxes" }
amount_excl_tax   = { type = "number", description = "Net amount excluding taxes" }
vendor            = { type = "text", description = "Name of the vendor/seller" }
category          = { type = "concept", concept_ref = "invoice_extraction.InvoiceDetails", description = "Category or type of expense" }
# ... and more fields
```

### Pipeline

```toml
[pipe.process_invoice]
type = "PipeSequence"
inputs = { document = "Document" }
output = "Invoice[]"
steps = [
  { pipe = "github.com/Pipelex/methods/documents->documents.extract_page_contents_and_views", result = "invoice_pages" },
  { pipe = "extract_invoice", batch_over = "invoice_pages", batch_as = "invoice_page", result = "invoice" },
]
```

Each page goes through a sub-pipeline that first classifies the invoice type (bill vs. receipt), then extracts the full structured data using both the page view and OCR text:

```toml
[pipe.extract_invoice]
type = "PipeSequence"
inputs = { invoice_page = "Page" }
output = "Invoice"
steps = [
  { pipe = "analyze_invoice", result = "invoice_details" },
  { pipe = "extract_invoice_data", result = "invoice" },
]
```

## How to run

```bash
pipelex run bundle examples/b_basics/document_extract/extract_invoice/bundle.mthds \
  -i examples/b_basics/document_extract/extract_invoice/inputs.json
```

## Related Documentation

- [PipeExtract Operator](../building-methods/pipes/pipe-operators/PipeExtract.md) - Extract text and images from documents
- [PipeLLM Operator](../building-methods/pipes/pipe-operators/PipeLLM.md) - The core operator for LLM interactions
- [PipeSequence Controller](../building-methods/pipes/pipe-controllers/PipeSequence.md) - Chain pipes into sequential workflows
- [Document Extraction](../features/document-extraction.md) - Overview of document extraction capabilities


## Proof of Purchase

# Example: Proof of Purchase Extraction

This example extracts structured data from receipts and invoices. It defines nested data concepts (`Product` inside `ProofOfPurchase`) and uses vision-based extraction to capture all details from the document.

## Get the code

[![GitHub](https://img.shields.io/badge/View_on_GitHub-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/b_basics/document_extract/extract_proof_of_purchase/bundle.mthds)

## What it demonstrates

- Nested structured concepts (`Product` referenced inside `ProofOfPurchase`)
- Vision-based extraction with `$vision` model and `structuring_method = "preliminary_text"`
- Using shared method packages for page extraction
- Batching over pages to process each independently

## The Method: `bundle.mthds`

### Concepts

```toml
[concept.Product]
description = "A product in a proof of purchase"

[concept.Product.structure]
name        = { type = "text", description = "Name of the product" }
quantity    = { type = "integer", description = "Quantity purchased" }
unit_price  = { type = "number", description = "Unit price of the product" }
total_price = { type = "number", description = "Total price for this product" }

[concept.ProofOfPurchase]
description = "Elements from a proof of purchase"

[concept.ProofOfPurchase.structure]
date_of_purchase = { type = "date", description = "Date of the purchase" }
amount_paid      = { type = "number", description = "Total amount paid" }
currency         = { type = "text", description = "Currency used for the purchase" }
payment_method   = { type = "text", description = "Method of payment used" }
purchase_number  = { type = "text", description = "Purchase or receipt number" }
products         = { type = "list", item_type = "concept", item_concept_ref = "extract_proof_of_purchase.Product", description = "List of products purchased" }
```

### Pipeline

```toml
[pipe.power_extractor_proof_of_purchase]
type = "PipeSequence"
inputs = { document = "Document" }
output = "ProofOfPurchase[]"
steps = [
  { pipe = "github.com/Pipelex/methods/documents->documents.extract_page_contents_and_views",
    result = "page_contents" },
  { pipe = "write_markdown_from_page_content_proof_of_purchase",
    batch_over = "page_contents", batch_as = "page_content",
    result = "proof_of_purchase" },
]
```

## How to run

```bash
pipelex run bundle examples/b_basics/document_extract/extract_proof_of_purchase/bundle.mthds \
  -i examples/b_basics/document_extract/extract_proof_of_purchase/inputs.json
```

## Related Documentation

- [PipeLLM Operator](../building-methods/pipes/pipe-operators/PipeLLM.md) - The core operator for LLM interactions
- [Document Extraction](../features/document-extraction.md) - Overview of document extraction capabilities


## DPE (Energy Diagnostic)

# Example: DPE Extraction

This example extracts information from French "Diagnostic de Performance Energetique" (DPE) documents. It uses a three-step pipeline: extract pages, convert each to markdown using vision, then conclude the DPE details from the combined markdown.

## Get the code

[![GitHub](https://img.shields.io/badge/View_on_GitHub-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/b_basics/document_extract/extract_dpe/bundle.mthds)

## What it demonstrates

- Custom structured concept with constrained choices (energy efficiency classes A-G)
- Three-step extraction: pages, markdown per page, then conclude
- Vision-based extraction focused on specific document elements (energy classes, graphs, tables)
- Using shared method packages for page extraction

## The Method: `bundle.mthds`

### DPE concept

```toml
[concept.Dpe]
description = "A diagnostic of the energy performance of a building"

[concept.Dpe.structure]
address                       = { type = "text", description = "The address of the building" }
date_of_issue                 = { type = "date", description = "The date the DPE was issued" }
date_of_expiration            = { type = "date", description = "The expiration date of the DPE" }
energy_efficiency_class       = { type = "text", description = "The energy efficiency class",
                                  choices = ["A", "B", "C", "D", "E", "F", "G"] }
per_year_per_m2_consumption   = { type = "number", description = "Energy consumption per year per m2" }
co2_emission_class            = { type = "text", description = "The CO2 emission class",
                                  choices = ["A", "B", "C", "D", "E", "F", "G"] }
per_year_per_m2_co2_emissions = { type = "number", description = "CO2 emissions per year per m2" }
yearly_energy_costs           = { type = "number", description = "Yearly energy costs" }
```

### Pipeline

```toml
[pipe.power_extractor_dpe]
type = "PipeSequence"
inputs = { document = "Document" }
output = "Dpe"
steps = [
  { pipe = "github.com/Pipelex/methods/documents->documents.extract_page_contents_and_views",
    result = "page_contents" },
  { pipe = "write_markdown_from_page_content_dpe",
    batch_over = "page_contents", batch_as = "page_content",
    result = "dpe_pages" },
  { pipe = "conclude_dpe", result = "dpe" },
]
```

The final `conclude_dpe` step takes all the markdown pages and produces a single structured `Dpe` object.

## How to run

```bash
pipelex run bundle examples/b_basics/document_extract/extract_dpe/bundle.mthds \
  -i examples/b_basics/document_extract/extract_dpe/inputs.json
```

## Related Documentation

- [PipeLLM Operator](../building-methods/pipes/pipe-operators/PipeLLM.md) - The core operator for LLM interactions
- [Document Extraction](../features/document-extraction.md) - Overview of document extraction capabilities


## Table from Image

# Example: Table Extraction from Image

This example extracts a table from an image and converts it into structured HTML. It uses a two-step "extract and review" pattern where the first pipe does the initial extraction and the second pipe reviews the result against the original image to correct errors.

## Get the code

[![GitHub](https://img.shields.io/badge/View_on_GitHub-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/b_basics/document_extract/extract_table/bundle.mthds)

## What it demonstrates

- Self-correction pattern: extract then review against the original image
- Custom structured concept (`HtmlTable` with title and HTML content)
- Vision-based extraction using `$vision-table` model
- Multi-modal inputs (image + structured data) in a single pipe

## The Method: `bundle.mthds`

### Pipeline

```toml
[pipe.extract_html_table_and_review]
type = "PipeSequence"
description = "Get an HTML table and review it"
inputs = { table_screenshot = "TableScreenshot" }
output = "HtmlTable"
steps = [
    { pipe = "extract_html_table_from_image", result = "html_table" },
    { pipe = "review_html_table", result = "reviewed_html_table" },
]

[pipe.review_html_table]
type = "PipeLLM"
description = "Review an HTML table"
inputs = { table_screenshot = "TableScreenshot", html_table = "HtmlTable" }
output = "HtmlTable"
prompt = """
Your role is to correct an html_table to make sure that it matches the one in the provided image: $table_screenshot

@html_table

Pay attention to the text and formatting (color, borders, ...).
Rewrite the entire html table with your potential corrections.
Make sure you do not forget any text.
"""
```

This self-correction pattern is a key technique for building robust AI methods — the first step provides a draft, and the second step verifies and corrects it against the source.

## How to run

```bash
pipelex run bundle examples/b_basics/document_extract/extract_table/bundle.mthds \
  -i examples/b_basics/document_extract/extract_table/inputs.json
```

## Related Documentation

- [PipeLLM Operator](../building-methods/pipes/pipe-operators/PipeLLM.md) - The core operator for LLM interactions
- [PipeSequence Controller](../building-methods/pipes/pipe-controllers/PipeSequence.md) - Chain pipes into sequential workflows


## Gantt Chart

# Example: Gantt Chart Extraction

This example extracts structured information from Gantt chart images: timescale, task names, task details (dates, dependencies), and milestones. It uses a "divide and conquer" approach where each aspect is extracted separately.

## Get the code

[![GitHub](https://img.shields.io/badge/View_on_GitHub-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/b_basics/document_extract/extract_gantt/bundle.mthds)

## What it demonstrates

- Divide-and-conquer extraction: timescale, task names, task details, then assemble
- Batching over task names to extract details for each independently
- Vision-based extraction using `$vision-diagram` model
- Rich structured output (`GanttChart` with `GanttTaskDetails` and `Milestone` lists)
- Two alternative approaches: step-by-step vs. direct one-shot extraction

## The Method: `bundle.mthds`

### Pipeline

```toml
[pipe.extract_gantt_by_steps]
type = "PipeSequence"
description = "Extract all details from a gantt chart"
inputs = { gantt_chart_image = "GanttChartImage" }
output = "GanttChart"
steps = [
    { pipe = "extract_gantt_timescale", result = "gantt_timescale" },
    { pipe = "extract_gantt_task_names", result = "gantt_task_names" },
    { pipe = "extract_details_of_task", batch_as = "gantt_task_name",
      result = "details_of_all_tasks" },
    { pipe = "gather_in_a_gantt_chart", result = "gantt_chart" },
]
```

Each task's details are extracted individually using the chart image, the timescale context, and the specific task name — making each extraction focused and accurate.

The bundle also includes a `transcript_gantt_direct` pipe for one-shot extraction to markdown, useful for simpler charts.

## How to run

```bash
pipelex run bundle examples/b_basics/document_extract/extract_gantt/bundle.mthds \
  -i examples/b_basics/document_extract/extract_gantt/inputs.json
```

## Related Documentation

- [PipeLLM Operator](../building-methods/pipes/pipe-operators/PipeLLM.md) - The core operator for LLM interactions
- [PipeSequence Controller](../building-methods/pipes/pipe-controllers/PipeSequence.md) - Chain pipes into sequential workflows


## Markdown from Document

# Example: Extract Markdown from Document

This example extracts text from a PDF document and composes the pages into a single clean markdown output. It leverages the shared `documents` method package for page extraction.

## Get the code

[![GitHub](https://img.shields.io/badge/View_on_GitHub-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/b_basics/document_extract/extract_markdown/bundle.mthds)

## What it demonstrates

- Using shared method packages (`github.com/Pipelex/methods/documents`)
- `PipeCompose` with Jinja2 templates to concatenate page text
- Simple two-step document-to-markdown pipeline

## The Method: `bundle.mthds`

The pipeline uses a shared extraction pipe and then composes all pages into a single text:

```toml
domain      = "extract_markdown"
main_pipe   = "extract_markdown_from_document"

[pipe.extract_markdown_from_document]
type = "PipeSequence"
description = "Extract markdown from a document"
inputs = { document = "Document" }
output = "Text"
steps = [
  { pipe = "github.com/Pipelex/methods/documents->documents.extract_markdown_pages", result = "pages" },
  { pipe = "write_markdown_from_pages", result = "markdown" },
]

[pipe.write_markdown_from_pages]
type = "PipeCompose"
description = "Write markdown from pages"
inputs = { pages = "Page[]" }
output = "Text"
template = """
{% for page in pages %}
{{ page.text_and_images.text }}
{% endfor %}
"""
```

## How to run

```bash
pipelex run bundle examples/b_basics/document_extract/extract_markdown/bundle.mthds \
  -i examples/b_basics/document_extract/extract_markdown/inputs.json
```

## Related Documentation

- [PipeExtract Operator](../building-methods/pipes/pipe-operators/PipeExtract.md) - Extract text and images from documents
- [PipeCompose Operator](../building-methods/pipes/pipe-operators/PipeCompose.md) - Template-based data composition
- [Packages](../building-methods/packages.md) - Using shared method packages


## Slides from Presentation

# Example: Extract Slides from Presentation

This example extracts structured slide information from a presentation document — including titles, content as markdown, and visual descriptions of each slide layout.

## Get the code

[![GitHub](https://img.shields.io/badge/View_on_GitHub-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/b_basics/document_extract/extract_slides/bundle.mthds)

## What it demonstrates

- Custom `Slide` concept with inline structure (title, text, description)
- Vision-based slide analysis using `PipeLLM` with page views
- Batching over extracted pages to process each slide individually
- `PipeCompose` for assembling structured output into formatted text

## The Method: `bundle.mthds`

### Slide concept

```toml
[concept.Slide]
description = "A slide from a presentation"

[concept.Slide.structure]
title         = { type = "text", description = "The title of the slide" }
text_markdown = { type = "text", description = "The content of the slide as markdown" }
description   = { type = "text", description = "A description of the slide: layout of the text, description of the graphics" }
```

### Pipeline

The pipeline extracts pages with views, describes each slide using a vision model, then concatenates the descriptions:

```toml
[pipe.extract_slides]
type = "PipeSequence"
description = "Extract markdown from a document"
inputs = { document = "Document" }
output = "Text"
steps = [
  { pipe = "extract_markdown_and_views_from_document", result = "pages" },
  { pipe = "describe_slide", batch_over = "pages", batch_as = "page", result = "slides" },
  { pipe = "concatenate_slide_descriptions", result = "slides_description" },
]
```

## How to run

```bash
pipelex run bundle examples/b_basics/document_extract/extract_slides/bundle.mthds \
  -i examples/b_basics/document_extract/extract_slides/inputs.json
```

## Related Documentation

- [PipeLLM Operator](../building-methods/pipes/pipe-operators/PipeLLM.md) - The core operator for LLM interactions
- [PipeExtract Operator](../building-methods/pipes/pipe-operators/PipeExtract.md) - Extract text and images from documents
- [PipeCompose Operator](../building-methods/pipes/pipe-operators/PipeCompose.md) - Template-based data composition


## Design Slides

# Example: Design Slides

This example takes a slide design brief and generates multiple themed design proposals, complete with visual mockups and an HTML report comparing them.

## Get the code

[![GitHub](https://img.shields.io/badge/View_on_GitHub-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/b_basics/generate_visuals/design_slides/bundle.mthds)

## What it demonstrates

- Rich concept hierarchy (nested structures: `Theme` contains `ColorPalette`, `Typography`, `LayoutSettings`, `StyleSettings`)
- `nb_output` parameter to generate multiple results from a single pipe
- `PipeImgGen` for generating visual mockup images
- `PipeCompose` with an HTML template for report rendering
- Multi-step creative pipeline: polish brief, generate themes, render visuals, compose report

## The Method: `bundle.mthds`

### Key concepts

The method defines a deep hierarchy of design concepts:

```toml
[concept.SlideDesignBrief]
description = "Client brief for slide deck design"

[concept.SlideDesignBrief.structure]
topic = { type = "text", description = "The main topic or subject of the presentation", required = true }
brand_guidelines = { type = "text", description = "The client's brand guidelines", required = false }
tone = { type = "text", description = "The tone of the presentation", choices = [
  "formal", "playful", "innovative", "trustworthy", "artsy",
], required = false }
# ... goal, audience fields
```

```toml
[concept.Theme]
description = "Complete presentation theme with colors, typography, layout, and style"

[concept.Theme.structure]
name       = { type = "text", description = "Theme name identifier", required = true }
colors     = { type = "concept", concept_ref = "ColorPalette", description = "Color palette for the theme", required = true }
typography = { type = "concept", concept_ref = "Typography", description = "Typography settings", required = true }
layout     = { type = "concept", concept_ref = "LayoutSettings", description = "Layout configuration", required = true }
style      = { type = "concept", concept_ref = "StyleSettings", description = "Visual style settings", required = true }
```

### Pipeline

```toml
[pipe.generate_design_proposals_from_rough_brief]
type = "PipeSequence"
description = "Transform a design brief into multiple themes with visual mockups and HTML report"
inputs = { brief = "SlideDesignBrief" }
output = "Html"
steps = [
  { pipe = "polish_brief", result = "polished_brief" },
  { pipe = "generate_multiple_themes", nb_output = 3, result = "themes" },
  { pipe = "render_visual_proposal", batch_over = "themes", batch_as = "theme", result = "design_proposals" },
  { pipe = "compose_proposals_report", result = "proposals_report" },
]
```

The `nb_output = 3` parameter tells `generate_multiple_themes` to produce 3 different theme proposals. Each theme is then rendered as a visual mockup using `PipeImgGen`, and everything is assembled into an HTML report.

## How to run

```bash
pipelex run bundle examples/b_basics/generate_visuals/design_slides/bundle.mthds \
  -i examples/b_basics/generate_visuals/design_slides/inputs.json
```

## Related Documentation

- [PipeLLM Operator](../building-methods/pipes/pipe-operators/PipeLLM.md) - The core operator for LLM interactions
- [PipeImgGen Operator](../building-methods/pipes/pipe-operators/PipeImgGen.md) - Generate images from prompts
- [PipeCompose Operator](../building-methods/pipes/pipe-operators/PipeCompose.md) - Template-based data composition
- [PipeSequence Controller](../building-methods/pipes/pipe-controllers/PipeSequence.md) - Chain pipes into sequential workflows


## Generate Image

# Example: Generate Image

This example generates creative, absurd images through a three-step pipeline: imagine a surreal scene concept, refine it into an optimized image prompt, then render the image.

## Get the code

[![GitHub](https://img.shields.io/badge/View_on_GitHub-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/b_basics/generate_visuals/gen_image/bundle.mthds)

## What it demonstrates

- Concept refinement (`SceneConcept` and `ImagePrompt` both refine `Text`)
- `PipeImgGen` for image generation
- Multi-step creative pipeline where each step builds on the previous one
- No external inputs required — the pipeline generates everything from scratch

## The Method: `bundle.mthds`

```toml
domain      = "crazy_image_generation"
main_pipe   = "generate_crazy_image"

[concept.SceneConcept]
description = "A creative, abstract description of an absurd and surreal scene."
refines = "Text"

[concept.ImagePrompt]
description = "A detailed textual description optimized for image generation models."
refines = "Text"

[pipe.generate_crazy_image]
type = "PipeSequence"
description = "Main pipeline that imagines a scene, refines it into a prompt, and renders the image"
output = "Image"
steps = [
  { pipe = "imagine_scene", result = "scene_concept" },
  { pipe = "generate_img_prompt", result = "image_prompt" },
  { pipe = "render_image", result = "crazy_image" },
]
```

The `render_image` pipe uses `PipeImgGen` to generate the final image:

```toml
[pipe.render_image]
type        = "PipeImgGen"
description = "Generates the absurd image based on the creative scene description"
inputs      = { image_prompt = "ImagePrompt" }
output      = "Image"
prompt      = "$image_prompt"
model       = "@default-small"
```

## How to run

```bash
pipelex run bundle examples/b_basics/generate_visuals/gen_image/bundle.mthds
```

No inputs needed — the pipeline imagines everything from scratch.

## Related Documentation

- [PipeImgGen Operator](../building-methods/pipes/pipe-operators/PipeImgGen.md) - Generate images from prompts
- [PipeLLM Operator](../building-methods/pipes/pipe-operators/PipeLLM.md) - The core operator for LLM interactions
- [Refining Concepts](../building-methods/concepts/refining-concepts.md) - How concept refinement works


## Synthetic Data

# Example: Synthetic Data Generation

This example generates diverse synthetic data samples from a text description. It first identifies a comprehensive set of sample cases, then creates detailed profiles for each one.

## Get the code

[![GitHub](https://img.shields.io/badge/View_on_GitHub-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/c_advanced/gen_synthetic_data/bundle.mthds)

## What it demonstrates

- Concept refinement (`DataDescription` refines `Text`, `NbOfSamples` refines `Number`)
- Rich structured concept with constrained choices (`Sample` with `current_performance`, `learns_best_with`, etc.)
- Two-step generation: identify diverse cases, then create a profile for each
- Batching over identified cases to generate samples in parallel

## The Method: `bundle.mthds`

### Sample concept

The `Sample` concept represents a student profile with constrained fields:

```toml
[concept.Sample]
description = "A complete synthetic data record representing a student profile."

[concept.Sample.structure]
student_name = { type = "text", description = "The name of the student", required = true }
current_performance = { type = "text", description = "The student's current performance level", choices = [
  "Struggling", "Average", "Advanced",
], required = true }
learns_best_with = { type = "text", description = "The learning modality that works best", choices = [
  "Visual examples", "Step-by-step text", "Hands-on practice", "Videos",
], required = true }
# ... pace, complexity, strengths, needs_help_with, etc.
```

### Pipeline

```toml
[pipe.generate_synthetic_data_samples]
type = "PipeSequence"
inputs = { data_description = "DataDescription", nb_samples = "NbOfSamples" }
output = "Sample[]"
steps = [
  { pipe = "identify_sample_cases", result = "cases" },
  { pipe = "create_profile_for_case", result = "samples",
    batch_over = "cases", batch_as = "case_characterization" },
]
```

The first step identifies diverse real-world scenarios to ensure comprehensive coverage (edge cases, common scenarios, important variations). The second step creates a full profile for each case.

## How to run

```bash
pipelex run bundle examples/c_advanced/gen_synthetic_data/bundle.mthds \
  -i examples/c_advanced/gen_synthetic_data/inputs.json
```

## Related Documentation

- [PipeLLM Operator](../building-methods/pipes/pipe-operators/PipeLLM.md) - The core operator for LLM interactions
- [PipeSequence Controller](../building-methods/pipes/pipe-controllers/PipeSequence.md) - Chain pipes into sequential workflows
- [Understanding Multiplicity](../building-methods/pipes/understanding-multiplicity.md) - How batching works
- [Refining Concepts](../building-methods/concepts/refining-concepts.md) - How concept refinement works


## Expense Data with Receipts

# Example: Expense Data with Receipts

This advanced example generates complete synthetic expense datasets for multiple employees, including realistic receipt images and HTML reports. It features fraud scenario generation for testing expense validation systems.

## Get the code

[![GitHub](https://img.shields.io/badge/View_on_GitHub-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/blob/main/examples/c_advanced/gen_expense_data/bundle.mthds)

## What it demonstrates

- Deep concept hierarchy (Employee, ExpenseScenario, CompanyProfile, ReceiptContent, Expense, etc.)
- Nested `PipeSequence` pipelines (main pipeline batches over employees, sub-pipeline batches over categories)
- `PipeImgGen` for generating realistic receipt images
- `PipeCompose` with `construct` for assembling structured objects from parts
- `PipeCompose` with HTML templates for report rendering
- Fraud scenario modeling (weekend expenses, inflated amounts, receipt mismatches, vague purposes)
- Custom Python runner for post-processing (PDF generation, file export)

## The Method: `bundle.mthds`

### Concept hierarchy (simplified)

```
NbOfEmployees (input, refines Number)
  -> Employee[]
     -> CompanyCategory[] (with ExpenseScenario)
        -> CompanyProfile
        -> ReceiptContent (from ReceiptHeader + ReceiptItemsAndTotals + ExpenseMetadata)
        -> Receipt (image via PipeImgGen)
        -> Expense + Receipt = ExpenseWithReceipt
     -> EmployeeExpenseReport (employee + expenses + HTML report)
```

### Main pipeline

```toml
[pipe.generate_expense_dataset]
type = "PipeSequence"
inputs = { nb_employees = "NbOfEmployees" }
output = "EmployeeExpenseReport[]"
steps = [
  { pipe = "generate_employees", result = "employees" },
  { pipe = "generate_employee_report", batch_over = "employees", batch_as = "employee", result = "reports" },
]
```

For each employee, the `generate_employee_report` sub-pipeline assigns expense categories (with fraud scenarios), then for each category generates a company profile, receipt content, receipt image, and expense record.

### Receipt generation

Receipt content is built in multiple steps: header (transaction number, date), items and totals (from the company's product catalog), and business purpose metadata. These are composed into formatted thermal-paper-style text, then rendered as a photo using `PipeImgGen`.

## How to run

**Using the CLI:**

```bash
pipelex run bundle examples/c_advanced/gen_expense_data/bundle.mthds \
  -i examples/c_advanced/gen_expense_data/inputs.json
```

**Using the Python runner** (for PDF export and file organization):

```bash
cd examples/c_advanced/gen_expense_data
python run_generate_expense_dataset.py
```

## Related Documentation

- [PipeSequence Controller](../building-methods/pipes/pipe-controllers/PipeSequence.md) - Chain pipes into sequential workflows
- [PipeImgGen Operator](../building-methods/pipes/pipe-operators/PipeImgGen.md) - Generate images from prompts
- [PipeCompose Operator](../building-methods/pipes/pipe-operators/PipeCompose.md) - Template-based data composition
- [Understanding Multiplicity](../building-methods/pipes/understanding-multiplicity.md) - How batching works


## Using Inference Plugins

# Example: Using Inference Plugins

This example shows how Pipelex discovers **inference-backend plugins**: installable Python packages that teach Pipelex how to serve models for a given `sdk` token. The cookbook ships a complete, minimal plugin — a deterministic "LLM" that always answers with the same haiku — so the whole flow runs anywhere with no API key and no network.

## Get the code

[![GitHub](https://img.shields.io/badge/View_on_GitHub-5a0dad?logo=github&logoColor=white&style=flat)](https://github.com/Pipelex/pipelex-cookbook/tree/main/examples/c_advanced/using_inference_plugins)

## What it demonstrates

- What a plugin **is**: a package exposing a `pipelex.plugins.kernel` entry point that resolves to a `PipelexPlugin` — an object with a `name`, a `targets_api` version, and a side-effect-free `register(registrar)` method
- Registration through the registrar: `register` contributes an inference backend for `(family="llm", sdk="hello")`, plus an optional model lister
- Discovery by presence: installing the package is the whole integration — no enable-list, no config switch
- The model-config side: how a model handle in a `.mthds` file resolves to the plugin's worker through `.pipelex/inference/`
- The failure mode when the plugin is missing — a loud, friendly error, not a silent fallback

## The plugin package

The plugin lives inside the example directory as its own installable package:

```
examples/c_advanced/using_inference_plugins/
├── hello_plugin.mthds                  # the method that uses the plugin-served model
└── hello_inference_plugin/             # the plugin package
    ├── pyproject.toml                  # declares the `pipelex.plugins.kernel` entry point
    └── hello_inference_plugin/
        ├── hello_plugin.py             # HelloInferencePlugin: name, targets_api, register()
        ├── hello_llm_worker.py         # HelloLLMWorker(LLMWorkerAbstract): the actual "model"
        └── hello_list.py               # optional lister behind `pipelex show models hello`
```

The entry point in `pyproject.toml` is the single line that wires everything:

```toml
[project.entry-points."pipelex.plugins.kernel"]
hello_inference = "hello_inference_plugin.hello_plugin:HelloInferencePlugin"
```

The plugin class registers what it serves — and nothing else happens at registration time (no I/O, no client construction; heavy work belongs inside the worker factory):

```python
class HelloInferencePlugin:
    name = "hello_inference"
    targets_api = PLUGIN_API_VERSION

    def register(self, registrar: PluginRegistrar) -> None:
        registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="hello", make_worker=_make_hello_llm_worker)
        registrar.add_model_lister(sdk="hello", lister=_list_hello_models)
```

The worker subclasses `LLMWorkerAbstract` and only implements the generation itself — the base class owns the job lifecycle, capability checks, usage reporting, and telemetry. Here it deterministically returns a haiku; a real plugin would call a remote model at this exact spot.

## The model-config side

Model configuration lives in the project's `.pipelex/inference/` directory (not in `.pipelex/pipelex.toml`). Three pieces connect a `.mthds` model handle to the plugin:

- `backends.toml` declares the `[hello]` backend (enabled, no API key)
- `backends/hello.toml` declares model `hello-1` with `sdk = "hello"` — the token that selects the plugin's worker factory
- `routing_profiles.toml` routes `hello-1` to the `hello` backend via an `optional_routes` entry, which only applies while the target backend is enabled

## The Method: `hello_plugin.mthds`

```toml
domain      = "hello_plugin"
description = "Using a model served by an inference-backend plugin"
main_pipe   = "hello_plugin"

[pipe]
[pipe.hello_plugin]
type = "PipeLLM"
description = "Write text about Hello World."
output = "Text"
model = { model = "hello-1", temperature = 0.5 }
prompt = """
Write a haiku about Hello World.
"""
```

The only difference from the standard [Hello World](./hello-world.md) example is the `model` field: `hello-1` is served by the plugin instead of a gateway backend.

## How to run

1. Install the plugin package (this is the step being demonstrated):

    ```bash
    uv pip install -e examples/c_advanced/using_inference_plugins/hello_inference_plugin
    ```

2. Check that Pipelex discovered it:

    ```bash
    pipelex plugins list        # hello_inference | external | registered
    pipelex show models hello   # lists hello-1
    ```

3. Run the method — no credentials needed:

    ```bash
    pipelex run bundle examples/c_advanced/using_inference_plugins/hello_plugin.mthds
    ```

    The output is the plugin's deterministic haiku.

## The failure mode

Uninstall the package and run again:

```bash
uv pip uninstall hello-inference-plugin
pipelex run bundle examples/c_advanced/using_inference_plugins/hello_plugin.mthds
```

Pipelex fails loud at worker-creation time: `No inference backend registered for sdk 'hello' in the llm family. Is its plugin installed and enabled?` — the model config still resolves; only the worker factory is missing. A dry run still passes without the plugin, because worker creation is lazy.

## Related Documentation

- [Inference Backend Plugins](../under-the-hood/inference-backend-plugins.md) - The plugin seam and the Inference SPI, including how to wrap a real SDK (dependency guards, client memoization)
- [Configuration System](../features/configuration.md) - How `.pipelex/` project configuration works
- [Pipelex Gateway & Model Access](../features/gateway.md) - Default model access through the gateway


# Viewpoint

## The Know-How Graph

### This is our viewpoint, this is why we built [Pipelex](https://pipelex.com).

Web version: https://knowhowgraph.com/

---
# Viewpoint: The Know-How Graph

**Why AI Agents Need the MTHDS Open Standard, and What Happens When They Get It**

*By Louis Choquel, Co-founder & CTO, Pipelex*

*Originally published: November 2025*
*Updated: February 2026 — revised to reflect the emergence of Agent Skills and the evolving agent ecosystem*

---

## TL;DR

Skills gave agents reusable guidance: portable, natural-language instructions that steer behavior. But they have no types, no validation, no guaranteed output structure. That's why we built **MTHDS**: an open standard for typed, validated AI workflows with deterministic orchestration, that agents can build, execute, and share. At scale, typed methods connect to form the **Know-How Graph**: a navigable network of reusable, composable AI know-how that becomes shared infrastructure.

## Skills guide intent, not execution

Skills are a great tool to control agents: portable, natural-language instructions that guide behavior. But they're just text injected into prompts. The agent reinterprets them at every invocation: different reasoning paths, different output shapes, limited control over structure. Skills guide intent. They don't guarantee execution.

Today's most advanced harnesses like Claude Code and Codex keep agents working on complex tasks by providing context lifecycle, checkpoints, verification gates, tool execution. But deterministic orchestration and validated outputs? That still requires code. What's missing from this picture: a declarative, portable way to capture a proven method as a typed, validated artifact that runs the same way every time.

## Three Ways to Build AI Workflows Today — All Inadequate

Teams everywhere are building AI workflows to extract data from contracts, process expense reports, classify documents, screen resumes. Identical problems solved in isolation. The options they face all have serious trade-offs.

**Code** (custom Python, frameworks) gives full control: typed outputs, version control, testing. But the 100 lines that capture your actual business logic are buried in 2,000 lines of plumbing. Every workflow is bespoke, nothing composes, and the domain expert who owns the logic cannot read or iterate on it.

**Skills and prompts** are the opposite. Anyone can write them in minutes. Human-readable, progressively complex. But there is no structure, no validation, no typed outputs, no guaranteed repeatability. You ship a prompt and hope.

**Automation platforms** (Zapier, Make, n8n) are good at what they're built for: connecting APIs and automating data flow between services. But they are not cognitive workflow systems. AI was bolted on after the fact. When you need multi-step cognition (extraction, analysis, synthesis, reasoning, decision making), these tools quickly become unwieldy.

Code expresses technical plumbing, not business logic. Skills express intent without guarantees. Visual builders express data routing, not cognitive work. **What's missing is a common language between code and Skills:** structured enough to validate, conceptual enough for business experts to read, and explicit enough for agents to build and execute reliably.

## We Need a Language for AI Methods

AI workflows should be first-class citizens of our technical infrastructure: not buried in code or trapped in platforms, but expressed in a language built for the job. The method should be an artifact you can version, diff, test, and optimize.

### The Language must be Declarative and conceptual

**We need a declarative language that states what you want, not how to compute it.** SQL separated intent from implementation for data. We need the same separation for AI workflows.

But it must go further. Thanks to LLMs, machines can now grasp your intent, which opens the door to a fundamentally different way of defining workflows. The right abstraction level is the business domain itself. An `Invoice` is a declared concept with an explicit definition: what it means, what it contains, why it matters. A `NonCompeteClause` is not just "some text from a contract"; it is a concept you define once, with its purpose and characteristics spelled out in the language itself, not in a comment or a README or someone's head. That declaration is what removes ambiguity.

A method *is* the documentation, except it's also executable. A business expert reads it and sees the logic of their domain. An engineer reads it and sees something they can validate, test, and deploy. An AI agent reads it and sees something it can build, modify, and execute. One artifact, three audiences, no translation required.

Imagine agents that transform natural-language requirements into working methods. They design each transformation step (or reuse existing ones), test against real or synthetic data, incorporate expert feedback, and iterate to improve quality while reducing costs.

> This is how agents finally retain know-how: by encoding what works into conceptual methods they can build, share, and execute on demand.

### The Language must be Structured

When concepts also carry typed structures, the language gains capabilities that text-only instructions cannot support.

**Validation before runtime.** A `JobDescription` and a `CandidateProfile` might both be strings under the hood. No traditional type checker flags the swap. But feeding a candidate profile where a job description is expected is a semantic error, and in an AI workflow, the result isn't a crash. It's confident, plausible, wrong output. Concept-level typing surfaces that mistake at design time, long before you hit production.

**Discovery and composability.** Every method carries a typed signature: declared input and output concepts. That makes methods queryable, composable, and verifiable at design time. When an ecosystem of typed methods exists, something more powerful emerges.

## The Know-How Graph: A Typed Network of Composable Methods

Each method should stand on the shoulders of others, composing like LEGO bricks to build increasingly sophisticated cognitive systems.

What emerges is a **Know-How Graph**: not just static knowledge, but executable methods that connect and build upon one another. Unlike a knowledge graph that maps facts, this maps procedures: the actual know-how of getting cognitive work done.

Every method has a typed signature: declared input concepts and output concepts. This makes the graph machine-readable in a way that text-based systems cannot match.

**Type-driven discovery.** "I have a `Contract`, I need a `NonCompeteClause`" becomes a structured lookup across all published methods, matching input and output concepts. Not keyword search. Not description matching. Concept-compatible resolution across the entire ecosystem.

**Auto-composition.** When no single method bridges the gap, you find a path through the graph. One method produces a `RiskAssessment` from a `LoanApplication`. Another produces an `UnderwritingDecision` from a `RiskAssessment`. The system verifies that the two connect cleanly at design time. Compatibility is provable, not assumed. **You get pathfinding in conceptual space.**

### Know-how is as shareable as knowledge

Think about the explosion of prompt sharing since 2023 — all those people trading their best ChatGPT prompts on Twitter, GitHub, Reddit. Then Agent Skills took it further: portable capability bundles that work across agent environments, shared like dotfiles, forked like repos, with real ecosystem effects already emerging. **Now imagine that same viral sharing, but with complete, tested, composable, *typed* workflows instead of fragile prompts or unstructured instructions.**

Software package managers, SQL views, Docker images, dbt packages: composable standards create ecosystems where everyone's work makes everyone else more productive. Methods are designed for the same pattern. Common tasks get solved once and shared openly. The workflows that give your company its edge? That's your secret sauce, and it stays private. Open commons, proprietary layers on top.

## The MTHDS Open Standard

MTHDS is an open standard for defining, packaging, and distributing AI methods as typed, composable, human-readable files.

A `.mthds` file captures a method in plain TOML: declared concepts as inputs and outputs at every level, from the method itself down to each processing step, where each step can tap into the cognitive power of any AI model or run as pure deterministic logic. Orchestration is explicit: sequences, conditions, batches, parallel branches.

A `METHODS.toml` manifest turns a collection of `.mthds` files into a distributable package with dependencies, version constraints, and exports. Distribution is federated: packages live in Git repositories, registries provide discovery.

Like Skills, MTHDS files live in directories, are Git-native, and scale from personal use to enterprise deployment. Unlike Skills, they carry typed structure, validated outputs (JSON), and deterministic orchestration.

MIT-licensed. Designed for portability. The method outlives any vendor or model version.

[mthds.ai](https://mthds.ai/latest/) | [github.com/mthds-ai](https://github.com/mthds-ai)

## Pipelex: The Reference Runtime

Pipelex is the open-source reference runtime for the MTHDS standard. It reads, validates, and executes `.mthds` files, and ships as a runtime, CLI, Claude Code plugin, cookbook, and documentation.

[github.com/Pipelex/pipelex](https://github.com/Pipelex/pipelex) | [docs.pipelex.com](https://docs.pipelex.com)

## Join Us

Start with one method: extract invoice data, process applications, analyze reports. Share what works. Build on what others share. **The future of AI needs both:** Skills that give agents latitude to explore and adapt, and typed methods that guarantee execution when the output matters. They are complementary.

---

*If this Viewpoint resonates, please share it.*

*Licensed under a [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) license.*

---

**Agents + MTHDS = The Know-How Graph**

---

[MTHDS Standard](https://mthds.ai/latest/) | [Pipelex Runtime](https://github.com/Pipelex/pipelex) | [Documentation](https://docs.pipelex.com) | [Discord](https://go.pipelex.com/discord)

© 2025-2026 Evotis S.A.S. — [Pipelex](https://pipelex.com) is a trademark of Evotis S.A.S.


# Features

## Overview

# Pipelex Features

Pipelex is the reference runtime for AI methods written in MTHDS. This section covers the main capabilities you use to define methods, run them, and operate them in production.

This section covers all Pipelex capabilities, organized by area.

---

## Declarative AI Methods

<div class="grid cards" markdown>

-   **[MTHDS Language](mthds-language.md)**

    A declarative, TOML-based `.mthds` file format for defining AI methods without writing code. Domains, bundles, packages, and cross-references.

-   **[Concepts & Structured Types](concepts.md)**

    Semantic typing system for AI data. Native types, inline structures, Python classes, concept refinement, and hierarchies.

-   **[Pipe Operators](pipe-operators.md)**

    The workers that do the actual processing: PipeLLM, PipeStructure, PipeExtract, PipeImgGen, PipeSearch, PipeCompose, and PipeFunc.

-   **[Pipeline Orchestration](pipeline-orchestration.md)**

    Controllers for building complex workflows: PipeSequence, PipeParallel, PipeBatch, PipeCondition, and working memory for data flow.

</div>

---

## AI Capabilities

<div class="grid cards" markdown>

-   **[Pipelex Gateway & Model Access](gateway.md)**

    Unified access to 60+ AI models through a single API key, or bring your own keys for direct provider access. Open-source model support via Hugging Face, Scaleway, and Groq.

-   **[LLM Integration](llm-integration.md)**

    Text generation, structured outputs, vision language models, prompting styles, system prompt inheritance, and model presets.

-   **[Document Extraction](document-extraction.md)**

    Multi-provider OCR with Mistral OCR, Azure Document Intelligence, docling, and Deepseek-OCR. PDF processing, page rendering, layout analysis, and table recognition.

-   **[Image Generation](image-generation.md)**

    Text-to-image generation with FLUX, GPT Image, and other providers. Cloud storage integration for generated images.

-   **[Web Search](web-search.md)**

    Structured web search results with source citations and advanced filters via PipeSearch.

</div>

---

## Developer Tools

<div class="grid cards" markdown>

-   **[Claude Code Skills Plugin](claude-code-skills-plugin.md)**

    Build, run, validate, and edit AI methods directly from Claude Code with the MTHDS skills plugin. Slash commands cover the full method lifecycle.

-   **[CLI](cli.md)**

    Full command-line interface: `login`, `init`, `doctor`, `build`, `validate`, `run`, `graph`, `show`, `which`, and more. Dry run mode, graph generation, and agent CLI.

-   **[plxt Formatter & Linter](plxt.md)**

    Fast formatting and linting for TOML, MTHDS, and PLX files. CI-ready with `--check` mode.

-   **[Execution Graph Visualization](execution-graph.md)**

    Interactive HTML visualization and Mermaid chart export for pipeline execution. Step-by-step data inspection with JSON, HTML, images, and PDFs.

</div>

---

## Production & Operations

<div class="grid cards" markdown>

-   **[Validation & Dry Run](validation-dry-run.md)**

    Pipeline validation without execution, dry run with mocked LLM responses, input validation, and allowed-to-fail pipes.

-   **[Distributed Execution](distributed-execution.md)**

    Run the same methods as durable workflows on a Temporal or Mistral Workflows backend, [delivered through the Pipelex platform](https://pipelex.com/products#durable-execution). Per-activity routing, named runtime profiles, and dashboard observability.

-   **[Telemetry & Observability](telemetry.md)**

    Production monitoring with Langfuse, OpenTelemetry, and PostHog integration. Gateway telemetry, span tracing, and custom destinations.

-   **[Cloud Storage](cloud-storage.md)**

    Store generated artifacts on AWS S3 or Google Cloud Storage with public or signed URLs.

-   **[Cost Tracking & Reporting](cost-tracking.md)**

    Automatic token cost tracking per operation with console logging and CSV report export.

</div>

---

## Configuration & Extensibility

<div class="grid cards" markdown>

-   **[Configuration System](configuration.md)**

    Multi-level TOML configuration: base defaults, project overrides, environment-specific, and run-mode-specific. Environment variable support.

-   **[Advanced Customizations](advanced-customizations.md)**

    Dependency injection framework for secrets, storage, observers, content generators, and pipe routers.

</div>


## MTHDS Language

# MTHDS Language

The declarative, TOML-based file format for defining AI methods without writing code.

## Overview

MTHDS (`.mthds` files) is a domain-specific language for defining AI methods. It lets developers declare concepts, pipes, and pipelines in a structured format that is both human-readable and machine-executable. The philosophy is separation of concerns: method logic is declared in MTHDS, while execution is handled by the Pipelex runtime. This keeps prompts, data flow, and orchestration version-controlled and reviewable.

## Domains

Semantic namespaces for organizing related concepts and pipes. Domains provide hierarchical naming and scoping, preventing name collisions across packages. Each `.mthds` file declares a single domain, and multiple files can contribute to the same domain.

## Bundles

A bundle is a complete method package defined in a single `.mthds` file. It contains a domain declaration, concepts, and pipes, and can optionally designate a `main_pipe` as the entry point. Bundles are self-contained and can be executed via the CLI or Python API.

## Packages & Dependencies

Use `METHODS.toml` to declare package identity and exports. The current public package-manifest workflow is documented through the lowercase `mthds` CLI: `mthds package init`, `mthds package list`, and `mthds package validate`.

## Cross-Package References

Reference concepts and pipes across package boundaries using the `->` operator with fully qualified names. This enables method composition — build complex methods from smaller, reusable packages.

## Pure MTHDS Development

Inline concept structures with nested concepts make Pipelex fully usable with just `.mthds` files and the CLI — no Python code required. Define structured outputs, orchestrate multi-step pipelines, and run everything from the command line.

## Language Specification

For the formal language specification, see [mthds.ai](https://mthds.ai/latest/). For package-manifest details, see [Packages](../building-methods/packages.md).

## Related Documentation

- [Packages](../building-methods/packages.md) - `METHODS.toml` manifests and exports
- [Concepts](./concepts.md) - Semantic typing and structured content
- [Pipe Operators](./pipe-operators.md) - The operators you can declare in `.mthds` files


## Concepts & Structured Types

# Concepts & Structured Types

The semantic typing system that gives meaning to your AI data.

## Overview

Concepts are Pipelex's type system. They define what kind of data flows through your pipelines — not just the shape, but the meaning. A concept can be as simple as "a tweet about a product" or as complex as "an invoice with line items, totals, and tax breakdowns." Every pipe declares the concepts it accepts as input and produces as output, enabling Pipelex to validate data flow at definition time rather than runtime.

## Native Concepts

Pre-built universal concepts available in every Pipelex project:

- **Dynamic** — Dynamic typing, resolved at runtime
- **Text** — Plain text content
- **Image** — Image data (base64, URL, or file path)
- **Document** — A document container (PDF, image, or other file)
- **Html** — HTML content
- **TextAndImages** — Combined text and image content in a single structure
- **Number** — Numeric value
- **YesNo** — The answer to a yes/no question
- **Date** — A calendar date, optionally with a time of day
- **Time** — A time of day, optionally with a UTC offset
- **Page** — A single page extracted from a document, with markdown text and optional images
- **JSON** — Arbitrary JSON data
- **SearchResult** — Web search result with source citations
- **Composite** — A named composition of contents
- **Anything** — Universal type that accepts any content

## Custom Concepts

Define your own concepts with natural language descriptions and optional structured fields. Custom concepts live in `.mthds` files within a domain and can carry a description that guides LLMs, structured fields for validation, or both.

## Inline Structures

Define structured fields directly in `.mthds` files without writing Python. Inline structures support typed fields (text, integer, boolean, number, date, datetime, time, list, dict) and nested concepts for complex data shapes — making it possible to build fully structured methods without leaving the MTHDS language.

## Python StructuredContent Classes

For advanced use cases, generate Pydantic BaseModels from your declarative concepts. This gives you full IDE autocomplete, type checking, and runtime validation. You can also hand-write `StructuredContent` classes when you need custom logic beyond what MTHDS declarations provide.

## Concept Refinement & Hierarchies

Create specialized versions of existing concepts using the `refines` keyword. Build concept hierarchies for domain modeling — for example, a `DetailedInvoice` that refines a base `Invoice` concept, inheriting its structure while adding more specific semantics.

For detailed guidance, see [Define Your Concepts](../building-methods/concepts/define_your_concepts.md).


## Pipe Operators

# Pipe Operators

The workers that perform the actual processing in your pipelines.

## Overview

Pipe operators are the building blocks of Pipelex methods. Each operator performs a specific type of work: calling an LLM, structuring text into typed data, extracting text from documents, generating images, searching the web, composing data, or running custom code. Operators declare their input and output concepts, and the runtime validates type compatibility at definition time.

## PipeLLM

The core operator for LLM interaction. Supports text generation, structured output generation (two-step or direct JSON), vision (images and PDFs in prompts via `@variable` syntax), system prompts, and model presets. Works with any supported LLM provider.

See [PipeLLM reference](../building-methods/pipes/pipe-operators/PipeLLM.md).

## PipeStructure

Turns free-form text into a structured typed concept via a single LLM call. Takes one `Text`-compatible input and produces a typed object (or list of objects) matching the output concept. Use it when the text comes from elsewhere — a PDF extraction, a search result, or an upstream step — and you need typed JSON.

See [PipeStructure reference](../building-methods/pipes/pipe-operators/PipeStructure.md).

## PipeExtract

OCR and document extraction from PDFs and images. Supports multiple providers (pypdfium2, Mistral OCR, Azure Document Intelligence, docling, Deepseek-OCR), page rendering, embedded image extraction, and layout analysis. Swap providers by changing configuration, no code changes needed.

See [PipeExtract reference](../building-methods/pipes/pipe-operators/PipeExtract.md).

## PipeImgGen

Text-to-image and image-to-image generation using models like FLUX, GPT Image, Nano Banana, and others. Outputs are stored locally or in cloud storage with configurable URL signing.

See [PipeImgGen reference](../building-methods/pipes/pipe-operators/PipeImgGen.md).

## PipeSearch

Web search with structured results and source citations. Powered by Linkup or Pipelex Gateway, results come back as typed SearchResult concepts ready for downstream processing.

See [PipeSearch reference](../building-methods/pipes/pipe-operators/PipeSearch.md).

## PipeCompose

Deterministic object construction without an LLM. Compose outputs from working memory variables, fixed values, Jinja2 templates, and nested structures. Ideal for aggregating results from previous steps or building inputs for downstream pipes.

See [PipeCompose reference](../building-methods/pipes/pipe-operators/PipeCompose.md).

## PipeFunc

Execute custom Python functions within pipelines. Functions are auto-discovered via the `@pipe_func()` decorator and receive the current working memory as their input. Use PipeFunc when you need logic that goes beyond what declarative operators provide.

See [PipeFunc reference](../building-methods/pipes/pipe-operators/PipeFunc.md).


## Pipeline Orchestration

# Pipeline Orchestration

Controllers for building complex workflows from simple building blocks.

## Overview

Pipeline controllers define how pipes are assembled and executed. They handle sequencing, parallelism, iteration, and conditional branching — all declaratively in `.mthds` files. Controllers are themselves pipes, so they can be nested to build arbitrarily complex workflows.

## PipeSequence

Run pipes one after another, passing data through working memory. The most common controller for multi-step methods. Each step can read variables written by previous steps, enabling progressive data enrichment.

See [PipeSequence reference](../building-methods/pipes/pipe-controllers/PipeSequence.md).

## PipeParallel

Execute multiple independent pipes concurrently for faster throughput. Each branch starts with a deep copy of the current working memory, and their outputs are merged back when all branches complete.

See [PipeParallel reference](../building-methods/pipes/pipe-controllers/PipeParallel.md).

## PipeBatch

Apply the same pipe to every item in a list — the map operation for pipelines. Takes a list input and produces a list output, processing each item independently.

See [PipeBatch reference](../building-methods/pipes/pipe-controllers/PipeBatch.md).

## PipeCondition

Conditional branching based on Jinja2 expressions evaluated against working memory. Route execution to different pipes depending on runtime data, enabling dynamic method behavior.

See [PipeCondition reference](../building-methods/pipes/pipe-controllers/PipeCondition.md).

## Working Memory

Temporary storage for data flowing between pipes within a single execution. Variables are typed by concepts and scoped to the pipeline. Each pipe reads from and writes to working memory, creating a shared data context across the method.

See [Working Memory](../building-methods/pipes/working-memory.md).

## Multiplicity

Control how many items pipes accept and produce: single values, variable-length lists (`[]`), or fixed-count lists (`[N]`). Multiplicity is declared in the pipe definition and validated at definition time.

See [Understanding Multiplicity](../building-methods/pipes/understanding-multiplicity.md).


## Pipelex Gateway & Model Access

# Pipelex Gateway & Model Access

Access AI models through the Pipelex Gateway or bring your own API keys.

## Pipelex Gateway

A fully managed infrastructure providing unified access to AI models through a single API key. The Gateway eliminates the need to manage multiple provider configurations.

- **Single API key** for all supported models
- **Remote model catalog** — always access the latest models without updating Pipelex
- **Enterprise-grade architecture** — built for reliability and scale
- **Extensive provider support** — OpenAI, Google, Anthropic, Mistral, xAI, and more

Browse all supported models in the [Gateway Models](../setup/gateway-models.md) reference.

Get your Gateway API key at [app.pipelex.com](https://app.pipelex.com/) or [join the waitlist](https://go.pipelex.com/waitlist).

## Bring Your Own Keys

Direct integration with major providers using your own API keys:

- **OpenAI** — GPT-4o, GPT-4.1, o1, o3, o4-mini, etc.
- **Anthropic** — Claude Sonnet 4, Claude Haiku, etc.
- **Google** — Gemini 2.5 Pro, Gemini 2.5 Flash, etc.
- **Mistral** — Mistral Large, Mistral Medium, etc.
- **Azure OpenAI** — Azure-hosted OpenAI models
- **Amazon Bedrock** — AWS-hosted models
- **Vertex AI** — Google Cloud-hosted models
- **xAI** — Grok models
- **Portkey** — AI gateway for routing and observability
- **OpenRouter** — Multi-provider aggregator
- **BlackboxAI** — Blackbox-hosted models
- **fal** — Image generation models

## Open-Source Models

Run open-source models through dedicated providers:

- **Ollama** — Run open-source LLMs locally (Llama, Mistral, Qwen, etc.)
- **Hugging Face Inference** — including qwen-image for text-to-image
- **Scaleway** — Deepseek R1, Llama 3.3, Qwen3, GPT-OSS
- **Groq** — Llama-4, Kimi-K2-Instruct

## Routing Profiles

Routing profiles control which backend handles each model. Define pattern-based routes, defaults, and fallback orders in `routing_profiles.toml`. Switch a pipeline from one provider to another — or from Pipelex Gateway to your own keys — without changing the method definition.

See [Configure AI Providers](../get-started/configure-ai-providers.md) for setup details.

## Offline Behavior

The Gateway fetches its model catalog from a remote config service. Pipelex stays usable when that service is briefly unreachable:

- **Gateway disabled (Bring Your Own Keys)** — no remote fetch is attempted at all. Setup, validation, and dry-runs are fully offline.
- **Gateway enabled** — Pipelex primes an on-disk copy of the remote config at `~/.pipelex/cache/remote_config.json` on every successful fetch and during `pipelex init` while online. If a later fetch fails, it falls back to this cache so setup, validation, and `--dry-run` still complete. When the cache is in use, a `RemoteConfigStale` warning is emitted (the agent CLI surfaces it on the JSON `warnings` array).

Only the actual inference call needs the network at runtime — offline support covers setup and dry-run, not live model calls. If the Gateway is enabled, the fetch fails, and no cache has ever been primed, setup raises `RemoteConfigUnavailableError`: run `pipelex init` while online to prime the cache, or disable `pipelex_gateway` for permanent offline (BYOK) operation.


## LLM Integration

# LLM Integration

Deep integration with large language models for text generation, structured outputs, and vision tasks.

## Overview

Pipelex provides a unified interface for working with LLMs across providers. Write your prompt once, run it on any supported model. The runtime handles provider-specific API differences, token management, and response parsing.

## Structured Output Generation

Two approaches for structured outputs:

- **Two-step** — Generate text first, then parse into a structured concept. More reliable for complex schemas.
- **Direct JSON** — Generate structured JSON directly from the LLM using provider-native structured output features.

See [`PipeStructure`](../building-methods/pipes/pipe-operators/PipeStructure.md) for direct text-to-structure conversion, [`PipeLLM`](../building-methods/pipes/pipe-operators/PipeLLM.md) for direct JSON generation, and [Build-time Elaboration](../under-the-hood/build-time-elaboration.md) for how `structuring_method = "preliminary_text"` is rewritten into a two-step sequence at load time.

## Vision Language Models

Include images and PDFs directly in LLM prompts using `@variable` syntax. Support for single documents, multiple documents, and mixed content (text + images + PDFs). The runtime automatically handles document rendering and provider-specific vision API formats.

## Prompting Styles

Adapt prompts for different LLM families (OpenAI, Anthropic, Mistral) to get the best results from each provider. Pipelex applies provider-specific formatting, system prompt handling, and structured output instructions automatically.

See [LLM Prompting Style](../building-methods/adapt-to-llm-prompting-style-openai-anthropic-mistral.md).

## System Prompt Inheritance

Define system prompts at the domain level and have them automatically inherited by all PipeLLM operators in that domain. Individual pipes can override the domain-level system prompt when needed.

## Model Presets

Named configurations (temperature, max tokens, model selection, etc.) for consistent model behavior. Define presets once in the model deck, then reference them by name across all your pipelines using `$preset_name` syntax.

See [Optimize Cost & Quality](../building-methods/configure-ai-llm-to-optimize-methods.md).


## Document Extraction

# Document Extraction

Multi-provider OCR, document processing, and web content extraction with a unified interface.

## Overview

From simple text extraction to advanced document understanding — Pipelex handles it all. Basic PDF text extraction works out of the box (via pypdfium2), but real documents demand more: OCR for scanned pages, layout analysis for complex structures, image extraction, and VLM-powered understanding.

Unlike LLM APIs (partly standardized around OpenAI's completions API), the OCR landscape is fragmented. Pipelex solves this with a unified interface: swap providers by changing your PipeExtract config, no code changes required. For web pages, Linkup Fetch extracts content directly from URLs using the same PipeExtract pattern.

## Supported Providers

| Provider | Type | Description |
|----------|------|-------------|
| **pypdfium2** | Built-in | Basic PDF text and image extraction without AI inference — works out of the box with no API keys |
| **Mistral OCR** | Cloud API | Industry-leading document understanding for media, text, tables, and equations |
| **docling** | Local SDK | IBM's open-source extraction library with local CPU processing and optional GPU acceleration |
| **Azure Document Intelligence** | Gateway | Enterprise-grade OCR with high accuracy for complex layouts, tables, and handwriting |
| **Deepseek-OCR** | Gateway | Open-source model optimized for markdown extraction from images |
| **Linkup Fetch** | Cloud API | Web page content extraction — fetches and extracts text from web URLs |

## Key Capabilities

- **Page view generation** — High-fidelity image rendering of extracted pages via pypdfium2
- **Embedded image extraction** — Capture images found within documents
- **Layout analysis** — Structured extraction of complex document layouts
- **Table recognition** — Automatic table detection and extraction
- **Handwriting support** — Via providers that support handwriting recognition (e.g., Azure Document Intelligence)
- **Multi-page processing** — Batch processing of document pages with per-page results
- **Web page extraction** — Fetch and extract content from web page URLs via Linkup Fetch

## Documents in LLM Prompts

Include PDFs directly in your prompts using `@variable` syntax. PipeLLM automatically handles document rendering — single documents, multiple documents, and mixed content combining text, images, and PDFs are all supported. Web page content extracted via PipeExtract follows the same pattern.

## Related Documentation

- [PipeExtract](../building-methods/pipes/pipe-operators/PipeExtract.md) - Operator reference and MTHDS fields
- [Generic Document Extraction Example](../cookbook/extract-generic.md) - Extract markdown from complex PDFs using vision


## Image Generation

# Image Generation

Text-to-image generation integrated directly into your pipelines.

## Overview

PipeImgGen generates images from text prompts using state-of-the-art models. Generated images can be stored locally or in cloud storage with public or signed URLs. Image generation supports both text-to-image and image-to-image workflows.

## Supported Models

Via **Pipelex Gateway**:

- **GPT-Image-2 / GPT-Image-1.5 / GPT-Image-1 / GPT-Image-1-mini** — OpenAI image generation models
- **Nano Banana / Nano Banana Pro / Nano Banana 2 / Nano Banana 2 Lite** — Google Gemini-based image generation

Via **direct provider SDKs**:

- **OpenAI** — Direct OpenAI API for GPT Image models
- **Google Gemini** — Native Google image generation
- **fal** — FLUX models (Black Forest Labs) and others via the fal platform
- **Hugging Face Inference** — Open-source models like qwen-image
- **BlackboxAI** — Via completions-based image generation
- **Azure REST** — Azure-hosted image generation
- **OpenRouter** — Multi-provider image generation access

## Cloud Storage Integration

Generated images can be automatically uploaded to AWS S3 or Google Cloud Storage with configurable URL signing. See [Cloud Storage](cloud-storage.md) for details.

## Usage in Pipelines

Use PipeImgGen in your `.mthds` files to generate images as part of a pipeline. The operator takes a `prompt` string template and outputs an `Image` concept. Declared inputs are injected into the template — `Text` variables as interpolated text, and `Image` variables as reference images for image-to-image generation. Model presets let you configure quality levels and default models in `2_img_gen_deck.toml`.

See [PipeImgGen reference](../building-methods/pipes/pipe-operators/PipeImgGen.md).


## Web Search

# Web Search

Structured web search integrated into your pipelines.

## Overview

PipeSearch brings web search directly into your methods. Results come back as structured data with source citations, making them ready for downstream processing by LLMs or other pipes. Search is powered by Linkup (direct SDK) or via Pipelex Gateway.

## Key Capabilities

- **Structured results** — Search results returned as typed SearchResult concepts with an answer and sourced citations
- **Source citations** — Automatic source tracking for provenance and attribution
- **Pipeline integration** — Feed search results directly into PipeLLM for grounded generation with real-time data

## Usage in Pipelines

Use PipeSearch in a PipeSequence to retrieve information from the web and then process it with an LLM. The search results are stored in working memory as SearchResult concepts, available to downstream pipes.

## Related Documentation

- [PipeSearch](../building-methods/pipes/pipe-operators/PipeSearch.md) - Operator reference and supported output shapes
- [Pipeline Orchestration](./pipeline-orchestration.md) - Compose search with downstream pipes
- [Working Memory](../building-methods/pipes/working-memory.md) - How search outputs flow through a method


## Claude Code Skills Plugin

# Claude Code Skills Plugin

Build, run, validate, and edit AI methods directly from Claude Code.

## Overview

The **MTHDS skills plugin** for Claude Code brings the full Pipelex workflow into your AI coding assistant. Build methods from natural language, validate them, run them, and iterate through conversational slash commands. The plugin leverages the `pipelex-agent` CLI under the hood.

Install from [mthds-ai/mthds-plugins](https://github.com/mthds-ai/mthds-plugins).

## Available Commands

| Command | Description |
|---------|-------------|
| `/mthds-build` | Build new AI method bundles from scratch |
| `/mthds-check` | Validate workflow bundles for issues (read-only) |
| `/mthds-edit` | Modify existing bundles — change pipes, update prompts, add or remove steps |
| `/mthds-explain` | Explain and document existing workflows in plain language |
| `/mthds-fix` | Automatically fix validation errors in a loop |
| `/mthds-run` | Execute methods and interpret the results (supports dry runs and graph generation) |
| `/mthds-inputs` | Prepare inputs: placeholder templates, synthetic test data, or user-provided files |
| `/mthds-install` | Install method packages from GitHub or local directories |
| `/mthds-pkg` | Manage package manifests and related package workflows |
| `/mthds-publish` | Publish methods to mthds.sh |

## Getting Started

```
/plugin marketplace add mthds-ai/mthds-plugins
```

then:

```
/plugin install mthds@mthds-plugins
```

Then reload plugins with `/reload-plugins` (or exit and reopen Claude Code if that doesn't work).

Once installed, type `/mthds-build` in Claude Code to create your first method from a natural language description.

## Related Documentation

- [Agent CLI](../tools/cli/agent-cli.md) - Machine-oriented interface used under the hood
- [Packages](../building-methods/packages.md) - `METHODS.toml` manifest workflow
- [Package Commands](../tools/cli/pkg.md) - Current `mthds package` commands


## CLI

# Command-Line Interface

A comprehensive CLI for developing, validating, and running AI methods.

## Overview

The `pipelex` CLI is the primary tool for working with Pipelex methods. It covers the full development lifecycle: authentication, initialization, building, validation, execution, inspection, and diagnostics.

## Core Commands

| Command | Description |
|---------|-------------|
| **`pipelex login`** | Authenticate with Pipelex Gateway via the browser and save your API key |
| **`pipelex init`** | Initialize configuration, backends, credentials, routing, and telemetry |
| **`pipelex update`** | Refresh the model deck to match the installed pipelex version |
| **`pipelex doctor`** | Check configuration health and suggest fixes |
| **`pipelex build`** | AI-powered method generation from natural language requirements |
| **`pipelex validate`** | Check pipeline syntax, structure, and run dry-run validation |
| **`pipelex fix`** | Apply deterministic safe fixes to a bundle and re-validate (with `--diff` preview) |
| **`pipelex run`** | Execute pipelines from bundle files or libraries |
| **`pipelex graph`** | Generate and render execution graphs |
| **`pipelex show`** | Display configuration, pipes, and list AI models |
| **`pipelex which`** | Locate where a pipe is defined, similar to `which` for executables |

## Related CLI

Package manifest management is currently exposed through the lowercase `mthds` CLI:

- `mthds package init`
- `mthds package list`
- `mthds package validate`

## Execution Options

- **Dry run** — `--dry-run` executes with mocked LLM responses to test pipeline logic without API calls
- **Mock inputs** — `--mock-inputs` generates synthetic inputs so you can test a pipeline without preparing real data (requires `--dry-run`)
- **Graph generation** — `--graph`, `--graph-full-data`, `--graph-no-data` for visual execution inspection

## Agent CLI

The `pipelex-agent` CLI is a machine-first interface designed for automated environments like Claude Code skills. Output format varies by command — markdown or JSON, raw TOML, or `plxt` passthrough — with no interactive prompts or Rich formatting. Structured commands emit errors to stderr; the error format is controlled by `--error-format` and defaults to the value of `--format` (so `--format json` flips both). `fmt` and `lint` pass through native `plxt` output.

| Command | Description |
|---------|-------------|
| `init` | Non-interactive configuration setup (`--format markdown\|json` success, default: markdown; `--error-format` for errors, defaults to `--format`'s value) |
| `run` | Execute a pipeline (`--format markdown\|json` success, default: markdown; `--error-format` for errors, defaults to `--format`'s value) |
| `validate` | Validate pipes, bundles, or methods (`--format markdown\|json` success, default: markdown; `--error-format` for errors, defaults to `--format`'s value) |
| `fix` | Apply deterministic safe fixes to a bundle in place and re-validate (`--format markdown\|json` success, default: markdown; `--error-format` for errors, defaults to `--format`'s value) |
| `fmt` | Format `.mthds`, `.toml`, or `.plx` files in-place |
| `lint` | Lint files for errors |
| `inputs` | Generate example input JSON for a pipe |
| `concept` | Convert a JSON concept spec into raw TOML (stdout) |
| `pipe` | Convert a JSON pipe spec into raw TOML (stdout) |
| `models` | List available model presets, aliases, and waterfalls (`--format markdown\|json` success, default: markdown; `--error-format` for errors, defaults to `--format`'s value) |
| `doctor` | Check configuration health (`--format markdown\|json` success, default: markdown; `--error-format` for errors, defaults to `--format`'s value) |

For detailed CLI documentation, see the [CLI reference](../tools/cli/index.md).

## Related Documentation

- [CLI Reference](../tools/cli/index.md) - Runtime CLI commands
- [Agent CLI](../tools/cli/agent-cli.md) - Machine-oriented interface for AI agents
- [Package Commands](../tools/cli/pkg.md) - Current `mthds package` manifest commands


## plxt Formatter & Linter

# plxt Formatter & Linter

A Rust-based formatter and linter for TOML that supports schema-based validation, customized for the MTHDS language.

## Overview

`plxt` is a fast, Rust-based TOML formatter and linter with schema-based validation support. It has been customized to understand the MTHDS language, providing formatting and linting tailored to `.mthds` and `.plx` files. It is distributed as part of the `pipelex-tools` package.

## Formatting

- **TOML files** — Standard TOML formatting
- **MTHDS files** — Schema-aware formatting for method definitions
- **PLX files** — Pipelex configuration formatting
- **Alignment options** — Configurable alignment for entries and comments
- **Per-file-type rules** — Different formatting rules for different file types

## Linting

Schema-based validation that catches structural issues beyond basic syntax checking. Detects invalid keys, missing required fields, type mismatches, and MTHDS-specific violations such as unknown pipe operators. Cross-reference checks, such as resolving concept references, are handled by `pipelex validate`, not `plxt lint`.

## CI Integration

Use `plxt fmt --check` to validate formatting without modifying files, and run `plxt lint` separately for schema-based checks. Both commands exit with a non-zero code on errors, making them easy to enforce in automated workflows.

For details, see the [plxt reference](../tools/plxt.md).


## Execution Graph Visualization

# Execution Graph Visualization

Full transparency into pipeline execution with interactive visualizations.

## Overview

Every pipeline execution can be visualized as an interactive graph, giving developers full transparency into what happened, in what order, and with what data at each step. Running with the `--graph` flag writes the graph outputs (graph spec JSON, Mermaid chart, interactive HTML) to the run's output directory. To render a saved graph spec and open it in your browser, use `pipelex graph render <graph.json> --open`.

## Interactive HTML Visualization

Inspect any pipeline execution with a local ReactFlow-based interface. Nodes represent pipes, edges show data flow, and clicking any node reveals the data at that stage. The visualization supports zooming, panning, and collapsing nested controllers.

## Mermaid Chart Export

Render pipeline diagrams anywhere that supports Mermaid: VS Code, GitHub, web applications, and documentation. Mermaid charts provide a static, embeddable view of the pipeline structure.

## Step-by-Step Data Inspection

View the actual data at each execution stage:

- **JSON** — Raw structured data
- **HTML preview** — Rendered content
- **Images** — Generated or extracted images
- **Embedded PDFs** — Document content

## CLI Flags

- `--graph` — Generate an execution graph after running a pipeline
- `--graph-full-data` — Include full data payloads in the graph
- `--graph-no-data` — Generate a structural graph without data

<!-- path verified: resolves to docs/images/flow-chart-example.png -->
![Execution Graph Example](../images/flow-chart-example.png){ width="400" }


## Validation & Dry Run

# Validation & Dry Run

Test your pipelines without making API calls.

## Overview

Pipelex provides multiple layers of validation to catch issues before they cost time and money. From static syntax checking to full dry-run execution with mocked responses, you can verify your methods at every stage of development.

## Pipeline Validation

Check pipeline syntax, structure, and compatibility without execution:

- **Syntax validation** — Catch MTHDS language errors via plxt linting
- **Structure validation** — Resolve pipe and concept references in the loaded library, and verify each pipe's declared inputs against the concepts that operator accepts — plus a controller's declared output against what it actually produces (for a `PipeSequence`, its last step's output concept and multiplicity)
- **Input validation** — Ensure required inputs are provided and correctly typed, and that every variable a step reads is bound by the time that step runs

A pipe that references a sub-pipe from a package that isn't loaded is reported as **skipped** rather than validated, so a passing run is not proof that every cross-package dependency resolves.

!!! note "Steps read from working memory, not from the previous step"
    A `PipeSequence` step resolves its inputs by name from working memory. They may come from the pipeline's inputs or from any earlier step — not necessarily the one immediately before it. Validation therefore checks that each name is bound when the step runs, rather than matching one step's output concept to the next step's declared input concept. There is no output-to-input chaining contract between consecutive steps to verify.

## Dry Run Mode

Execute pipelines with mocked LLM responses to test pipeline logic, data flow, and orchestration without making API calls.

- **Mock generation** — Format-compliant mock values for constrained fields, including structured outputs
- **Configurable mock behavior** — Control mock list sizes, template handling, and response formats
- **Full pipeline execution** — Working memory, controllers, and data flow all work as in production

## Allowed-to-Fail Pipes

List specific pipe codes in `pipelex.dry_run_config.allowed_to_fail_pipes` so dry-run validation can tolerate expected failures without failing the overall validation pass.

## CLI Usage

- `pipelex validate` — Static validation
- `pipelex run pipe my_pipe --dry-run` — Dry run execution
- `pipelex run pipe my_pipe --dry-run --mock-inputs` — Generate synthetic inputs

For configuration details, see [Dry Run Configuration](../configuration/config-pipeline-validation/dry-run-config.md).

Dry run is one of two run modes (live, dry run), each of which works in-process or distributed on a durable backend — see [Run Modes & Backends](../building-methods/pipes/run-modes-and-backends.md) for the full matrix.


## Telemetry & Observability

# Telemetry & Observability

Production-ready monitoring for your AI methods.

## Overview

Pipelex provides comprehensive telemetry and observability capabilities to monitor your AI methods in production. Track costs, latency, errors, and execution patterns across multiple destinations. Custom telemetry is opt-in and configurable per integration mode (CLI, pytest, API). When Pipelex Gateway is enabled as an inference backend, privacy-respecting Gateway metrics are collected automatically.

## Langfuse Integration

Full LLM observability with complete span data. Track every LLM call, its inputs, outputs, tokens, cost, and latency. Supports both Langfuse Cloud and self-hosted instances via the OTLP exporter.

## OpenTelemetry (OTLP)

Send execution spans to any OTLP-compatible backend for integration with your existing observability stack. Configure multiple OTLP exporters with custom endpoints and headers to fan out telemetry to different destinations.

## PostHog Integration

Event tracking and AI span tracing with fine-grained privacy controls. Choose between anonymous and identified modes. Configure what data to capture: content, pipe codes, output class names, and content length limits.

## Gateway Telemetry

When using Pipelex Gateway as your inference backend, privacy-respecting metrics are automatically collected: models used, token counts, latency, and error rates. This data is tied to your Gateway API key (hashed for security) and requires no additional configuration.

## Privacy Controls

- **DO_NOT_TRACK** — Universal telemetry disable flag, respected by all integrations
- **Configurable destinations** — Choose independently what data goes to PostHog, Langfuse, OTLP, or Gateway
- **Content redaction** — Control whether prompt/completion content is included in spans
- **Mode-based gating** — Enable custom telemetry for CLI usage but disable it during unit tests

For configuration, see [Telemetry Configuration](../configuration/config-practical/telemetry-config.md) and [Telemetry Setup](../setup/telemetry.md).


## Cloud Storage

# Cloud Storage

Store generated artifacts on cloud infrastructure.

## Overview

Pipelex can automatically upload generated artifacts — images, extracted pages, and other outputs — to cloud storage providers with configurable access control. For development and testing, local filesystem and in-memory storage are also available.

## Supported Providers

- **Local filesystem** — Store artifacts on the local disk (default for development)
- **In-memory** — Ephemeral storage for testing, no persistence
- **AWS S3** — Amazon Simple Storage Service with configurable region and bucket
- **Google Cloud Storage** — GCS buckets with project-level credentials

## URL Options

- **Public URLs** — Directly accessible URLs for public content
- **Signed URLs** — Time-limited secure URLs for private content, with configurable lifespan

## Configuration

Storage is configured in `pipelex.toml` under `[pipelex.storage_config]`. Set `method` to `local`, `in_memory`, `s3`, or `gcp`, then provide provider-specific settings such as `bucket_name`, `region`, or `project_id` in the matching subsection.

For AWS configuration, see [AWS Configuration](../configuration/config-technical/aws-config.md).


## Cost Tracking & Reporting

# Cost Tracking & Reporting

Automatic cost tracking for every AI operation in your methods.

## Overview

Pipelex automatically tracks token usage and calculates costs for every LLM call, extraction operation, and image generation. Stay informed about your AI spend at the method, pipe, and operation level.

## Console Reporting

Real-time cost information displayed in the console during pipeline execution. Each inference call reports its token usage and estimated cost, with a summary total at the end of the run.

## CSV Export

Export detailed cost reports as CSV files for analysis, billing, and audit trails. Each row captures the pipe, model, token counts (input/output), and cost for a single operation.

## Rendering Reports When Embedding

Cost is part of the run result: a finished run carries its assembled usage on `pipe_output.tokens_usages`. If you embed Pipelex and want the same end-of-run console table / CSV that the CLI prints, render it in one call from the output:

```python
from pipelex.reporting.cost_report_renderer import render_cost_report_for_output

render_cost_report_for_output(pipe_output)
```

The `--costs` gate is read off the output itself — `tokens_usages` is `None` exactly when cost reporting was off for the run — so you never re-read global config to reconstruct whether costs were on.

## Per-node Attribution in the Run Graph

The same usage stream also lands on the run graph: every node of `pipe_output.graph_spec` carries a `usage` object with its own inference calls, tokens and cost, plus a `subtree_*` rollup covering it and everything below it — so a controller shows what its whole branch spent. The graph's `usage.total` is the run-level rollup.

Cost there is deliberately three-valued: `null` means *unrated* (no rate table — own-GPU models, dry and mock runs), never `0.00`. See [Per-node Usage Attribution](../under-the-hood/per-node-usage-attribution.md) for the invariants a consumer must branch on.

## Custom Reporting

Implement custom reporting backends via the [Reporting Delegate](advanced-customizations.md) injection point. The reporting protocol receives structured cost events during execution, allowing you to push data to your own analytics, dashboards, or billing systems.

For configuration, see [Reporting Configuration](../configuration/config-practical/reporting-config.md).

## Usage on the API Wire

API clients (the `/execute` response and the durable `tokens_usages.json` result artifact) receive token usage in a trimmed, client-facing record shape with a computed per-call `cost` (`null` for unrated models) — not the full internal usage models this page describes. See [TokensUsage Wire Records](../under-the-hood/tokens-usage-wire-records.md).


## Configuration System

# Configuration System

Multi-level TOML configuration for full control over Pipelex behavior.

## Overview

Pipelex uses a layered TOML configuration system that lets you define sensible defaults and override them at any level: project, environment, or run mode. Each layer merges on top of the previous one, so you only need to specify what you want to change.

## Configuration Levels

1. **Package defaults** — Built into Pipelex (`pipelex.toml` in the package)
2. **Global tier** — `~/.pipelex/` for user-wide settings: the base `pipelex.toml`, then its override sequence
3. **Project tier** — `{project_root}/.pipelex/` for project-specific settings: the base `pipelex.toml`, then the same override sequence

Within each tier, the base `pipelex.toml` is followed by an override sequence, merged in order (later wins per key):

- `pipelex_local.toml` — git-ignored, for machine-specific values
- `pipelex_{environment}.toml` — e.g. `pipelex_dev.toml`, `pipelex_prod.toml`. The environment is selected by the `PIPELEX_ENV` environment variable (accepted values: `local`, `dev`, `staging`, `prod`; defaults to `dev`).
- `pipelex_{run_mode}.toml` — e.g. `pipelex_unit_test.toml`, `pipelex_ci_test.toml`. The run mode is selected by the `RUN_MODE` environment variable (defaults to `normal`).
- `pipelex_override.toml` — general-purpose overrides
- `pipelex_temporary_override.toml` — last-resort, highest-precedence overrides within the tier

The entire global tier merges before the project tier, so any project-tier file — including the plain project `pipelex.toml` — overrides every global-tier file.

## Environment Variables

Two configuration areas support `${...}` variable substitution in their TOML files: the inference backend configuration (`.pipelex/inference/backends.toml` and the per-backend TOML files under `.pipelex/inference/backends/`) and the telemetry configuration (`.pipelex/telemetry.toml`). In those files, use `${env:VAR}` for environment variables, `${secret:VAR}` for the secrets provider, and fallback chains like `${env:VAR|secret:VAR}`. The `pipelex.toml` layering described above does not perform substitution — a `${env:VAR}` there stays a literal string.

## Key Configuration Areas

- **[Inference Backends](../configuration/config-technical/inference-backend-config.md)** — LLM providers, models, and routing
- **[Logging](../configuration/config-practical/logging-config.md)** — Log levels and output
- **[Telemetry](../configuration/config-practical/telemetry-config.md)** — Observability settings
- **[Reporting](../configuration/config-practical/reporting-config.md)** — Cost tracking and reports
- **[Dry Run](../configuration/config-pipeline-validation/dry-run-config.md)** — Mock generation settings

For full configuration reference, see [Configuration](../configuration/index.md).


## Advanced Customizations

# Advanced Customizations

A dependency injection framework for extending and customizing Pipelex behavior.

## Overview

Pipelex provides well-defined injection points that let you replace or extend core behaviors without modifying the framework itself. Each injection point follows a strict protocol contract, making implementations testable and swappable. Register custom providers at initialization time and the runtime uses them throughout execution.

## Injection Points

| Injection Point | Purpose |
|-----------------|---------|
| **[Secrets Provider](../advanced/secrets-provider-injection.md)** | Custom secret management (environment, vaults, etc.) |
| **[Storage Provider](../advanced/storage-provider-injection.md)** | Custom cloud storage backends |
| **[Observer](../advanced/observer-provider-injection.md)** | Custom execution data capture |
| **[Reporting Delegate](../advanced/reporting-delegate-injection.md)** | Custom cost reporting |
| **[Content Generator](../advanced/content-generator-injection.md)** | Override LLM output generation |
| **[Pipe Router](../advanced/pipe-router-injection.md)** | Dynamic routing of pipes to implementations |

## Built-in Defaults

Some injection points provide safe built-in defaults or no-op implementations, so you only need to implement the ones relevant to your use case.

For detailed documentation, see [Advanced Customizations](../advanced/index.md).


# Building Methods

## Kick off a Method Project

# Kicking off a Pipelex Method Project

## Creating Your First Pipeline

A pipeline in Pipelex is a collection of related concepts and pipes. Start by creating a MTHDS file in your project:

`tutorial.mthds`
```toml
domain = "tutorial"
description = "My first Pipelex library"
system_prompt = "You are a helpful assistant."

[concept]
Question = "A question that needs to be answered"
Answer = "A response to a question"

[pipe]
[pipe.answer_question]
type = "PipeLLM"
description = "Answer a question"
inputs = { question = "tutorial.Question" }
output = "tutorial.Answer"
prompt = """
Please answer the following question:

@question

Provide a clear and concise answer.
"""
```

This creates a simple Q&A pipeline with:

- A domain called "tutorial"
- Two concepts: Question and Answer
- One pipe that transforms a Question into an Answer

The `domain` property is the most important part of your pipeline file. It groups all your concepts and pipes into a single, easy to read bundle.

!!! info "Learn More About Domains"
    Domains are a core concept in Pipelex that organize your pipelines, concepts, and pipes into semantic namespaces. For a comprehensive guide on domains, see [Understanding Domains](./domain.md).

See more about designing pipes in [Designing Pipelines](./pipes/index.md).
See more about concepts in [Define Your Concepts](./concepts/define_your_concepts.md)
See more about domains in [Understanding Domains](./domain.md)

## File Naming Conventions

Consistent naming makes your pipeline code discoverable and maintainable:

### MTHDS Files
- Use lowercase with underscores: `legal_contracts.mthds`, `customer_service.mthds`
- Match the domain name when possible: domain "legal" → `legal.mthds`
- For multi-word domains, use underscores: domain "customer_service" → `customer_service.mthds`

See more about pipelex bundle specification in [Pipelex Bundle Specification](./pipelex-bundle-specification.md)

### Python Model Files
- It is recommended to name structure files with a `_struct.py` suffix: `legal.mthds` → `legal_struct.py`
- Pipelex will automatically discover and load structure classes from all Python files in your project (excluding common directories like `.venv`, `.git`, etc.)

## Project Structure

**Key principle:** Put `.mthds` files where they belong in YOUR codebase. Pipelex automatically finds them.

### Recommended Patterns

**Topic-Based (Best for organized codebases):**
```
your-project/
├── my_project/                # Your Python package
│   ├── finance/
│   │   ├── models.py
│   │   ├── services.py
│   │   ├── invoices.mthds          # Pipeline with finance code
│   │   └── invoices_struct.py    # Structure classes
│   └── legal/
│       ├── models.py
│       ├── contracts.mthds         # Pipeline with legal code
│       └── contracts_struct.py
├── .pipelex/                     # Config at repo root
│   ├── pipelex.toml
│   └── inference/
└── requirements.txt
```

**Centralized (If you prefer grouping pipelines):**
```
your-project/
├── my_project/
│   ├── pipelines/              # All pipelines together
│   │   ├── finance.mthds
│   │   ├── finance_struct.py
│   │   ├── legal.mthds
│   │   └── legal_struct.py
│   └── core/
│       └── (your code)
└── .pipelex/
```

**Flat (Small projects):**
```
your-project/
├── my_project/
│   ├── invoice_pipeline.mthds
│   ├── invoice_struct.py
│   └── main.py
└── .pipelex/
```

### Key Points

- **Flexible placement**: `.mthds` files work anywhere in your project
- **Automatic discovery**: Pipelex scans and finds them automatically
- **Configuration location**: `.pipelex/` stays at repository root
- **Naming convention**: Use `_struct.py` suffix for structure files
- **Excluded directories**: `.venv`, `.git`, `__pycache__`, `node_modules` are skipped
- **Best practice**: Keep related pipelines with their related code


## Pipelex Bundle Specification

# Pipelex Bundle Specification

!!! tip "MTHDS Standard Reference"
    The `.mthds` bundle format is defined by the MTHDS open standard. For the authoritative language specification, see [Bundles](https://mthds.ai/latest/language/bundles/) on mthds.ai. This page documents Pipelex-specific behavior and usage.

A **Pipelex bundle** is the fundamental unit of organization in Pipelex. It's a single `.mthds` file that defines a cohesive set of concepts and pipes for a specific domain of work.

## What is a Pipelex Bundle?

A Pipelex bundle (`.mthds` file) brings together:

- **Domain declaration** - The semantic namespace for all concepts and pipes in this bundle
- **Concepts** - The knowledge structures that flow through your pipes (optional)
- **Pipes** - The processing units that transform and manipulate your concepts (optional)

Think of a bundle as a self-contained module that solves a specific problem domain. For example, you might have:

- `invoice_processing.mthds` - Bundle for invoice extraction and validation
- `marketing.mthds` - Bundle for generating marketing content
- `document_analysis.mthds` - Bundle for analyzing documents

## Bundle Structure

Every Pipelex bundle follows this structure:

```toml
# 1. Domain Declaration (MANDATORY)
domain = "domain_code"
description = "Description of what this domain handles"

# 2. Optional: Main Pipe Declaration
main_pipe = "pipe_code"

# 3. Concept Definitions (OPTIONAL)
[concept]
ConceptName1 = "Description"
ConceptName2 = "Description"

# 4. Pipe Definitions (OPTIONAL)
[pipe]
[pipe.pipe_code]
type = "PipeLLM"
# ... pipe configuration

[pipe.pipe_code_2]
...
```

### 1. Domain Declaration (MANDATORY)

Every bundle **must** declare a domain. Only the `domain` field is mandatory; all other fields are optional:

```toml
domain = "invoice_processing"
description = "Tools for extracting and validating invoice data"
system_prompt = "You are an expert in financial document processing."
main_pipe = "extract_and_validate_invoice"
```

**Bundle Header Fields:**

- **`domain`** (required) - The unique identifier for this bundle's namespace
- **`description`** (Optional) - A human-readable description of what this bundle does
- **`system_prompt`** (Optional) - A system prompt that applies to all `PipeLLM` operators in this bundle
- **`main_pipe`** (Optional) - The default pipe to execute when no specific pipe is requested when running the bundle (see "About `main_pipe`" below)

**Why domains matter:**

- Domains create unique namespaces for your concepts and pipes
- They prevent naming conflicts when multiple bundles define concepts with the same name
- They enable organized, modular pipeline architecture

Learn more about domains in [Understanding Domains](./domain.md).

**About `system_prompt`:**

When you define a `system_prompt` at the bundle level, it automatically applies to every `PipeLLM` operator in the bundle. This is useful for setting domain-specific context:

```toml
domain = "medical_records"
system_prompt = "You are a medical records specialist with expertise in HIPAA compliance and clinical documentation."

[concept]
MedicalRecord = "A patient's medical record"
Diagnosis = "A medical diagnosis extracted from a record"

[pipe.extract_diagnosis]
type = "PipeLLM"
# This pipe will automatically use the bundle's system_prompt
description = "Extract the primary diagnosis from a medical record"
inputs = { record = "MedicalRecord" }
output = "Diagnosis"
prompt = """
Extract the primary diagnosis from this medical record:

@record
"""
```

Individual pipes can override the bundle's system prompt by defining their own `system_prompt` field.

**About `main_pipe`:**

You can specify a default pipe to execute when no specific pipe is requested:

```toml
domain = "invoice_processing"
main_pipe = "extract_and_validate_invoice"
```

**How `main_pipe` works:**

- When executing a bundle without specifying a `pipe_code`, Pipelex runs the `main_pipe`
- If no `main_pipe` is specified, you must explicitly provide the `pipe_code` when executing
- The `main_pipe` value must reference a pipe defined in the same bundle

See more about executing pipes in [Executing Pipelines](./pipes/executing-pipelines.md).

### 3. Concept Definitions (OPTIONAL)

Concepts define the knowledge structures in your domain. While optional, most bundles define at least a few concepts (See more about concepts in [Define Your Concepts](./concepts/define_your_concepts.md)):

```toml
[concept]
Invoice = "A commercial document issued by a seller to a buyer"
Vendor = "A company or individual that provides goods or services"
LineItem = "An individual item or service listed in an invoice"
```

**Concept Naming:**

Within a bundle, you reference concepts by their simple name (`Invoice`). However, their **full identifier** includes the domain code:

```
domain_code.ConceptName
```

For example, if your domain_code is `invoice_processing`, the concept `Invoice` has the full identifier:

```
invoice_processing.Invoice
```

This full identifier is how other bundles reference your concepts.

### 4. Pipe Definitions (OPTIONAL)

Pipes are the processing units that transform data. Like concepts, they're optional:

```toml
[pipe.extract_invoice]
type = "PipeExtract"
description = "Extract text and images from an invoice PDF"
inputs = { document = "Document" }
output = "Page[]"
```
See more about designing pipes in [Designing Pipelines](./pipes/index.md).

## Referencing Concepts: When to Use Domain code prefixes

Understanding when to use domain code prefixes is crucial for writing clean, maintainable bundles.

### Same-Bundle References (No Prefix Needed)

When a pipe in a bundle references a concept **defined in the same bundle**, you don't need the domain code prefix:

```toml
domain = "invoice_processing"

[concept]
Invoice = "A commercial document issued by a seller to a buyer"
ValidationResult = "Result of validating an invoice"

[pipe.validate_invoice]
type = "PipeLLM"
description = "Validate invoice data"
inputs = { invoice = "Invoice" }          # ✅ No prefix needed
output = "ValidationResult"               # ✅ No prefix needed
prompt = "Validate this invoice: $invoice"
```

This is the most common case and keeps your code clean.

### Cross-Bundle References (Prefix Required)

When a pipe references a concept **from another bundle/domain**, you must use the full domain code prefix:

```toml
domain = "accounting"

[concept]
Payment = "A financial transaction recording money transfer"
ReconciliationReport = "Report of payment reconciliation"
```

```toml
[pipe.reconcile_payment]
type = "PipeLLM"
description = "Reconcile a payment with an invoice"
inputs = { 
    invoice = "invoice_processing.Invoice",      # ✅ Different domain, needs prefix
    payment = "Payment"                          # ✅ Same domain, no prefix
}
output = "ReconciliationReport"
prompt = """
Reconcile this payment with the invoice:

@invoice

@payment
"""
```

## Bundles Working Together

One of Pipelex's strengths is how bundles can reference and build upon each other, creating modular, reusable pipeline architectures.

When you execute pipelines, multiple bundles are loaded together into a **Library** - a runtime collection that provides access to all domains, concepts, and pipes from the loaded bundles. This library system enables bundle composition and ensures proper isolation between different pipeline runs. Learn more about how libraries work in [Libraries](libraries.md).

### Calling Pipes from Other Bundles

A pipe in one bundle can call a pipe from another bundle using the full pipe identifier:

```toml
domain = "accounts_payable"

[pipe.process_vendor_invoice]
type = "PipeSequence"
description = "Process a vendor invoice end-to-end"
inputs = { pdf_document = "Document" }
output = "PaymentSchedule"
steps = [
    # Call pipe from invoice_processing domain
    { pipe = "invoice_processing.extract_invoice", result = "extracted_data" },
    
    # Call pipe from validation domain
    { pipe = "validation.validate_data", result = "validation_report" },
    
    # Call pipe from this domain (no prefix needed)
    { pipe = "schedule_payment", result = "payment_schedule" }
]

[pipe.schedule_payment]
type = "PipeLLM"
description = "Schedule payment based on invoice data"
inputs = { 
    extracted_data = "invoice_processing.Invoice",   # Concept from another domain
    validation_report = "validation.Report"           # Concept from another domain
}
output = "PaymentSchedule"
prompt = "..."
```

## Related Documentation

- [Understanding Domains](./domain.md) - Deep dive into domain organization
- [Designing Pipelines](./pipes/index.md) - Learn how to design and compose pipes
- [Define Your Concepts](./concepts/define_your_concepts.md) - Complete guide to concept definitions
- [Kick off a Pipelex Method Project](./kick-off-a-methods-project.md) - Start a new project



## Domain

# Understanding Domains

!!! tip "MTHDS Standard Reference"
    Domains are part of the MTHDS open standard. For the authoritative language specification, see [Domains](https://mthds.ai/latest/language/domains/) on mthds.ai. This page documents Pipelex-specific behavior and usage.

A domain in Pipelex is a **semantic namespace** that organizes related concepts and pipes. It's declared at the top of every `.mthds` file and serves as an identifier for grouping related functionality.

## What is a Domain?

A domain is defined by three properties:

- **`code`** (required) - The unique identifier for the domain (must be snake_case)
- **`description`** (optional) - A human-readable description of the domain's purpose
- **`system_prompt`** (optional) - A default system prompt for all PipeLLM operators in bundles using this domain

## Declaring a Domain

Every `.mthds` file must declare its domain at the beginning:

```toml
domain = "invoice_processing"
description = "Tools for extracting, validating, and processing invoice documents"
system_prompt = "You are an expert in financial document analysis and invoice processing."
```

!!! important "Domain Code Naming Convention"
    Domain codes **MUST** be in `snake_case` (lowercase with underscores). Use descriptive names that clearly indicate the domain's purpose.
    
    **Valid domain codes:**
    ```toml
    ✅ domain = "finance"
    ✅ domain = "invoice_processing"
    ✅ domain = "medical_records"
    ✅ domain = "customer_service"
    ```
    
    **Invalid domain codes:**
    ```toml
    ❌ domain = "Finance"           # No uppercase
    ❌ domain = "invoice-processing" # No hyphens
    ❌ domain = "INVOICE_PROCESSING" # No all caps
    ❌ domain = "invoiceProcessing"  # camelCase not allowed
    ```

## Hierarchical Domains

Domains support **dotted paths** to express a hierarchy:

```toml
domain = "legal"
domain = "legal.contracts"
domain = "legal.contracts.shareholder"
```

Each segment must be `snake_case`. The hierarchy is organizational — there is no scope inheritance between parent and child domains. `legal.contracts` and `legal` are independent namespaces; defining concepts in one does not affect the other.

**Valid hierarchical domains:**

```toml
✅ domain = "legal.contracts"
✅ domain = "legal.contracts.shareholder"
✅ domain = "finance.reporting"
```

**Invalid hierarchical domains:**

```toml
❌ domain = ".legal"              # Cannot start with a dot
❌ domain = "legal."              # Cannot end with a dot
❌ domain = "legal..contracts"    # No consecutive dots
❌ domain = "Legal.Contracts"     # Segments must be snake_case
```

Hierarchical domains are used in the `[exports]` section of `METHODS.toml` to control pipe visibility across domains. See [Packages](./packages.md) for details.

## How Domains Work

### Concept Namespacing

Every concept defined in a bundle belongs to the domain declared in that bundle. The full identifier for a concept is:

```
domain_code.ConceptName
```

**Example:**

```toml
domain = "finance"

[concept]
Invoice = "A commercial document for a sale"
Payment = "A financial transaction"
```

This creates two concepts:
- `finance.Invoice`
- `finance.Payment`

### Domain as Namespace

The domain code prevents naming conflicts. Multiple bundles can define concepts with the same name if they're in different domains:

```toml
# finance.mthds
domain = "finance"
[concept]
Report = "A financial report"
```

```toml
# marketing.mthds
domain = "marketing"
[concept]
Report = "A marketing campaign report"
```

Result: Two different concepts (`finance.Report` and `marketing.Report`) with no conflict.

### Multiple Bundles, Same Domain

Multiple `.mthds` files can declare the same domain. They all contribute to that domain's namespace:

```toml
# finance_invoices.mthds
domain = "finance"
[concept]
Invoice = "..."
```

```toml
# finance_payments.mthds
domain = "finance"
[concept]
Payment = "..."
```

Both files contribute to the `finance` domain, creating:
- `finance.Invoice`
- `finance.Payment`

## Using Domains

### In Bundle Files

**Same-domain references** (no prefix needed):

```toml
domain = "finance"

[concept]
Invoice = "A commercial document"

[pipe.process_invoice]
type = "PipeLLM"
inputs = { invoice = "Invoice" }    # Same domain, no prefix
output = "Text"
prompt = "Process this invoice: $invoice"
```

**Cross-domain references** (prefix required):

```toml
domain = "accounting"

[pipe.reconcile]
type = "PipeLLM"
inputs = { invoice = "finance.Invoice" }    # Different domain, needs prefix
output = "Report"
prompt = "Reconcile this invoice: $invoice"
```

### In Python Code

Always use the full identifier when referencing concepts in code:

```python
from pipelex.core.stuffs.stuff_factory import StuffFactory

invoice_stuff = StuffFactory.make_from_concept_ref(
    concept_ref="finance.Invoice",  # domain_code.ConceptName
    name="invoice_123",
    content=invoice_data
)
```

## System Prompt Inheritance

When you set a `system_prompt` at the domain level, it applies to all `PipeLLM` operators in bundles that declare that domain:

```toml
domain = "medical_records"
system_prompt = "You are a medical records specialist with expertise in HIPAA compliance."

[pipe.extract_diagnosis]
type = "PipeLLM"
# Automatically inherits the domain's system_prompt
inputs = { record = "MedicalRecord" }
output = "Diagnosis"
prompt = "Extract the primary diagnosis: $record"
```

Individual pipes can override the domain system prompt by defining their own `system_prompt` field.

## Related Documentation

- [Packages](./packages.md) - Controlling pipe visibility with exports
- [Pipelex Bundle Specification](./pipelex-bundle-specification.md) - How domains are declared in bundles
- [Kick off a Pipelex Method Project](./kick-off-a-methods-project.md) - Getting started
- [Define Your Concepts](./concepts/define_your_concepts.md) - Creating concepts within domains
- [Designing Pipelines](./pipes/index.md) - Building pipes within domains


## Define Your Concepts

# Defining Your Concepts

!!! tip "MTHDS Standard Reference"
    Concepts are part of the MTHDS open standard. For the authoritative language specification, see [Concepts](https://mthds.ai/latest/language/concepts/) on mthds.ai. This page documents Pipelex-specific behavior and usage.

Concepts are the foundation of reliable AI methods. They define what flows through your pipes—not just as data types, but as meaningful pieces of knowledge with clear boundaries and validation rules.

## Writing Concept Definitions

Every concept starts with a natural language definition. This definition serves two audiences: developers who build with your pipeline, and the LLMs that process your knowledge.

### Basic Concept Definition

The simplest way to define a concept is with a descriptive sentence:

```toml
[concept]
Invoice = "A commercial document issued by a seller to a buyer"
Employee = "A person employed by an organization"
ProductReview = "A customer's evaluation of a product or service"
```

!!! important "Concept Naming Convention"
    Concept names **MUST** be in `PascalCase` (also known as UpperCamelCase). Each word starts with a capital letter, with no underscores or hyphens.
    
    **Valid concept names:**
    ```toml
    ✅ Invoice
    ✅ ProductReview
    ✅ CustomerFeedback
    ✅ InvoiceLineItem
    ```
    
    **Invalid concept names:**
    ```toml
    ❌ invoice           # Not PascalCase
    ❌ product_review    # snake_case not allowed
    ❌ Product-Review    # Hyphens not allowed
    ❌ productReview     # camelCase not allowed (must start with capital)
    ```

Those concepts will be Text-based by default. If you want to use structured output, declare the structure directly in the concept definition (recommended) or create a Python class for advanced use cases.

**Key principles for concept definitions:**

1. **Define what it is, not what it's for**
   ```toml
   # ❌ Wrong: includes usage context
   TextToSummarize = "Text that needs to be summarized"
   
   # ✅ Right: defines the essence
   Article = "A written composition on a specific topic"
   ```

2. **Use singular forms**
   ```toml
   # ❌ Wrong: plural form
   Invoices = "Commercial documents from sellers"
   
   # ✅ Right: singular form
   Invoice = "A commercial document issued by a seller to a buyer"
   ```

3. **Avoid unnecessary adjectives**
   ```toml
   # ❌ Wrong: includes subjective qualifier
   LongArticle = "A lengthy written composition"
   
   # ✅ Right: neutral description
   Article = "A written composition on a specific topic"
   ```

### Organizing Related Concepts

Group concepts that naturally belong together in the same domain. A domain acts as a namespace for a set of related concepts and pipes, helping you organize and reuse your pipeline components. You can learn more about them in [Understanding Domains](../domain.md).

```toml
# finance.mthds
domain = "finance"
description = "Financial document processing"

[concept]
Invoice = "A commercial document issued by a seller to a buyer"
Receipt = "Proof of payment for goods or services"
PurchaseOrder = "A buyer's formal request to purchase goods or services"
PaymentTerms = "Conditions under which payment is to be made"
LineItem = "An individual item or service listed in a financial document"
```

## Get Started with Inline Structures

To add structure to your concepts, the **recommended approach** is using **inline structures** directly in your `.mthds` files. Inline structures support all field types including nested concepts:

```toml
[concept.Customer]
description = "A customer for an invoice"

[concept.Customer.structure]
name = { type = "text", description = "Customer's full name", required = true }
email = { type = "text", description = "Customer's email address", required = true }

[concept.Invoice]
description = "A commercial document issued by a seller to a buyer"

[concept.Invoice.structure]
invoice_number = "The unique invoice identifier"
issue_date = { type = "date", description = "The date the invoice was issued" }
customer = { type = "concept", concept_ref = "finance.Customer", description = "The customer" }
total_amount = { type = "number", description = "The total invoice amount" }
```

This automatically generates fully-typed Pydantic models with validation—no Python code needed!

**For complete details on inline structures, nested concepts, and all features, see [Inline Structures](inline-structures.md).**

### Generating Python Classes

If you need type hints and IDE autocomplete, use the `pipelex build structures` command to generate Python classes from your inline definitions:

```bash
pipelex build structures ./my_pipelines/
```

### Alternative: Hand-Written Python Classes

For advanced features like custom validation, computed properties, or reusable business logic, you can create explicit Python classes.

**Learn more in [Python StructuredContent Classes](python-classes.md).**

## Refining Concepts

You can create more specific versions of existing concepts through refinement. For example, an `Invoice` is a specific kind of `Document`:

```toml
[concept.Invoice]
description = "A commercial document issued by a seller to a buyer"
refines = "Document"
```

The refined concept inherits the structure of the base concept while adding semantic specificity.

**For complete details on refinement, see [Refining Concepts](refining-concepts.md).**

## Native Concepts

Pipelex includes these built-in native concepts: `Text`, `Image`, `Document`, `TextAndImages`, `Number`, `YesNo`, `Date`, `Time`, `Page`, `Dynamic`, `JSON`, `Html`, `SearchResult`, `Composite`, and `Anything`.

These concepts are automatically available in all pipelines—no setup required. You can use them directly in your pipes or refine them to create more specific concepts.

**For complete details on all native concepts and their structures, see [Native Concepts](native-concepts.md).**


## Inline structures

# Inline Structure Definition

Define structured concepts directly in your `.mthds` files using pipelex syntax. This is the **recommended approach** for most use cases, offering rapid development without Python boilerplate.

For an introduction to concepts themselves, see [Define Your Concepts](define_your_concepts.md). For advanced features requiring Python classes, see [Python StructuredContent Classes](python-classes.md).

## Quick Example

```toml
domain = "finance"

[concept.LineItem]
description = "A single line item in an invoice"

[concept.LineItem.structure]
product_name = { type = "text", description = "Name of the product", required = true }
quantity = { type = "integer", description = "Quantity ordered", required = true }
unit_price = { type = "number", description = "Price per unit", required = true }

[concept.Customer]
description = "A customer for an invoice"

[concept.Customer.structure]
name = { type = "text", description = "Customer's full name", required = true }
email = { type = "text", description = "Customer's email address", required = true }

[concept.Invoice]
description = "A commercial document issued by a seller to a buyer"

[concept.Invoice.structure]
invoice_number = "The unique invoice identifier"
issue_date = { type = "date", description = "The date the invoice was issued", required = true }
customer = { type = "concept", concept_ref = "finance.Customer", description = "The customer for this invoice", required = true }
line_items = { type = "list", item_type = "concept", item_concept_ref = "finance.LineItem", description = "List of line items", required = true }
total_amount = { type = "number", description = "The total invoice amount", required = true }
```

Behind the scenes, Pipelex automatically generates fully-typed Pydantic models with validation—all from TOML. Nested concepts are fully supported!

!!! tip "Generate Python Classes from Inline Structures"
    Want type hints and IDE autocomplete? Use the `pipelex build structures` command to generate Python classes from your inline definitions:

    ```bash
    pipelex build structures ./my_pipelines/
    ```

    This gives you the best of both worlds: rapid prototyping in TOML, then Python classes when you need them.

## Field Properties

Each field can specify:

- **type**: The data type (required for detailed definitions)
- **description**: Human-readable description
- **default_value**: Default value if not provided
- **choices**: For `enum`-like fields, a list of valid values
- **item_type**: For `list` fields, the type of list items
- **key_type** and **value_type**: For `dict` fields, the types of keys and values

Field names must be valid Python identifiers and cannot be Python keywords; generated Python projections use them as model attributes.

## Supported Field Types

Inline structures support these field types:

### text

String values. Use for any text content.

```toml
[concept.Person.structure]
email = { type = "text", description = "Email address" }
```

!!! tip "Simple Syntax Shorthand"
    When you write `field_name = "description"`, Pipelex automatically creates a **text** field with:
    
    - **type**: `text`
    - **description**: The string you provided
    - **required**: `true`
    
    This is a shorthand for the most common case: required text fields. To make a text field optional, use the explicit form instead, e.g. `email = { type = "text", description = "Email address" }`.

### integer

Whole numbers (no decimals). Use for counts, IDs, ages, etc.

```toml
[concept.Product.structure]
product_id = { type = "integer", description = "Unique product ID" }
quantity = { type = "integer", description = "Stock quantity", default_value = 0 }
```

### boolean

True/false values. Use for flags, states, toggles.

```toml
[concept.Account.structure]
is_active = { type = "boolean", description = "Account status" }
is_verified = { type = "boolean", description = "Email verified", default_value = false }
```

### number

Numeric values (integers or floats). Use for prices, measurements, scores.

```toml
[concept.Invoice.structure]
total_amount = { type = "number", description = "Total amount" }
tax_rate = { type = "number", description = "Tax rate as decimal", default_value = 0.0 }
```

### date

A calendar date, no time of day. Generates a `datetime.date` field (JSON schema `format: date`). Use it for issue dates, due dates, dates of birth — anything you'd write as `2026-07-07`.

```toml
[concept.Event.structure]
event_date = { type = "date", description = "The date of the event" }
```

### datetime

A timestamp — a date *and* a time of day. Generates a `datetime.datetime` field (JSON schema `format: date-time`). Use it only when the time genuinely matters; a plain calendar date should be `date`, so the LLM is never forced to fabricate a time.

```toml
[concept.Event.structure]
created_at = { type = "datetime", description = "Creation timestamp" }
```

### time

A time of day, optionally with a UTC offset — no date attached. Generates a `datetime.time` field (JSON schema `format: time`). Use it for opening hours, schedules, recurring times — anything you'd write as `09:00:00` or `15:40:00+02:00`.

```toml
[concept.OpeningHours.structure]
opens_at = { type = "time", description = "Daily opening time" }
```

!!! note "Field type vs the native `Date` / `Time` concepts"
    A structure field's `date` / `datetime` / `time` type is a plain machine type — you pick one up front. If a field should keep *document fidelity* (a time only when the source states one), model it as `type = "concept", concept_ref = "Date"` instead: the native `Date` concept carries a required date plus an optional `time` field.

### list

Arrays/lists of items. **Must specify `item_type`** to indicate what the list contains.

```toml
[concept.Project.structure]
tags = { type = "list", item_type = "text", description = "Project tags" }
milestones = { type = "list", item_type = "text", description = "Milestone names" }
scores = { type = "list", item_type = "number", description = "Test scores" }
```

### dict

Dictionaries/maps with key-value pairs. **Must specify `key_type` and `value_type`**. Dictionary keys are text on the JSON wire, so `key_type` must be `"text"` (`"str"` remains accepted as a legacy spelling).

```toml
[concept.Configuration.structure]
settings = { type = "dict", key_type = "text", value_type = "text", description = "Config settings" }
metadata = { type = "dict", key_type = "text", value_type = "number", description = "Numeric metadata" }
```

### concept

References to other custom concepts. Use for nested structures and relationships between concepts.

**Single concept reference:**

```toml
[concept.Order.structure]
customer = { type = "concept", concept_ref = "myapp.Customer", description = "The customer placing the order" }
```

**List of concepts:**

```toml
[concept.Order.structure]
items = { type = "list", item_type = "concept", item_concept_ref = "myapp.OrderItem", description = "Items in the order" }
```

!!! note "Concept Reference Format"
    Use the format `domain.ConceptCode` (e.g., `"finance.Invoice"`) or just `ConceptCode` if the concept is in the same domain.

## Nested Concepts

Inline structures fully support concept-to-concept references, allowing you to build complex data models entirely in TOML.

### Single Concept Reference

Use `type = "concept"` with `concept_ref` to reference another concept:

```toml
[concept.Address]
description = "A physical address"

[concept.Address.structure]
street = { type = "text", description = "Street address", required = true }
city = { type = "text", description = "City name", required = true }
country = { type = "text", description = "Country", required = true }

[concept.Company]
description = "A company with headquarters"

[concept.Company.structure]
name = { type = "text", description = "Company name", required = true }
headquarters = { type = "concept", concept_ref = "myapp.Address", description = "Company headquarters" }
```

### List of Concepts

Use `type = "list"` with `item_type = "concept"` and `item_concept_ref` for lists of concepts:

```toml
[concept.Employee]
description = "An employee"

[concept.Employee.structure]
name = { type = "text", description = "Employee name", required = true }
role = { type = "text", description = "Job role", required = true }

[concept.Team]
description = "A team of employees"

[concept.Team.structure]
team_name = { type = "text", description = "Team name", required = true }
members = { type = "list", item_type = "concept", item_concept_ref = "myapp.Employee", description = "Team members", required = true }
lead = { type = "concept", concept_ref = "myapp.Employee", description = "Team lead" }
```

### Dependency Ordering

Pipelex automatically handles concept dependencies. You can define concepts in any order—Pipelex loads them in the correct sequence based on their relationships.

## Choice Fields (Enums)

For fields that should only accept specific values, use the `choices` property. When using `choices`, you don't need to specify a `type`.

```toml
[concept.Task.structure]
title = "Task title"
priority = { choices = ["low", "medium", "high"], description = "Task priority level" }
status = { choices = ["todo", "in_progress", "done"], description = "Current status", default_value = "todo" }
```

## Required Fields

Fields declared with the explicit inline-table form are **optional by default** (`required = false`). To make one mandatory, explicitly set `required = true`. Note that fields declared with the shorthand string syntax (`field_name = "description"`) are **required**:

```toml
[concept.User.structure]
username = { type = "text", description = "Username", required = true }
email = { type = "text", description = "Email address", required = true }
bio = { type = "text", description = "User bio" }  # Optional (default)
```

In this example, `username` and `email` are mandatory, while `bio` is optional.

## Generating Python Classes

The `pipelex build structures` command generates Python classes from your inline definitions. This is useful when you want:

- Type hints and IDE autocomplete
- To import the generated classes in your Python code
- A starting point before adding custom validation

### Usage

```bash
# Generate from a directory of .mthds files
pipelex build structures ./my_pipelines/

# Generate from a specific .mthds file
pipelex build structures ./my_pipeline/bundle.mthds

# Specify output directory
pipelex build structures ./my_pipelines/ -o ./generated/
```

### Example Output

Given the Invoice example from above, Pipelex generates one stamped `structures.py` module (plus a `codegen.lock` for the offline drift check) containing every concept of the closure:

```python
# generated/structures.py (below the codegen stamp header)
class Invoice(StructuredContent):
    """A commercial document issued by a seller to a buyer"""

    invoice_number: str = Field(..., description="The unique invoice identifier")
    issue_date: date = Field(..., description="The date the invoice was issued")
    customer: Customer = Field(..., description="The customer for this invoice")
    line_items: List[LineItem] = Field(..., description="List of line items")
    total_amount: float = Field(..., description="The total invoice amount")
```

You can then import and use these classes with full type safety. Generated files are never edited in place — to customize a class, subclass it in a sibling module (the generated header shows how).

## When to Use Inline Structures

**Inline structures are the recommended approach for most use cases.** They support:

- ✅ All basic field types (text, integer, number, boolean, date, datetime, list, dict)
- ✅ Nested concepts and complex relationships
- ✅ Choice fields (enums)
- ✅ Required/optional fields with defaults
- ✅ Full runtime validation

Use `pipelex build structures` when you additionally need:

- ✅ Type hints and IDE autocomplete in your Python code
- ✅ To import the classes in your application

Use hand-written Python classes when you need:

- ✅ Custom validation logic (e.g., cross-field validation)
- ✅ Computed properties
- ✅ Custom methods and business logic

See [Python StructuredContent Classes](python-classes.md) for advanced features.

## Related Documentation

- [Define Your Concepts](define_your_concepts.md) - Learn about concept semantics and naming
- [Python StructuredContent Classes](python-classes.md) - Advanced features with Python
- [MTHDS Language Tutorial](../../get-started/mthds-language-tutorial.md) - Get started with structured outputs


## Python classes

# Python StructuredContent Classes

For advanced features beyond inline structures, create explicit Python classes that inherit from `StructuredContent`. This approach gives you full Pydantic power with custom validation, computed properties, and reusable business logic.

!!! tip "Start with Inline Structures"
    For most use cases, [Inline Structures](inline-structures.md) are the recommended approach. They support all field types including nested concepts, and you can generate Python classes from them using `pipelex build structures`.

    Use hand-written Python classes only when you need **custom validation logic**, **computed properties**, or **custom methods**.

## When to Use Python Classes

Most structured data needs are covered by inline structures + `pipelex build structures`. Use hand-written Python classes only when you need:

1. **Custom validation logic** - Cross-field validation, complex business rules
2. **Computed properties** - Derived values, formatted outputs
3. **Custom methods** - Business logic, helper functions
4. **Advanced Pydantic features** - `Literal`, complex unions, model validators

## Basic Python Class Example

```python
from pipelex.core.stuffs.structured_content import StructuredContent
from pydantic import Field

class Invoice(StructuredContent):
    """A commercial invoice."""
    
    invoice_number: str = Field(description="Unique invoice identifier")
    issue_date: datetime = Field(description="Date the invoice was issued")
    total_amount: float = Field(ge=0, description="Total invoice amount")
    vendor_name: str = Field(description="Name of the vendor")
    line_items: list[str] = Field(default_factory=list, description="List of items")
```

Classes inheriting from `StructuredContent` are automatically discovered and registered by Pipelex.

!!! warning "Module Execution During Auto-Discovery"
    When Pipelex discovers `StructuredContent` classes, it imports the module containing them. **Any code at the module level (outside functions/classes) will be executed during import.** This can have unintended side effects.
    
    **Best practice:** Keep your `StructuredContent` classes in dedicated modules (e.g., `*_struct.py` files) with minimal module-level code, or ensure module-level code is safe to execute during discovery.

## With Custom Validation

```python
from pipelex.core.stuffs.structured_content import StructuredContent
from pydantic import Field, field_validator

class Invoice(StructuredContent):
    """A commercial invoice with validation."""
    
    total_amount: float = Field(ge=0, description="Total invoice amount")
    tax_amount: float = Field(ge=0, description="Tax amount")
    net_amount: float = Field(ge=0, description="Net amount before tax")
    
    @field_validator('tax_amount')
    @classmethod
    def validate_tax(cls, v, info):
        """Ensure tax doesn't exceed total."""
        total = info.data.get('total_amount', 0)
        if v > total:
            raise ValueError('Tax amount cannot exceed total amount')
        return v
```

## With Computed Properties

```python
from datetime import datetime
from pipelex.core.stuffs.structured_content import StructuredContent
from pydantic import Field

class Subscription(StructuredContent):
    """A subscription with computed properties."""
    
    start_date: datetime = Field(description="Subscription start date")
    end_date: datetime = Field(description="Subscription end date")
    monthly_price: float = Field(ge=0, description="Monthly subscription price")
    
    @property
    def duration_days(self) -> int:
        """Calculate subscription duration in days."""
        return (self.end_date - self.start_date).days
    
    @property
    def total_cost(self) -> float:
        """Calculate total subscription cost."""
        months = self.duration_days / 30.0
        return months * self.monthly_price
```

## Generating Python Classes Automatically

The easiest way to get Python classes is to use `pipelex build structures`:

```bash
pipelex build structures ./my_pipelines/
```

This generates a single stamped `structures.py` module from your inline definitions, giving you type hints and IDE autocomplete without writing boilerplate. Generated files are never edited in place — they are overwritten on every regeneration, and `pipelex codegen check` flags hand edits as drift.

### When You Need Custom Logic

If you need custom validation or computed properties, you have two options:

1. **Generate then subclass**: Run `pipelex build structures`, then subclass the generated class in a sibling module and add your custom logic there — never copy or edit the generated file; a subclass survives regeneration and stays in sync with the source concept
2. **Write from scratch**: Create a Python class manually with your custom logic

### Example: Adding Custom Validation

**Step 1: Start with inline structure**

```toml
[concept.UserProfile]
description = "A user profile"

[concept.UserProfile.structure]
username = "The user's username"
email = "The user's email address"
age = { type = "integer", description = "User's age", required = false }
```

**Step 2: Generate the base class**

```bash
pipelex build structures ./my_pipeline.mthds -o ./structures/
```

**Step 3: Subclass the generated class in a sibling module**

Never copy or edit the generated file — it is overwritten on regeneration. A subclass in a sibling module survives regeneration and inherits all the generated fields, so you only write the custom logic:

```python
# structures/user_profile_ext.py
import re

from pydantic import field_validator

from .structures import UserProfile


class ValidatedUserProfile(UserProfile):
    """A user profile with email validation."""

    @field_validator('email')
    @classmethod
    def validate_email(cls, v):
        """Validate email format."""
        pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
        if not re.match(pattern, v):
            raise ValueError('Invalid email format')
        return v
```

**Step 4: Keep the inline structure in your `.mthds` file**

The inline structure remains the source of truth: on the next regeneration the base class is rewritten from it and your subclass picks up the changes automatically. At pipeline time the concept still resolves to the structure declared inline — apply your custom logic by validating with the subclass wherever your own code consumes the result (e.g. `ValidatedUserProfile.model_validate(profile.model_dump())`). If Pipelex itself must enforce the validation when producing the concept, move the structure to a hand-written class instead (see the [Migration Guide](#migration-guide) below).

## Migration Guide

### From Inline Structure to Python Class

Before migrating, check whether you need to migrate at all: if you only want custom logic on top of the structure, subclass the generated class in a sibling module (see above) and keep the inline structure as the source of truth. Migrate only when Pipelex itself must enforce your custom validation when producing the concept. In that case, write the Python class by hand — it becomes the source of truth:

**1. You have this inline structure:**

```toml
domain = "ecommerce"

[concept.Product]
description = "A product in the catalog"

[concept.Product.structure]
product_id = { type = "integer", description = "Unique product ID", required = true }
name = "Product name"
price = { type = "number", description = "Product price", required = true }
in_stock = { type = "boolean", description = "Stock availability", default_value = true }
```

**2. Write the Python class in your own module** with your custom logic:

```python
from pipelex.core.stuffs.structured_content import StructuredContent
from pydantic import Field, field_validator

class Product(StructuredContent):
    """A product in the catalog."""

    product_id: int = Field(description="Unique product ID")
    name: str = Field(description="Product name")
    price: float = Field(ge=0, description="Product price")
    in_stock: bool = Field(default=True, description="Stock availability")

    @field_validator('price')
    @classmethod
    def validate_price(cls, v):
        """Ensure price is positive and reasonable."""
        if v < 0:
            raise ValueError('Price cannot be negative')
        if v > 1_000_000:
            raise ValueError('Price seems unreasonably high')
        return v

    @property
    def display_price(self) -> str:
        """Format price for display."""
        return f"${self.price:.2f}"
```

**3. Update your `.mthds` file:**

```toml
domain = "ecommerce"

[concept]
Product = "A product in the catalog"

# Structure section removed - now defined in Python
```

**4. Test your pipeline** - The behavior should be identical, plus your custom validation.

## Recommendations

### The Recommended Workflow

1. **Start with inline structures** - Define your concepts entirely in TOML
2. **Generate Python classes when needed** - Use `pipelex build structures` for type hints and IDE support
3. **Add custom logic only when necessary** - Custom validation, computed properties, business methods

### When to Use Each Approach

| Need | Solution |
|------|----------|
| Rapid prototyping | Inline structures |
| Simple structured data | Inline structures |
| Nested concepts | Inline structures |
| Type hints in Python | `pipelex build structures` |
| IDE autocomplete | `pipelex build structures` |
| Custom validation | Hand-written Python class |
| Computed properties | Hand-written Python class |
| Business methods | Hand-written Python class |

## Related Documentation

- [Inline Structures](inline-structures.md) - Fast prototyping with TOML
- [Define Your Concepts](define_your_concepts.md) - Learn about concept semantics and naming
- [MTHDS Language Tutorial](../../get-started/mthds-language-tutorial.md) - Get started with structured outputs



## Native Concepts

# Native Concepts

Pipelex includes several built-in native concepts that cover common data types in AI methods. These concepts come with predefined structures and are automatically available in all pipelines—no setup required.

For an introduction to concepts, see [Define Your Concepts](define_your_concepts.md).

## What Are Native Concepts?

Native concepts are ready-to-use building blocks for AI methods. They represent common data types you'll frequently work with: text, images, documents, numbers, and combinations thereof.

**Key characteristics:**

- **Pre-defined**: Built into Pipelex, no need to declare them
- **Structured**: Each has a corresponding Python data model
- **Universal**: Available across all pipelines and domains
- **Extensible**: You can refine them to create more specific concepts

## Available Native Concepts

Here are all the native concepts you can use out of the box:

| Concept | Description | Content Class Name |
|-----------------|-------------|---------------------|
| `Text` | A text | `TextContent` |
| `Image` | An image | `ImageContent` |
| `Document` | A document (PDF, DOCX, PPTX, web page) | `DocumentContent` |
| `TextAndImages` | Text with its associated images | `TextAndImagesContent` |
| `Number` | A number | `NumberContent` |
| `YesNo` | The answer to a yes/no question | `YesNoContent` |
| `Date` | A calendar date, optionally with a time of day | `DateContent` |
| `Time` | A time of day, optionally with a UTC offset | `TimeContent` |
| `Page` | A document page with text, images, and optional page view | `PageContent` |
| `Dynamic` | A dynamic concept that adapts to context | `DynamicContent` |
| `JSON` | A JSON object | `JSONContent` |
| `Html` | HTML content with inner HTML and CSS class | `HtmlContent` |
| `SearchResult` | A web search result with answer and sources | `SearchResultContent` |
| `Composite` | A named composition of contents | `CompositeContent` |
| `Anything` | Any type of content | *No specific implementation* |

## Native Concept Structures

Each native concept has a corresponding Python structure that defines its data model. Understanding these structures helps you work with the data they contain.

### TextContent

The simplest native concept:

```python
class TextContent(StuffContent):
    text: str
```

**Use for:** Plain text outputs, summaries, descriptions, etc.

### ImageContent

Represents an image with optional metadata:

```python
class ImageContent(StuffContent):
    url: str
    public_url: str | None = None
    source_prompt: str | None = None
    source_negative_prompt: str | None = None
    caption: str | None = None
    mime_type: str | None = None
    width: int | None = None
    height: int | None = None
    filename: str | None = None
```

**Fields:**

- `url`: Location of the image (a storage URI, an HTTP(S) URL, or a base64 data URL)
- `public_url`: Optional public-facing URL (when `url` is a private/internal reference)
- `source_prompt` / `source_negative_prompt`: The prompts used to generate the image (if applicable)
- `caption`: Descriptive text for the image
- `mime_type`: Optional MIME type of the image
- `width` / `height`: Pixel dimensions — present together or not at all
- `filename`: Optional original filename

**Use for:** Photos, generated images, diagrams, screenshots.

### DocumentContent

Represents a document (PDF, DOCX, PPTX, etc.):

```python
class DocumentContent(StuffContent):
    url: str
    public_url: str | None = None
    mime_type: str | None = None
    filename: str | None = None
    title: str | None = None
    snippet: str | None = None
```

**Fields:**

- `url`: Location of the document file, storage URL, or web page URL
- `public_url`: Optional public-facing URL (when `url` is a private/internal reference)
- `mime_type`: Optional MIME type of the document (e.g., "application/pdf", "application/vnd.openxmlformats-officedocument.wordprocessingml.document")
- `filename`: Optional filename of the document
- `title`: Optional title of the document or source
- `snippet`: Optional text snippet or excerpt from the document

**Use for:** Contracts, invoices, reports, presentations, web pages, search source citations, any document file.

### NumberContent

Represents numeric values:

```python
class NumberContent(StuffContent):
    number: Union[int, float]
```

**Use for:** Counts, calculations, metrics, scores.

### YesNoContent

Represents the answer to a yes/no question — a single boolean verdict:

```python
class YesNoContent(StuffContent):
    yes_no: bool
```

Renders as `yes` or `no` when injected into a prompt. Especially handy as a `PipeLLM` output for judgments — `output = "YesNo"` makes the model return a typed boolean instead of free text answering "yes"/"no":

```toml
[pipe.judge_is_urgent]
type = "PipeLLM"
description = "Decide whether a message is urgent"
inputs = { message = "Text" }
output = "YesNo"
prompt = "Is the following message urgent? Answer yes or no.\n\n$message"
```

Read the verdict from a Python caller via `pipe_output.main_stuff_as_yes_no.yes_no`.

**Use for:** Yes/no judgments, boolean classifications, presence/absence checks, pass/fail verdicts.

### DateContent

Represents a calendar date, as precise as its source states — a required `date` plus an optional time of day:

```python
class DateContent(StuffContent):
    date: datetime.date
    time: datetime.time | None = None
```

The time is present only when the source states one — the concept never invents a time, so an LLM extracting "delivery by March 15" produces a date with no time, while a ticket's "7 Jul 2026, 15:40" keeps the time (and its UTC offset, when stated). It renders as ISO 8601 truncated to the stated precision (`2026-07-07`, or `2026-07-07T15:40:00+02:00`) when injected into a prompt. As a `PipeLLM` output, `output = "Date"` makes the model return the two-field structure:

```toml
[pipe.extract_departure]
type = "PipeLLM"
description = "Extract the scheduled departure from a ticket"
inputs = { ticket = "Ticket" }
output = "Date"
prompt = "Extract the scheduled departure from this ticket: $ticket"
```

As a pipeline input, a top-level TOML date or datetime literal maps to `Date` directly (`departure = 2026-07-07T15:40:00+02:00`); read it from a Python caller via `pipe_output.main_stuff_as_date`.

### TimeContent

Represents a time of day, as precise as its source states — never attached to an invented date:

```python
class TimeContent(StuffContent):
    time: datetime.time
```

The UTC offset is kept on the `time` when the source states one, exactly as `Date` handles its optional time. It renders as ISO 8601 (`15:40:00`, or `15:40:00+02:00`) when injected into a prompt. As a pipeline input, a top-level TOML time-of-day literal maps to `Time` directly (`opening = 09:00:00`).

**Use for:** Opening hours, schedules, times of day with no specific date.

**Use for:** Issue dates, due dates, dates of birth, departures, effective/termination dates — any date found on a document. For a date value without any date attached (a bare time of day), degrade to `Text`.

### TextAndImagesContent

Combines text with one or more images:

```python
class TextAndImagesContent(StuffContent):
    text: Optional[TextContent]
    images: Optional[List[ImageContent]]
    raw_html: Optional[str]
```

**Fields:**

- `text`: The text content
- `images`: A list of images extracted from the content
- `raw_html`: The raw HTML of the fetched page, when requested via `include_raw_html`

**Use for:** Rich content combining text and visuals, extracted document content, reports with diagrams.

### PageContent

Represents a document page with both content and visual representation:

```python
class PageContent(StructuredContent):
    text_and_images: TextAndImagesContent
    page_view: Optional[ImageContent] = None
```

**Fields:**
- `text_and_images`: The extracted text and embedded images from the page
- `page_view`: A screenshot or rendering of the entire page

**Use for:** Document pages extracted by `PipeExtract`, individual pages from multi-page documents.

### DynamicContent

A flexible concept that adapts to context:

```python
class DynamicContent(StuffContent):
    # Dynamic content that can adapt to context
    # Structure is flexible and determined at runtime
    pass
```

**Use for:** Methods where the content structure isn't known in advance.

### JSONContent

A concept that represents a JSON object. This enables pipes to receive as input or to output any JSON object.

```python
class JSONContent(StuffContent):
    json_obj: dict[str, Any]
```

### HtmlContent

Represents HTML content with styling:

```python
class HtmlContent(StuffContent):
    inner_html: str
    css_class: str
```

**Fields:**

- `inner_html`: The inner HTML of the content
- `css_class`: The CSS class applied to the wrapping element

**Use for:** Rendered HTML fragments, styled content blocks, HTML-based reports.

### SearchResultContent

Represents the result of a web search query. Produced by `PipeSearch`:

```python
class SearchResultContent(StuffContent):
    answer: str
    sources: list[DocumentContent] = Field(default_factory=empty_list_factory_of(DocumentContent))
```

**Fields:**

- `answer`: The synthesized answer text from the search
- `sources`: A list of `DocumentContent` source citations (each with `url`, `title`, and `snippet`)

**Use for:** Web search results, research findings, information retrieval with citations.

### CompositeContent

An untyped named composition of contents. Produced by `PipeParallel` when its output is `Composite`:

```python
class CompositeContent(StuffContent):
    model_config = ConfigDict(extra="allow")
```

Each named sub-content is a top-level field of the composite — there is no wrapper key, so the serialized shape matches what a bespoke structured concept with the same field names would produce.

**Use for:** Combining parallel branch results without declaring a bespoke concept.

### Anything

!!! note "Special Concepts"
    `Anything` is referenced in the native concept definitions but does not have specific implementations. It is handled through the generic content system and is primarily used as semantic markers.

## Using Native Concepts

Native concepts can be used directly in your pipeline definitions without any additional setup:

### In Pipe Inputs

```toml
[pipe.analyze_document]
type = "PipeLLM"
description = "Analyze a document"
inputs = { document = "Document" }
output = "Text"
prompt = """
Analyze this document and provide a summary:

@document
"""
```

### In Pipe Outputs

```toml
[pipe.process_image]
type = "PipeLLM"
description = "Describe an image"
inputs = { photo = "Image" }
output = "Text"
prompt = "Describe what you see in this image: $photo"
```

### With Page Content

The `Page` concept is particularly useful with `PipeExtract`:

```toml
[pipe.extract_pages]
type = "PipeExtract"
description = "Extract content from a document"
inputs = { document = "Document" }
output = "Page[]"
```

This extracts each page with both its text/images and a visual representation.

### In Complex Methods

```toml
[pipe.create_report]
type = "PipeSequence"
description = "Generate a report with text and images"
inputs = { data = "Text" }
output = "TextAndImages"
steps = [
    { pipe = "analyze_data", result = "analysis" },
    { pipe = "create_charts", result = "charts" },
    { pipe = "combine_content", result = "report" }
]
```

## Refining Native Concepts

You can create more specific concepts by refining native ones—for example, creating an `Invoice` concept that refines `Document` or a `ProductPhoto` that refines `Image`. This gives you semantic clarity while inheriting the native concept's structure.

**For complete details on refinement syntax, type compatibility, limitations, best practices, and future features, see [Refining Concepts](refining-concepts.md).**

## When to Use Native Concepts

Use native concepts directly when:

- ✅ Working with simple, unstructured data
- ✅ The native structure is sufficient for your needs
- ✅ You want maximum interoperability across pipes
- ✅ Prototyping and quick experiments

Refine native concepts when:

- ✅ You need semantic specificity (e.g., `Invoice` vs `Document`)
- ✅ You want to add custom structure on top of the base structure
- ✅ Building domain-specific methods
- ✅ Need type safety for specific document types

## Common Patterns

### Text Processing

```toml
[pipe.summarize]
type = "PipeLLM"
description = "Summarize any text"
inputs = { content = "Text" }
output = "Text"
prompt = """
Summarize this content:

@content
"""
```

### Document Extraction

```toml
[pipe.extract_pages]
type = "PipeExtract"
description = "Extract content from a document"
inputs = { document = "Document" }
output = "Page[]"

[pipe.analyze_pages]
type = "PipeLLM"
description = "Analyze pages"
inputs = { pages = "Page[]" }
output = "Text"
prompt = """Analyze those pages: 
@pages
"""

[pipe.extract_and_analyze]
type = "PipeSequence"
description = "Extract and analyze a document"
inputs = { document = "Document" }
output = "Text"
steps = [
    { pipe = "extract_pages", result = "pages" },
    { pipe = "analyze_pages", result = "analysis" }
]
```

### Multi-Modal Processing

```toml
[pipe.analyze_with_context]
type = "PipeLLM"
description = "Analyze image with text context"
inputs = { image = "Image", context = "Text" }
output = "Text"
prompt = """
Given this context: $context

Analyze this image: $image
"""
```

### Web Search

```toml
[pipe.search_topic]
type = "PipeSearch"
description = "Search the web for information"
inputs = { topic = "Text" }
output = "SearchResult"
model = "$standard"
prompt = "What is $topic?"
```

## Related Documentation

- [Define Your Concepts](define_your_concepts.md) - Learn about concept semantics
- [Inline Structures](inline-structures.md) - Add structure to refined concepts
- [Python StructuredContent Classes](python-classes.md) - Advanced customization
- [MTHDS Language Tutorial](../../get-started/mthds-language-tutorial.md) - Use native concepts in pipelines



## Refining Concepts

# Refining Concepts

Concept refinement allows you to create more specific versions of existing concepts while inheriting their structure. This provides semantic clarity and type safety for domain-specific methods.

## What is Concept Refinement?

Refinement is the process of creating a specialized concept from a more general one. When you refine a concept, the new concept:

- **Inherits the structure** of the base concept
- **Adds semantic specificity** to clarify its purpose

Think of it as creating a subtype: an `Invoice` is a specific kind of `Document`, and a `Photo` is a specific kind of `Image`.

## Why Refine Concepts?

### 1. Semantic Clarity

Refined concepts make your pipeline's intent explicit:

```toml
# ❌ Less clear
[pipe.process_invoice]
inputs = { invoice = "Document" }

# ✅ More clear
[pipe.process_invoice]
inputs = { invoice = "Invoice" }
```

### 2. Self-Documenting Code

```toml
[pipe.extract_contract_terms]
type = "PipeLLM"
description = "Extract key terms from a contract"
inputs = { contract = "Contract" }  # Clear what type of document is expected
output = "ContractTerms"
```

### 3. Domain-Specific Methods

Build pipelines tailored to specific use cases:

```toml
domain = "finance"

[concept.Invoice]
description = "A commercial invoice"
refines = "Document"

[concept.Receipt]
description = "Proof of payment"
refines = "Document"

[pipe.process_invoice]
type = "PipeLLM"
inputs = { invoice = "Invoice" }
output = "InvoiceData"
# ... invoice-specific processing

[pipe.process_receipt]
type = "PipeLLM"
inputs = { receipt = "Receipt" }
output = "ReceiptData"
# ... receipt-specific processing
```

### 4. Type Validation

Using specific concept names helps catch errors early:

```toml
[pipe.analyze_invoice]
inputs = { invoice = "Invoice" }  # Only accepts Invoice
output = "Analysis"
```

## Current Limitations

!!! warning "No structure on refined concepts (For Now)"
    When you refine a concept, you **cannot** also define a `structure` — neither by naming a structure class nor with an inline `[concept.X.structure]` table. The validator rejects the combination: a concept cannot have both `refines` and `structure`. This limitation will be lifted in future releases.
    
    **Not allowed:**
```toml
[concept.Invoice]
description = "A commercial invoice"
refines = "Document"
structure = "InvoiceModel"  # ❌ Not allowed: refines and structure are mutually exclusive
```

```toml
[concept.Invoice]
description = "A commercial invoice"
refines = "Document"

[concept.Invoice.structure]  # ❌ Not allowed: an inline table is the same structure field
invoice_number = "Invoice ID"
```

**Allowed:**
```toml
[concept.Invoice]
description = "A commercial invoice"
refines = "Document"  # ✅ Inherits DocumentContent structure
```

## Basic Refinement Syntax

Define a refined concept using the `refines` field:

```toml
[concept.ConceptName]
description = "Description of the refined concept"
refines = "BaseConceptName"
```

For concepts in a different domain, use the fully qualified reference:

```toml
[concept.SpecializedConcept]
description = "A more specialized version of an existing concept"
refines = "otherdomain.BaseConceptName"
```

### Refining Document

```toml
[concept.Invoice]
description = "A commercial document issued by a seller to a buyer"
refines = "Document"

[concept.Contract]
description = "A legally binding agreement between parties"
refines = "Document"
```

Both concepts inherit the `DocumentContent` structure (with a `url` field) but represent semantically distinct document types.

### Refining Image

```toml
[concept.ProductPhoto]
description = "A photograph of a product for marketing purposes"
refines = "Image"

[concept.Screenshot]
description = "A screen capture image"
refines = "Image"
```

Each inherits `ImageContent` structure (url, caption, mime_type, etc.) with specific semantic meaning.

### Refining Text

```toml
[concept.Article]
description = "A written composition on a specific topic"
refines = "Text"

[concept.Summary]
description = "A condensed version of a longer text"
refines = "Text"
```

### Building Concept Hierarchies

You can build hierarchies by refining concepts that have their own structures. The refined concept inherits the structure from the base concept:

```toml
domain = "crm"

# Base concept with structure
[concept.Customer]
description = "A customer in our system"

[concept.Customer.structure]
name = { type = "text", required = true, description = "Customer name" }
email = { type = "text", required = true, description = "Customer email" }

# Refined concept - inherits Customer's structure
[concept.VIPCustomer]
description = "A VIP customer with special privileges"
refines = "Customer"

# Another refined concept from the same base
[concept.InactiveCustomer]
description = "A customer who has not been active recently"
refines = "Customer"
```

Both `VIPCustomer` and `InactiveCustomer` will have access to the `name` and `email` fields defined in `Customer`. When you create content for these concepts, it will be compatible with the base `Customer` structure.

## Cross-Package Refinement

You can refine concepts that live in a different package. This lets you specialize a shared concept from a dependency without modifying the dependency itself.

### Syntax

Use the `->` cross-package reference operator in the `refines` field:

```toml
[concept.RefinedConcept]
description = "A more specialized version of a cross-package concept"
refines = "alias->domain.BaseConceptCode"
```

| Part | Description |
|------|-------------|
| `alias` | The dependency alias declared in your `METHODS.toml` `[dependencies]` section |
| `->` | Cross-package reference operator |
| `domain` | The dot-separated domain path inside the dependency package |
| `BaseConceptCode` | The `PascalCase` concept code to refine |

### Full Example

Suppose you depend on a scoring library that defines a `WeightedScore` concept:

**Dependency package** (`scoring-lib`):

```toml title="METHODS.toml"
[package]
address = "github.com/acme/scoring-lib"
version = "2.0.0"
description = "Scoring utilities."

[exports.scoring]
pipes = ["compute_weighted_score"]
```

```toml title="scoring.mthds"
domain = "scoring"

[concept.WeightedScore]
description = "A weighted score result"

[pipe.compute_weighted_score]
type = "PipeLLM"
description = "Compute a weighted score"
inputs = { item = "Text" }
output = "WeightedScore"
prompt = "Compute a weighted score for: {{ item }}"
```

**Your consumer package**:

```toml title="METHODS.toml"
[package]
address = "github.com/acme/analysis-app"
version = "1.0.0"
description = "Analysis application."

[dependencies]
scoring_lib = { address = "github.com/acme/scoring-lib", version = "^2.0.0" }

[exports.analysis]
pipes = ["compute_detailed_score"]
```

```toml title="analysis.mthds"
domain = "analysis"

[concept.DetailedScore]
description = "An extended score with additional detail"
refines = "scoring_lib->scoring.WeightedScore"

[pipe.compute_detailed_score]
type = "PipeLLM"
description = "Compute a detailed score"
inputs = { item = "Text" }
output = "DetailedScore"
prompt = "Compute a detailed score for: {{ item }}"
```

`DetailedScore` inherits the structure of `WeightedScore` from the `scoring_lib` dependency's `scoring` domain.

!!! important
    The base concept must be accessible from the dependency. The dependency must export the pipes in the domain that contains the concept, or the concept's domain must be reachable via an exported pipe's bundle.

For more on how dependencies and cross-package references work, see [Packages](../packages.md#cross-package-references).

## Type Compatibility

Understanding how refined concepts interact with pipe inputs is crucial.

### How Refinement Affects Type Checking

!!! important "Key Rule"
    A pipe that accepts a **base concept** will also accept any concept that **refines** it. Refined concepts are substitutable for their parent — this is standard subtype compatibility.

```toml
[pipe.extract_text]
inputs = { document = "Document" }  # Accepts Document, Invoice, Contract, or any concept that refines Document
```

### Practical Example

```toml
[concept.Invoice]
refines = "Document"

[concept.Contract]
refines = "Document"

# This pipe accepts Document and any concept that refines it (Invoice, Contract, etc.)
[pipe.extract_from_document]
type = "PipeExtract"
inputs = { document = "Document" }
output = "Page[]"

# This pipe accepts Invoice (and any concept that refines Invoice)
[pipe.process_invoice]
type = "PipeLLM"
inputs = { invoice = "Invoice" }
output = "InvoiceData"

# This pipe accepts Contract (and any concept that refines Contract)
[pipe.process_contract]
type = "PipeLLM"
inputs = { contract = "Contract" }
output = "ContractData"
```

In this setup:

- `extract_from_document` accepts `Document`, `Invoice`, `Contract`, or any other concept that refines `Document`
- `process_invoice` accepts `Invoice` or any concept that refines `Invoice`
- `process_contract` accepts `Contract` or any concept that refines `Contract`

## Best Practices

### 1. Choose Meaningful Names

```toml
# ❌ Avoid generic or vague names
[concept.Document1]
refines = "Document"

# ✅ Use specific, descriptive names
[concept.Invoice]
refines = "Document"
```

### 2. Write Clear Descriptions

```toml
# ❌ Too vague
[concept.Invoice]
description = "A document"
refines = "Document"

# ✅ Clear and specific
[concept.Invoice]
description = "A commercial document issued by a seller to a buyer, detailing products or services provided and payment terms"
refines = "Document"
```

### 3. Don't Over-Refine

```toml
# ❌ Too specific, creates unnecessary complexity
[concept.SmallInvoice]
description = "An invoice under $100"
refines = "Document"

[concept.LargeInvoice]
description = "An invoice over $1000"
refines = "Document"

# ✅ Keep it general, handle specifics in processing logic
[concept.Invoice]
description = "A commercial invoice"
refines = "Document"
```

## When to Refine vs. When to Create New Concepts

### Refine When:

- ✅ Your concept is semantically a specific type of an existing concept
- ✅ The base concept's structure is sufficient for your needs
- ✅ You want to inherit existing validation and behavior
- ✅ You're building domain-specific methods with clear document/content types
- ✅ You need to create specialized versions of an existing concept

**Examples:**
```toml
[concept.Invoice]  # Clearly a type of Document
refines = "Document"

[concept.VIPCustomer]  # A specialized type of Customer
refines = "Customer"
```

### Create New Concept When:

- ✅ Your concept needs custom structure with multiple fields
- ✅ Your concept doesn't naturally fit any existing concept
- ✅ You need complex validation or computed properties


## Related Documentation

- [Define Your Concepts](define_your_concepts.md) - Introduction to concepts
- [Native Concepts](native-concepts.md) - Complete guide to native concepts
- [Inline Structures](inline-structures.md) - Add structure to concepts
- [Python StructuredContent Classes](python-classes.md) - Advanced customization
- [Packages](../packages.md) - Package system, dependencies, and cross-package references



## Overview

# Designing Pipelines

In Pipelex, a pipeline is not just a rigid sequence of steps; it's a dynamic and intelligent method built by composing individual, reusable components called **pipes**. This approach allows you to break down complex AI tasks into manageable, testable, and reliable units.

This guide provides an overview of how to design your pipelines.

## The Building Blocks: Pipes

A pipeline is composed of pipes. There are two fundamental types of pipes you will use to build your methods:

*   **[Pipe Operators](./pipe-operators/index.md)**: These are the "workers" of your pipeline. They perform concrete actions like calling an LLM (`PipeLLM`), extracting text from a document (`PipeExtract`), searching the web (`PipeSearch`), or running a Python function (`PipeFunc`). Each operator is a specialized tool designed for a specific task.
*   **[Pipe Controllers](./pipe-controllers/index.md)**: These are the "managers" of your pipeline. They don't perform tasks themselves but orchestrate the execution flow of other pipes. They define the logic of your method, such as running pipes in sequence (`PipeSequence`), in parallel (`PipeParallel`), or based on a condition (`PipeCondition`).

A third, design-time-only kind of pipe — **[Signature Pipes](./signature-pipes.md)** — lets you stub a pipe's contract (inputs and output) by declaring it with no `type` and no implementation: a typeless, contract-only pipe *is* a signature. Signatures dry-run as mocks under `--allow-signatures`, so you can sketch a pipeline top-down before writing each operator. Strict validation refuses pipelines that still depend on a signature.

## Designing a Pipeline: Composition in MTHDS

The most common way to design a pipeline is by defining and composing pipes in a `.mthds` configuration file. This provides a clear, declarative way to see the structure of your method.

Each pipe, whether it's an operator or a controller, is defined in its own `[pipe.<pipe_code>]` table. The `<pipe_code>` becomes the unique identifier for that pipe.

!!! important "Pipe Code Naming Convention"
    Pipe codes **MUST** be in `snake_case` (lowercase with underscores). Use descriptive names that clearly indicate what the pipe does.
    
    **Valid pipe codes:**
    ```toml
    ✅ [pipe.generate_tagline]
    ✅ [pipe.extract_invoice]
    ✅ [pipe.validate_and_process]
    ✅ [pipe.send_email]
    ```
    
    **Invalid pipe codes:**
    ```toml
    ❌ [pipe.GenerateTagline]     # PascalCase not allowed
    ❌ [pipe.generateTagline]      # camelCase not allowed
    ❌ [pipe.generate-tagline]     # Hyphens not allowed
    ❌ [pipe.GENERATE_TAGLINE]     # All caps not allowed
    ```

Let's look at a simple example. Imagine we want a method that:
1.  Takes a product description.
2.  Generates a short, catchy marketing tagline for it.

We can achieve this with a `PipeLLM` operator.

`marketing_pipeline.mthds`
```toml
domain = "marketing"
description = "Marketing content generation domain"

# 1. Define the concepts used in our pipes
[concept]
ProductDescription = "A description of a product's features and benefits"
Tagline = "A catchy marketing tagline"

# 2. Define the pipe that does the work
[pipe.generate_tagline]
type = "PipeLLM"
description = "Generate a catchy tagline for a product"
inputs = { description = "ProductDescription" }
output = "Tagline"
prompt = """
Product Description:
@description

Generate a catchy tagline based on the above description.
The tagline should be memorable, concise, and highlight the key benefit.
"""
```

This defines a single-step pipeline. The pipe `generate_tagline` takes a `ProductDescription` as input and outputs a `Tagline` (both are custom concepts defined by a simple description, so they are Text-based by default — see more about native concepts [here](../concepts/native-concepts.md)).

The inputs specified will be required before the pipe is executed. Those inputs should be stored in the [Working Memory](working-memory.md).

The output concept is very important. Indeed, the output of your pipe will be corresponding to the concept you specify. If the concept is structured, the output will be a structured object. If the concept is Text-based — like `Tagline` here — the output is text. Other native concepts produce their corresponding content type (an `Image` output is an image, a `Number` output is a number, and so on).

### Understanding the Pipe Contract

Every pipe defines a **contract** through its `inputs` and `output` fields. This contract is fundamental to how Pipelex ensures reliability in your methods:

*   **`inputs`**: This dictionary defines the **mandatory and necessary** data that must be present in the [Working Memory](working-memory.md) before the pipe can execute. Each key in the dictionary becomes a variable name that you can reference in your pipe's logic (e.g., in prompts), and each value specifies the concept type that the data must conform to. If any required input is missing or doesn't match the expected concept, the pipeline will fail a clear error message.
You can specify multiple inputs by using a list of concepts. For example, `inputs = { description = "ProductDescription", keywords = "Keyword[]" }` will require a `ProductDescription` and a list of `Keyword`s. (See more about [Understanding Multiplicity](./understanding-multiplicity.md) for details.)

*   **`output`**: This field declares what the pipe will produce. The output will always be an instance of the specified concept. The structure and type of the output depend on the concept definition (See more about concepts [here](../concepts/native-concepts.md)).
    *   You can specify **multiple outputs** using bracket notation (e.g., `Keyword[]` for a variable list, or `Image[3]` for exactly 3 images)

### Multi-Step Methods

To create a multi-step method, you use a controller. The `PipeSequence` controller is the most common one. It executes a series of pipes in a specific order.


`marketing_pipeline.mthds`
```toml
domain = "marketing"
description = "Marketing content generation domain"

# 1. Define the concepts used in our pipes
[concept]
ProductDescription = "A description of a product's features and benefits"
Tagline = "A catchy marketing tagline"
Keyword = "A marketing keyword extracted from text"

# 2. Define the first step of your pipe
[pipe.generate_tagline]
type = "PipeLLM"
description = "Generate a catchy tagline for a product"
inputs = { description = "ProductDescription" }
output = "Tagline"
prompt = """
Product Description:
@description

Generate a catchy tagline based on the above description.
The tagline should be memorable, concise, and highlight the key benefit.
"""

# 3. Define the second step of your pipe
[pipe.extract_keywords_from_tagline]
type = "PipeLLM"
description = "Extract keywords from a tagline"
inputs = { tagline = "Tagline" }
output = "Keyword[]"
prompt = """
Extract the most relevant keywords from the following tagline. Focus on features, benefits, and unique selling points:

@tagline

"""

# 4. Define the controller that orchestrates the two steps
[pipe.description_to_keywords]
type = "PipeSequence"
description = "From product description to tagline keywords"
inputs = { description = "ProductDescription" }
output = "Keyword[]"
steps = [
    { pipe = "generate_tagline", result = "tagline" },
    { pipe = "extract_keywords_from_tagline", result = "keywords" },
]
```

!!! note "Multiple Outputs"
    The `[]` bracket notation in `output = "Keyword[]"` allows the LLM to generate as many keywords as it finds relevant. For a comprehensive guide on controlling how many items pipes produce or accept, see [Understanding Multiplicity](./understanding-multiplicity.md).


This defines a two-step pipeline. The `PipeSequence` controller `description_to_keywords` takes a `ProductDescription` as input and outputs a list of `Keyword`s — a sequence's declared `output` must match what its last step yields. The first step is `generate_tagline` which generates a `Tagline` from the `ProductDescription`. The second step is `extract_keywords_from_tagline` which extracts keywords from the `Tagline`.

See here all the different operators, see [Pipe Operators](./pipe-operators/index.md).
See here all the different controllers, see [Pipe Controllers](./pipe-controllers/index.md).

See more about the possibilities in designing your pipelines in the [Pipelex Bundle Specification](../pipelex-bundle-specification.md).

## Libraries

# Libraries

A **Library** in Pipelex is a complete collection of domains, concepts, and pipes that can be loaded and used together. It represents the full set of Pipelex definitions available for execution within a specific context, typically for a single pipeline run.

## Library Structure

A Library is composed of three core components:

- **DomainLibrary**: Manages all domain definitions
- **ConceptLibrary**: Manages all concept definitions across domains
- **PipeLibrary**: Manages all pipe definitions

These three components together form what we call a **Pipelex Bundle** (the content you define in `.mthds` files). Learn more about bundle structure and syntax in the [Pipelex Bundle Specification](./pipelex-bundle-specification.md).

## Understanding Library Scope

When you execute pipelines using `execute` or `start`, a library is created to hold all the necessary definitions. This library:

- Contains the pipes and concepts available for execution
- Provides isolation between different pipeline runs when using different library IDs
- Can be loaded from local directories or from MTHDS content strings

## Uniqueness Rules

Libraries enforce specific uniqueness constraints to maintain consistency:

| Component | Uniqueness Scope | Example |
|-----------|-----------------|---------|
| **Domains** | Unique per library | Each library can have one `marketing` domain |
| **Pipes** | Unique per domain | Each domain can have one `generate_tagline` pipe |
| **Concepts** | Unique per domain | Each domain can have one `ProductDescription` concept |
| **Domain.Concept** | Unique per library | `marketing.ProductDescription` is unique within a library |

Pipes are identified by their domain-qualified reference (e.g. `marketing.generate_tagline`), so different domains within the same library can have pipes with the same code.

## How a pipe reference resolves

A reference written **inside** a pipe — a sequence step, a parallel branch, a condition outcome, a `batch_over` target — resolves within its own domain:

- `generate_tagline` written inside a `marketing` pipe means `marketing.generate_tagline`.
- To reference a pipe in another domain, write that domain out: `sales.generate_tagline`.

A bare reference never reaches another domain. This is what makes a domain a real boundary: adding a pipe to one domain cannot change what a pipe in another domain means, and `[exports]` — which deliberately exempts bare references from its check, because they were never supposed to leave home — is enforceable through every reference form.

If a bare reference names nothing in its own domain, validation says so, names the reference it actually tried, and points at the domain that does declare that code:

```
Pipe 'marketing.run_campaign' references 'marketing.render_html', which does not exist. A pipe
reference resolves inside its own domain, so 'render_html' is looked for in domain 'marketing'.
Referencing a pipe in another domain requires writing that domain out. 'render_html' is declared
elsewhere in this library — did you mean 'presentation.render_html'?
```

**Pipe codes you type yourself are a different matter.** `pipelex run generate_tagline`, `pipelex show generate_tagline`, `pipelex which generate_tagline`, and the pipe code in an API request all accept a bare code and find it in whichever domain declares it — you are pointing at a pipe, not writing a reference. If two domains declare the same code, these commands ask you to say which one rather than picking for you.

**Concept codes you supply in inputs follow the same entry rule.** A bare code in an input envelope (`{"concept": "Memo", "content": ...}`) is resolved against the entry pipe's own domain first; if that domain does not declare it, a code unique across the library is found wherever it lives, and a code two domains declare raises an error asking you to qualify it (`marketing.Memo`). Concept references written **inside** a bundle never search at all — they are qualified to their own domain when the library is built, exactly like pipe references.

## Local vs Remote Libraries

### Local Libraries (Current)

Currently, all libraries are **local**, meaning they are loaded from:

- Directories on your filesystem (using the `library_dirs` constructor parameter)
- MTHDS content strings (using the `mthds_contents` parameter)
- The current working directory (default behavior)

```python
# Loading from local directories
runner = PipelexMTHDSProtocol(library_dirs=["./pipelines", "./shared_pipes"])
response = await runner.execute(
    pipe_code="generate_tagline",
    inputs={...},
)
pipe_output = response.pipe_output
```

### Remote Libraries (Coming Soon)

In the future, you'll be able to import and use remote libraries, enabling:

- Sharing pipe definitions across teams and projects
- Versioned library dependencies
- Centralized library management

## Library Lifecycle

### 1. Library Creation

When the runner executes a method, a library is created with a unique `library_id`. Pipelex-specific configuration (library id, directories) is provided on the `PipelexMTHDSProtocol` constructor:

```python
# Explicit library ID
runner = PipelexMTHDSProtocol(library_id="my_custom_library")
response = await runner.execute(
    pipe_code="my_pipe",
    inputs={...},
)

# Automatic library ID (defaults to pipeline_run_id)
runner = PipelexMTHDSProtocol()
response = await runner.execute(
    pipe_code="my_pipe",
    inputs={...},
)
```

### 2. Library Loading

The library is populated based on the parameters you provide:

**Option A: Loading from directories**

```python
# Loads all .mthds files from specified directories
runner = PipelexMTHDSProtocol(library_dirs=["./pipelines"])
response = await runner.execute(
    pipe_code="my_pipe",
    inputs={...},
)
```

**Option B: Loading from MTHDS content**

```python
# Loads only the provided MTHDS content
mthds_content = """
domain = "marketing"

[concept]
ProductDescription = "A product description"
Tagline = "A catchy tagline for a product"

[pipe.my_pipe]
type = "PipeLLM"
description = "Generate a tagline for a product"
inputs = { desc = "ProductDescription" }
output = "Tagline"
prompt = "Generate a tagline for: $desc"
"""

runner = PipelexMTHDSProtocol()
response = await runner.execute(
    mthds_contents=[mthds_content],
    pipe_code="my_pipe",
    inputs={...},
)
```

### 3. Library Validation

After loading, the library validates:

- All pipe input/output concepts exist in the concept library
- All pipe dependencies (for pipe controllers) exist in the pipe library
- Domain and concept definitions are consistent

## Working with Multiple Libraries

You can manage multiple libraries simultaneously by using different `library_id` values:

```python
# Library for marketing pipelines
marketing_runner = PipelexMTHDSProtocol(library_id="marketing_lib", library_dirs=["./marketing_pipes"])
marketing_response = await marketing_runner.execute(
    pipe_code="generate_tagline",
    inputs={...},
)

# Library for analytics pipelines
analytics_runner = PipelexMTHDSProtocol(library_id="analytics_lib", library_dirs=["./analytics_pipes"])
analytics_response = await analytics_runner.execute(
    pipe_code="analyze_data",
    inputs={...},
)
```

## Best Practices

### 1. Use Explicit Library IDs for Long-Running Applications

```python
# Good: Explicit ID for maintaining state
runner = PipelexMTHDSProtocol(library_id="app_library")
response = await runner.execute(
    pipe_code="my_pipe",
    inputs={...},
)
```

### 2. Use MTHDS Content for Dynamic Pipelines

When generating or modifying pipelines dynamically, use `mthds_contents`:

```python
# Generate MTHDS content dynamically
mthds_content = generate_custom_pipeline(user_requirements)

runner = PipelexMTHDSProtocol()
response = await runner.execute(
    mthds_contents=[mthds_content],
    inputs={...},
)
```

A temporary library will be created holding the Pipelex bundle, and the library_id will be pipeline_run_id. Since no `pipe_code` is passed, the generated content must declare `main_pipe` at the top of the bundle (e.g. `main_pipe = "my_pipe"`) so the runner knows which pipe to execute — otherwise pass `pipe_code=` explicitly.

### 3. Reuse Library IDs for Related Executions

If multiple pipeline runs should share the same library context:

```python
runner = PipelexMTHDSProtocol(library_id="shared_context", library_dirs=["./pipes"])

# First execution
response1 = await runner.execute(
    pipe_code="pipe1",
    inputs={...},
)

# Second execution with same library
response2 = await runner.execute(
    pipe_code="pipe2",
    inputs={...},
)
```

## Related Documentation

- [Executing Pipelines](pipes/executing-pipelines.md) - Learn how to execute pipelines with different library configurations
- [Pipelex Bundle Specification](./pipelex-bundle-specification.md) - Understand the structure of MTHDS files
- [Domains](./domain.md) - Learn about organizing pipes into domains
- [Concepts](./concepts/define_your_concepts.md) - Understand how concepts work within libraries



## Packages

# Packages

A **package** is a collection of `.mthds` bundles with a `METHODS.toml` manifest at the project root. The manifest gives the package an identity and defines which pipes are exported to the outside world.

!!! info "Opt-in packaging"
    If your project has no `METHODS.toml`, Pipelex still loads your local bundles. Packaging is optional.

## What the Manifest Does

`METHODS.toml` currently serves two main purposes:

- **Identity**: give the package an address, version, and description
- **Visibility**: declare which pipes are exported through `[exports]`

## The Package Manifest: `METHODS.toml`

Here is a representative manifest:

```toml
[package]
address = "github.com/acme/legal-tools"
version = "1.0.0"
description = "Legal document analysis and contract review methods."
authors = ["Acme Corp"]
license = "MIT"
mthds_version = ">=0.8.0"

[exports.legal.contracts]
pipes = ["extract_clause", "analyze_contract"]

[exports.scoring]
pipes = ["compute_weighted_score"]
```

### Field Reference

| Field | Required | Description |
|-------|----------|-------------|
| `address` | Yes | Package address following a hostname/path pattern such as `github.com/org/repo` |
| `version` | Yes | Semantic version such as `1.0.0` or `2.1.3-beta.1` |
| `description` | Yes | Human-readable package description |
| `name` | No | Optional short method name |
| `display_name` | No | Optional display label |
| `authors` | No | List of author names |
| `license` | No | License string such as `MIT` |
| `main_pipe` | No | Optional main pipe code |
| `mthds_version` | No | Required MTHDS version constraint |

## Exports and Visibility

The `[exports]` section controls which pipes are visible outside the package.

### Default Behavior

- **Without `METHODS.toml`**: local bundles are still available to your project
- **With `METHODS.toml`**: pipes are private by default — only pipes listed in `[exports]` (and each bundle's `main_pipe`) are visible from outside

### Declaring Exports

Exports are grouped by domain path:

```toml
[exports.legal.contracts]
pipes = ["extract_clause", "analyze_contract"]

[exports.scoring]
pipes = ["compute_weighted_score"]
```

This manifest exports two pipes from `legal.contracts` and one pipe from `scoring`.

## Cross-Package References

Pipelex can resolve cross-package references when the required packages are available to the runtime. The reference form is:

```text
alias->domain.code
```

Example:

```toml
[pipe.analyze_item]
type = "PipeSequence"
description = "Analyze an item using another package"
steps = [
    { pipe = "scoring_lib->scoring.compute_weighted_score" },
]
```

This runtime-level capability exists separately from the current public `mthds package` CLI workflow documented on this page. If you rely on cross-package references, validate them against the specific runtime setup you are using.

### Example Layout

```text
your-project/
├── METHODS.toml
├── my_project/
│   ├── finance/
│   │   ├── invoices.mthds
│   │   └── invoices_struct.py
│   └── legal/
│       ├── contracts.mthds
│       └── contracts_struct.py
├── .pipelex/
│   └── pipelex.toml
└── requirements.txt
```

The `METHODS.toml` file sits at the project root. Pipelex discovers it by walking up from bundle locations until it finds the manifest.

## Current Public CLI Workflow

The public package-management commands documented in this repo are provided by the lowercase `mthds` CLI:

```bash
mthds package init
mthds package list
mthds package validate
```

### Initialize a Manifest

```bash
mthds package init
```

This command creates `METHODS.toml` interactively. It asks for package metadata and writes a manifest with an empty `[exports]` section.

### Inspect a Manifest

```bash
mthds package list
```

This command reads the manifest from the target package directory and prints the parsed package metadata and exports.

### Validate a Manifest

```bash
mthds package validate
```

This command validates the manifest syntax and package fields.

### Target a Different Directory

All three commands accept `-C, --package-dir <path>`:

```bash
mthds package init -C ./my-package
mthds package list -C ./my-package
mthds package validate -C ./my-package
```

## Important Compatibility Note

The current public `mthds package validate` command focuses on package identity and exports. It does **not** document a public dependency-management workflow here, and it rejects a `[dependencies]` section in the current JS CLI validation path.

For this reason, this documentation only describes the stable public package-manifest workflow exposed by the current `mthds` CLI.

## Related Documentation

- [Domain](./domain.md) — How domains organize concepts and pipes
- [Libraries](./libraries.md) — How libraries load and validate bundles
- [Pipelex Bundle Specification](./pipelex-bundle-specification.md) — The `.mthds` file format
- [Package Commands](../tools/cli/pkg.md) — `mthds package` command reference


## Executing Pipelines

# Executing Pipelines

Once your pipes are defined in `.mthds` files, you can execute them in multiple ways.

## The Simplest Approach: Run a Bundle File

The easiest way to execute a pipeline is to point directly to your `.mthds` bundle file. No library configuration needed.

### Using the CLI

```bash
# Run the bundle's main_pipe
pipelex run bundle path/to/my_bundle.mthds

# Run a specific pipe from the bundle
pipelex run bundle path/to/my_bundle.mthds --pipe my_specific_pipe

# Run with inputs
pipelex run bundle path/to/my_bundle.mthds --inputs inputs.json
```

!!! tip "Preparing Inputs"
    You can generate an input template with `pipelex build inputs bundle path/to/my_bundle.mthds`, which writes an `inputs.json` next to the bundle (in the bundle's own directory) with the required input structure — the light, signature-driven values ready to fill in. See [Build Inputs](../../tools/cli/build/inputs.md).

### Using Python

```python
from pathlib import Path

from pipelex.pipelex import Pipelex
from pipelex.pipeline.runner import PipelexMTHDSProtocol

Pipelex.make()

bundle_content = Path("path/to/my_bundle.mthds").read_text(encoding="utf-8")

# Run the bundle's main_pipe
runner = PipelexMTHDSProtocol()
response = await runner.execute(
    mthds_contents=[bundle_content],
    # Provide values directly — each is interpreted against the pipe's declared input signature.
    inputs={
        "my_input": "Hello world",
    },
)
pipe_output = response.pipe_output

# Or run a specific pipe from the bundle
response = await runner.execute(
    mthds_contents=[bundle_content],
    pipe_code="my_specific_pipe",
    inputs={...},
)
pipe_output = response.pipe_output
```

!!! tip "Already loading libraries elsewhere?"
    If a `PIPELEXPATH` environment variable or `Pipelex.make(library_dirs=...)` already loads this same bundle from disk, passing its content again via `mthds_contents` loads it twice and raises a duplicate-declaration error. See [Advanced: Using Library Directories](#advanced-using-library-directories) for `bundle_uris` (deduplication) and `library_dirs=[]` (run with only the provided content).

!!! info "How `main_pipe` Works"
    When you run a bundle without specifying a `pipe_code`, Pipelex executes the bundle's `main_pipe` (declared at the top of the `.mthds` file). If no `main_pipe` is defined and no `pipe_code` is provided, an error is raised.

    If you provide both `mthds_contents` and `pipe_code`, the explicit `pipe_code` takes priority over `main_pipe`.

See the [Pipelex Bundle Specification](../pipelex-bundle-specification.md) for more about the `main_pipe` property.

---

## Advanced: Using Library Directories

For larger projects with multiple bundles spread across directories, you can configure **library directories** to load all your pipes at once and reference them by code.

### Understanding Libraries

When executing pipelines programmatically, Pipelex can load pipe **libraries** - collections of pipes loaded from one or more directories. This is useful when:

- You have multiple bundles across different directories
- You want to reference pipes by code without specifying the bundle path each time
- You're building an application that needs access to many pipes

#### Library Parameters

When using `PipelexMTHDSProtocol`, you can control library behavior with these parameters:

- **`library_id`**: A unique identifier for the library instance. If not specified, it defaults to the `pipeline_run_id` (a unique ID generated for each pipeline execution).

- **`library_dirs`**: A list of directory paths to load pipe definitions from. **These directories must contain both your `.mthds` files AND any Python files defining `StructuredContent` classes** (e.g., `*_struct.py` files). If not specified, Pipelex falls back to the default library directories passed to `Pipelex.make(library_dirs=...)`, then to the `PIPELEXPATH` environment variable; if none of these are set, no library directories are loaded.

- **`mthds_contents`**: When provided to `PipelexMTHDSProtocol.execute()`, Pipelex parses these strings and loads them into the library in addition to any resolved library directories. To run with only this content, disable directory loading explicitly (e.g. `library_dirs=[]`). This is useful for dynamic pipeline execution without file-based definitions.

- **`bundle_uris`**: Optional file paths identifying the bundles passed as `mthds_contents` (matched by position). When a URI matches a bundle already loaded from the library directories, that content is skipped instead of being loaded twice. Without `bundle_uris`, every `mthds_contents` entry is loaded unconditionally — so passing content that duplicates a directory-loaded bundle raises a duplicate-declaration error.

!!! info "Python Structure Classes"
    If your concepts use Python `StructuredContent` classes instead of inline structures, those Python files must be in the directories specified by `library_dirs`. Pipelex auto-discovers and registers these classes during library loading. Learn more about [Python StructuredContent Classes](../concepts/python-classes.md).

!!! info "Learn More About Libraries"
    For a comprehensive understanding of libraries, including their structure, uniqueness rules, lifecycle, and best practices, see [Libraries](../libraries.md).

!!! tip "Library Directory Configuration"
    For complete configuration options including the `PIPELEXPATH` environment variable, CLI options, and priority resolution, see [Configuring Library Directories](../../configuration/config-technical/library-config.md#configuring-library-directories).

### Running Pipes by Code

This approach loads pipe definitions from directories and executes a specific pipe by its code.

**Basic usage** (default library directories set at initialization):

```python
from pipelex.pipelex import Pipelex
from pipelex.pipeline.runner import PipelexMTHDSProtocol

# First, initialize Pipelex with default library directories
# (used by every run unless overridden per call)
Pipelex.make(library_dirs=["./pipelines"])

# Execute the pipeline and wait for the result
runner = PipelexMTHDSProtocol()
response = await runner.execute(
    pipe_code="description_to_tagline",
    inputs={
        "description": "EcoClean Pro is a revolutionary biodegradable cleaning solution that removes 99.9% of germs while being completely safe for children and pets. Made from plant-based ingredients.",
    },
)
pipe_output = response.pipe_output
```

**Loading from specific directories**:

```python
# Load pipes from specific directories
runner = PipelexMTHDSProtocol(
    library_dirs=["./pipelines", "./shared_pipes"],
)
response = await runner.execute(
    pipe_code="description_to_tagline",
    inputs={
        "description": "...",
    },
)
pipe_output = response.pipe_output
```

**Using a custom library ID**:

```python
# Use a custom library ID for managing multiple library instances
runner = PipelexMTHDSProtocol(
    library_id="my_marketing_library",
    library_dirs=["./marketing_pipes"],
)
response = await runner.execute(
    pipe_code="description_to_tagline",
    inputs={
        "description": "...",
    },
)
pipe_output = response.pipe_output
```

!!! tip "Listing available pipes"
    Use `pipelex show pipe --all` to list the pipes available in your project.

### Using MTHDS Content Directly

You can directly pass MTHDS content as a string to `PipelexMTHDSProtocol.execute()`, useful for dynamic pipeline execution without file-based definitions.

```python
from pipelex.pipelex import Pipelex
from pipelex.pipeline.runner import PipelexMTHDSProtocol

my_pipe_content = '''
domain = "marketing"
description = "Marketing content generation domain"
main_pipe = "generate_tagline"

[concept]
ProductDescription = "A description of a product's features and benefits"
Tagline = "A catchy marketing tagline"

[pipe.generate_tagline]
type = "PipeLLM"
description = "Generate a catchy tagline for a product"
inputs = { description = "ProductDescription" }
output = "Tagline"
prompt = """
Product Description: $description

Generate a catchy tagline based on the above description. The tagline should be memorable, concise, and highlight the key benefit.
"""
'''

Pipelex.make()

runner = PipelexMTHDSProtocol()
response = await runner.execute(
    mthds_contents=[my_pipe_content],
    inputs={
        "description": "EcoClean Pro is a revolutionary biodegradable cleaning solution that removes 99.9% of germs while being completely safe for children and pets. Made from plant-based ingredients.",
    },
)
pipe_output = response.pipe_output
```

!!! note "Pipe Code Resolution"
    When using `mthds_contents`:

    - If the content has a `main_pipe` property and you don't provide `pipe_code`, the `main_pipe` is executed
    - If you provide `pipe_code`, it overrides `main_pipe`
    - If there's no `main_pipe` and no `pipe_code`, an error is raised

---

## Using the Pipelex API

Pipelex has a REST API for executing pipelines. See more about it [here](https://pipelex.github.io/pipelex-api/).

---

## Background Execution

<!-- start exists on PipelexMTHDSProtocol as a protocol stub (raises NotImplementedError).
     The public entry point is execute(). Keep this note so reviewers don't flag the stub as contradictory. -->
The public Python entry point is `PipelexMTHDSProtocol.execute()`. The older standalone `start` function is no longer a documented entry point.

---

## Run Modes & Backends

Every execution method above accepts a run mode (live or dry run) and a backend (in-process or Temporal). See [Run Modes & Backends](run-modes-and-backends.md) for the full matrix and when to use each combination.


## Run Modes & Backends

# Run Modes & Backends

Pipelex separates **what the inference leaves do** (the run mode) from **where the work executes** (the backend). The two axes are independent: any run mode works on any backend, because the run mode travels with every inference job and is honored at the leaf — the last stop before a provider would actually be called.

## Axis 1: Run Mode

| Run mode | CLI trigger | Python trigger | AI providers | Usage & cost report |
|---|---|---|---|---|
| **Live** (default) | — | — | Real calls | Real usage, report rendered |
| **Dry run** | `--dry-run` | `pipe_run_mode=PipeRunMode.DRY` | Never called — leaves build mocks | Zero tokens, report suppressed |

**Live** runs call the configured AI providers for real: real cost, real latency, and storage IO for generated images and extracted pages.

**Dry run** exercises the full pipeline — controllers, working memory, data flow, execution-graph tracing — with no AI cost. Each inference leaf branches to a format-compliant mock instead of calling the provider (see [Dry Run Mock Generation](../../under-the-hood/dry-run-mock-generation.md)). A dry run needs no API keys, performs no storage IO (mocked images and pages never touch the storage provider), reports zero-token usage, and suppresses the end-of-run cost report. Add `--mock-inputs` to also synthesize any missing required inputs (it requires `--dry-run`).

## Axis 2: Backend

| Backend | Trigger | What runs where |
|---|---|---|
| **Direct** (default) | — | Everything in your process |
| **Temporal** | `--orchestrator temporal`, or config `plugins.boot_orchestrator = "temporal"` | Controllers as child workflows, inference leaves as activities on workers |

!!! info "Durable backends are a commercial capability"
    The Temporal backend is part of Pipelex's [durable & distributed execution offer](https://pipelex.com/products#durable-execution) — the [Temporal backend](https://pipelex.com/products#temporal) is delivered through the Pipelex platform. The `--orchestrator <name>` flag (and its `plugins.boot_orchestrator` config equivalent) boots the process under the named orchestrator plugin; omit it for in-process execution.

The run mode rides across the wire with every inference job, so the backend never changes *what* a leaf does — only *where* it does it. In particular, when you run a **pipeline** in dry mode on Temporal, the real inference activities are still dispatched and mock **inside** them on the worker. That's by design: it lets you test the entire distribution machinery — dispatch, scheduling, serialization, task-queue routing, cross-worker graph tracing — with zero AI spend and no credentials.

One job type deliberately opts out of this fan-out: a **validation sweep** submitted through the runner API runs as a single in-process activity instead of dispatching nested activities — see [Validation Sweeps](#validation-sweeps) below for how that fits the model.

## The Matrix

All cells below describe **pipeline runs** — `pipelex run ...` or `PipelexMTHDSProtocol.execute()`. Validation sweeps have their own distribution shape, covered in the next section.

| Run mode \ Backend | Direct (in-process) | Temporal |
|---|---|---|
| **Live** | Standard local run. | Durable distributed run; real inference executes on workers. |
| **Dry run** | Leaves mock inline. Validates pipeline logic and data flow at zero cost. | Activities are genuinely dispatched and mock inside the worker — proves out the distribution machinery with no API keys, no spend, no storage IO. |

```bash
# Live + direct (the default)
pipelex run bundle my_bundle.mthds

# Dry run + direct
pipelex run bundle my_bundle.mthds --dry-run --mock-inputs

# Live + Temporal
pipelex run bundle my_bundle.mthds --orchestrator temporal

# Dry run + Temporal — dispatches real activities, mocks inside them
pipelex run bundle my_bundle.mthds --orchestrator temporal --dry-run --mock-inputs
```

From Python, the same knobs live on the runner — the backend comes from config:

```python
from pathlib import Path

from pipelex.system.pipe_run_mode import PipeRunMode
from pipelex.pipeline.runner import PipelexMTHDSProtocol

runner = PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.DRY)
response = await runner.execute(
    mthds_contents=[Path("path/to/my_bundle.mthds").read_text(encoding="utf-8")],
)
```

## Validation Sweeps

`pipelex validate` (see the [validate commands](../../tools/cli/validate.md)) combines static validation with a dry run of every pipe. It relates to the matrix in two specific ways:

- **The CLI sweep always runs in-process**, even when Temporal is enabled. The `--orchestrator temporal` flag on `pipelex validate` exists precisely to verify that: it boots validation under the orchestrator plugin so you can confirm a validation sweep does not dispatch anything under a Temporal-enabled setup.
- **The runner API's `/validate` route ships the sweep to a worker — as one activity**: with Temporal enabled, it offloads the *whole* sweep as a single activity (`act_dry_validate`, wrapped in the `WfDryValidate` workflow). Inside that activity, the pipe router and the content generator are scoped to their in-process variants, so nothing nested is dispatched: every dry run in the sweep behaves exactly like the dry × direct cell of the matrix, just hosted on a worker instead of your machine. The per-pipe statuses plus the execution graph come back in one round-trip. See [Runtime Bridge & Transport](../../under-the-hood/runtime-bridge-and-transport.md) for how a job crosses the worker boundary.

This is not a contradiction of "the run mode never changes the backend" — it's the same leaf-mock foundation with a different *backend choice* for the nested work. A `--orchestrator temporal --dry-run` pipeline run picks the Temporal backend all the way down, fanning out for real to test the distribution machinery. A Temporal validation sweep picks the Temporal backend for exactly one hop (get the sweep onto a worker) and the in-process backend for everything inside, because its goal is a cheap interactive answer, not distribution testing. Two deliberate opposites built from the same parts.

## Forced Dry Mode

When a process boots without inference configured — `Pipelex.make(needs_inference=False)`, e.g. no AI provider set up — every run that process initiates is forced to dry-run mode. This is a property of the run, not of the backend: under a Temporal-enabled boot the forced-dry run still dispatches activities and mocks inside them, like any other dry run. A warning logs whenever a live-requested run is forced dry.

## See Also

- [Validation & Dry Run](../../features/validation-dry-run.md) — feature overview
- [Dry Run Configuration](../../configuration/config-pipeline-validation/dry-run-config.md) — mock list sizes, allowed-to-fail pipes
- [Cost Tracking & Reporting](../../features/cost-tracking.md) — what the usage reports contain
- [Distributed Execution](../../distributed-execution/index.md) — running on Temporal
- [Dry Run Mock Generation](../../under-the-hood/dry-run-mock-generation.md) — how format-compliant mocks are built


## Providing Inputs to Pipelines

# Providing Inputs to Pipelines

Every pipe declares what it needs: an `inputs` signature mapping each variable name to a concept (and, for lists, a multiplicity). When you run a pipe, **you just provide the values** — Pipelex interprets each one *top-down against that declared signature*, so a bare string becomes the declared concept (a `legal.Question`, not a generic `native.Text`), a bare number satisfies a numeric input, and a plain object validates against a structured concept.

This is the fast path. There is also an explicit `{"concept": ..., "content": ...}` escape hatch for the rare cases where you need to override or disambiguate the concept — it still works, and it is now compatibility-checked against the declaration. Both are covered below.

## See what a pipe expects

Generate a template that mirrors the pipe's signature:

```bash
pipelex build inputs bundle path/to/my_pipe.mthds
```

This writes `inputs.json` (add `--format toml` for TOML) with an example value per input, shaped exactly the way the pipe expects them. Fill in the values and run:

```bash
pipelex run bundle path/to/my_pipe.mthds --inputs inputs.json
```

For a pipe declaring `question = "Question"`, `priority = "Priority"`, `invoice = "Invoice"`, and `tags = "Tag[]"`, the template is the **light** shape — the values themselves, not envelopes:

```json
{
  "question": "text_value",
  "priority": 1,
  "invoice": {
    "invoice_number": "invoice_number_value",
    "amount": 0.0
  },
  "tags": ["text_value"]
}
```

The TOML template carries the declared concept for each key as a comment, so you can see what you are filling in (JSON has no comments — use `--explicit` there to see concepts):

```toml
# concept: my_domain.Question
question = "text_value"
# concept: my_domain.Priority
priority = 1
# concept: my_domain.Invoice
invoice = {invoice_number = "invoice_number_value", amount = 0.0}
# concept: my_domain.Tag[]
tags = ["text_value"]
```

!!! tip "Prefer the light template"
    The light values are exactly what the runner accepts, so a template you generate and fill in runs as-is. Add `--explicit` to `build inputs` if you want the ceremonial `{concept, content}` envelope form instead (see [the escape hatch](#the-explicit-format-escape-hatch)).

## Providing values by concept type

Match the value to the input's declared concept. The declared concept is applied automatically — you never repeat it in the value.

| Declared concept refines… | Provide | Example |
| --- | --- | --- |
| `Text` | a string | `"What are the fees?"` |
| `Number` | a number (not a boolean) | `42` or `3.14` |
| `YesNo` | a boolean | `true` / `false` |
| `Date` | an extended ISO 8601 date/datetime string (or a TOML date literal) | `"2026-09-01"` |
| `Image` / `Document` | a URL or file path | `"photo.jpg"`, `"https://…/a.pdf"` |
| a structured concept | an object | `{"name": "Alice", "age": 30}` |

```json
{
  "question": "What are the late-payment fees?",
  "priority": 3,
  "is_urgent": true,
  "hearing": "2026-09-01",
  "contract": "contracts/nda.pdf",
  "client": {
    "name": "Acme Corp",
    "country": "France"
  }
}
```

Each value is typed as the **declared** concept — `question` becomes a `legal.Question`, not a bare `native.Text` — so downstream steps see the concept the pipe was designed around.

!!! note "Numbers vs. booleans"
    A boolean is never read as a number, even though `true`/`false` look numeric to some languages. Provide `true`/`false` only for a `YesNo`-refining input; provide `42` for a `Number`-refining one.

!!! note "Concepts the signature can't shape"
    A handful of concepts carry no single expected value shape — `Dynamic`, `Anything`, and the container/structural natives (`Html`, `JSON`, `Page`, `TextAndImages`, `SearchResult`, `Composite`). For an input declared as one of these, the signature can't guide interpretation, so the value is read by its own shape (a bare string becomes `native.Text`, and so on). Use the explicit format if you need a specific concept there.

## Lists and multiplicity

When an input is declared multiple (`Tag[]`, or a fixed count `Tag[3]`), provide a JSON/TOML list — each element is shaped element-wise into the declared item concept:

```json
{
  "tags": ["billing", "urgent", "enterprise"]
}
```

- A **single** value auto-wraps into a one-item list, so `"tags": "urgent"` is accepted for a `Tag[]` input.
- An **empty** list is legal and produces an empty collection.
- A fixed count `Tag[3]` validates the number of items — too few or too many is a clear error.
- Providing a list where the signature declares a **single** value is a clear error (Pipelex won't silently pick one).

See [Understanding Multiplicity](understanding-multiplicity.md) for the full model of single items versus collections.

### Tabular data for a structured list

A declared structured list (e.g. `people = "Person[]"`) also accepts a path to a `.csv`/tabular file — each row becomes one item:

```json
{
  "people": "team.csv"
}
```

The columns map to the concept's fields. See [CSV Input & Output](csv-input-and-output.md).

## Files and relative paths

For `Image`/`Document` inputs, a **relative** path is resolved against the **inputs file's** directory (a leading `~` expands to your home directory; absolute paths, `http(s)://`, and storage URIs are used as-is). So an `inputs.json` sitting next to a `contracts/` folder can say `"contract": "contracts/nda.pdf"` and it resolves correctly regardless of where you launch the command.

Tabular references use the same relative-path resolution, but v1 reads local tabular files only. A `.csv` path must resolve to a local file; download remote or storage-hosted tables before referencing them.

Callers that don't load from a file — the Python API and the hosted client — should pass absolute URLs or storage URIs, since there is no inputs-file directory to resolve against.

## The explicit format (escape hatch)

The explicit envelope `{"concept": "...", "content": ...}` gives you direct control over which concept a value is interpreted as. It is now **compatibility-checked**: the concept you name must be compatible with the input's declared concept, otherwise the run fails with a clear error.

```json
{
  "question": {
    "concept": "legal.Question",
    "content": "What are the fees?"
  }
}
```

Reach for it when:

- **You want a more-specific concept than the signature declares.** If an input is declared `native.Text` but you have a `legal.Question`, name it explicitly — a compatible, more-specific concept wins.
- **A concept name is ambiguous.** When you reference a concept by bare name and it exists in multiple domains, prefix the domain: `"accounting.Invoice"` instead of `"Invoice"`.
- **You are passing a Python object** (see below).

Generate the envelope template with `pipelex build inputs … --explicit`:

```json
{
  "question": {"concept": "my_domain.Question", "content": {"text": "text_value"}},
  "priority": {"concept": "my_domain.Priority", "content": {"number": 1}},
  "invoice": {"concept": "my_domain.Invoice", "content": {"invoice_number": "invoice_number_value", "amount": 0.0}},
  "tags": {"concept": "my_domain.Tag", "content": [{"text": "text_value"}]}
}
```

### Concept name resolution

Inside an explicit envelope, the `concept` you write is resolved against your loaded domains:

- A bare name (`"Invoice"`) found in exactly one domain is used directly.
- A bare name found in **multiple** domains is an error — prefix the domain (`"accounting.Invoice"`).
- A name found in **no** domain is an error.

Domain-prefixed names (`"domain_code.ConceptName"`) bypass the search and are always unambiguous.

## Python API and client

From Python, the same light values work — and you can additionally pass already-built content objects:

```python
from pipelex.core.stuffs.document_content import DocumentContent
from pipelex.core.stuffs.image_content import ImageContent

inputs = {
    # Light values, shaped against the signature
    "topic": "A robot learning to love",
    "priority": 3,

    # Native content objects (typed directly)
    "photo": ImageContent(url="photo.jpg"),
    "contract": DocumentContent(url="nda.pdf"),

    # Explicit envelope with a custom concept
    "draft_tweet": {
        "concept": "social.DraftTweet",
        "content": "Check out this framework!",
    },
}
```

A list of content objects (`[MyConcept(...), MyConcept(...)]`) or an already-wrapped `ListContent(items=[...])` is accepted for a declared-multiple input; a `DictStuff(concept=..., content=...)` is the object form of the explicit envelope. All are compatibility-checked against the declaration.

!!! note "Nothing to provide for a list input? Send the empty list."
    A plural slot is never *absent* — when nothing is found, it **is** the empty list (see [Understanding Multiplicity](understanding-multiplicity.md)). So a caller with no pictures provides `"photos": []`, in either spelling — bare, or as the envelope `{"concept": "native.Image", "content": []}`. Both build an empty `ListContent` typed with the declared item concept, and a template that guards its use (`{% if photos %}`) renders nothing.

    **Omitting the key is not the same thing**, and fails the run with *"missing required inputs"*: the presence markers are for singular values, and `Image[]?` is rejected outright (`optional_marker_invalid`) precisely because a plural slot has no absence to declare. See [Understanding Optionality](understanding-optionality.md).

!!! note "Bare numbers and booleans from strictly-typed Python"
    A bare number or boolean (`"priority": 3`) is accepted at runtime, but the `PipelineInputs` protocol type does not yet admit bare scalars, so a strict static type-checker may flag it until that type is widened. If your codebase runs a type-checker, type the inputs dict as `dict[str, Any]`, or pass the value through the explicit envelope (`{"concept": "...", "content": 3}`).

## Unknown input names are rejected

If you provide a name the pipe does not declare, the run fails with an error that lists the declared names. This catches typos and stale inputs files early, rather than silently ignoring the extra value.

```text
Input 'invoce' is not declared by this pipe. Declared inputs: 'question', 'priority', 'invoice', 'tags'.
```

## What can go wrong

When a value cannot be shaped against its declared concept, Pipelex raises a typed error that names the input, the declared concept, what you provided, and the **expected shape**. Common cases:

- **Wrong kind** — a string where a number is declared, a list where a single value is declared, an object where a scalar is declared.
- **Count mismatch** — the wrong number of items for a fixed-count `X[N]` input.
- **Structure validation** — an object missing a required field of a structured concept.
- **Incompatible explicit concept** — an envelope naming a concept that is not compatible with the declaration.
- **Null value** — a top-level `null`; absence is expressed by *omitting* the key, not by providing `null`. (An input declared optional with `?` may simply be left out — see [Understanding Optionality](understanding-optionality.md).)

## Input files: JSON or TOML

An inputs file is a dictionary of input names to values, accepted as **both JSON and TOML**, discriminated by file extension (`.toml` → TOML, everything else → JSON). Both produce the same dictionary, so every shape above applies to either. TOML's multi-line strings (`"""…"""`) make text-heavy inputs far easier to author, and a top-level TOML date/datetime literal (`hearing = 2026-09-01`) maps directly to the native [`Date`](../concepts/native-concepts.md) concept.

See the [run CLI reference](../../tools/cli/run.md#input-file-formats) for the extension rule, auto-detection, and TOML details.

## Related Documentation

- [Executing Pipelines](./executing-pipelines.md) — run pipelines with these inputs
- [Understanding Multiplicity](understanding-multiplicity.md) — single items vs. lists
- [Understanding Optionality](understanding-optionality.md) — optional (`?`) inputs
- [CSV Input & Output](csv-input-and-output.md) — tabular inputs and outputs
- [Native Concepts](../concepts/native-concepts.md) — the built-in concept families
- [Define Your Concepts](../concepts/define_your_concepts.md) — concepts and their role
- [Build Inputs (CLI)](../../tools/cli/build/inputs.md) — generate an inputs template


## Working Memory

# Working Memory

!!! tip "MTHDS Standard Reference"
    Working memory is part of the MTHDS open standard. For the authoritative language specification, see [Working Memory](https://mthds.ai/latest/language/working-memory/) on mthds.ai. This page documents Pipelex-specific behavior and usage.

The **Working Memory** is the mechanism that enables data flow between pipes in your pipeline. It acts as a **temporary** storage space that exists for the duration of a single pipeline run.

## How Data Flows Between Pipes

When you compose pipes together with [PipeControllers](./pipe-controllers/index.md), you need a way to pass data from one pipe to another. This is where the Working Memory comes in.

Consider our marketing pipeline example from the [Designing Pipelines](index.md) guide:

```toml
[pipe.description_to_tagline]
type = "PipeSequence"
description = "From product description to tagline"
inputs = { description = "ProductDescription" }
output = "Tagline"
steps = [
    { pipe = "generate_tagline", result = "tagline" },
    { pipe = "extract_keywords_from_tagline", result = "keywords" },
]
```

How does data get from `generate_tagline` to `extract_keywords_from_tagline`? This is handled by the Working Memory:

1.  When a pipe in a sequence executes, its output is given a name using the `result` key (e.g., `result = "tagline"`).
2.  This named result is placed into the Working Memory as a `Stuff`.
3.  Subsequent pipes can then reference this data by its name in their `inputs` field (e.g., `inputs = { tagline = "Tagline" }`).

This mechanism allows you to chain pipes together, creating a flow of information through your pipeline.

## Working Memory Lifecycle

*   **Creation**: The Working Memory is created at the start of a pipeline run.
*   **Population**: Initial inputs are placed in the Working Memory before the first pipe executes.
*   **Updates**: As each pipe completes, its output (named via the `result` field) is added to the Working Memory.
*   **Access**: Any pipe can access data from the Working Memory by declaring it in their `inputs` field.
*   **Disposal**: The Working Memory is cleared when the pipeline run completes.

## Best Practices

*   **Meaningful Names**: Use descriptive names for your `result` values to make your pipeline easier to understand.
*   **Clear Contracts**: Each pipe's `inputs` field clearly declares what data it needs from the Working Memory.
*   **Fail Fast**: If a required input is missing from the Working Memory, the pipeline will fail immediately with a clear error message before the pipe executes.

For more information on how to execute pipelines and provide initial inputs to the Working Memory, see [Executing Pipelines](executing-pipelines.md).



## Pipe Output

# Pipe Output

When you execute a pipeline, each pipe produces a `Stuff` that is automatically stored in the [Working Memory](working-memory.md). The final output of the pipeline execution is accessible through a `PipeOutput` object.

## How Pipe Outputs Work

Every pipe in your pipeline creates a **stuff** as its output. This stuff is a structured data container that holds:

A `Stuff` is a structured data container that holds:

- The concept that defines the structure
- The actual instance of the concept (text, structured data, images, etc.)

Each pipe's output is automatically added to the working memory with a name (specified by the `result` field in the [Pipe Sequence](./pipe-controllers/PipeSequence.md)). 

The **main_stuff** is the final output produced by the last pipe in your pipeline.

## Accessing Pipeline Output

When you execute a pipeline, you receive a `PipeOutput` object:

```python
runner = PipelexMTHDSProtocol()
response = await runner.execute(
    pipe_code="process_invoice",
    inputs={...},
)
pipe_output = response.pipe_output
```

The `PipeOutput` object provides methods to extract and work with the main output.

## Extracting the Main Output

How you extract the main output depends on how the concept structure was defined.

### Option 1: Python Class Structure

If the output concept was defined as a Python class (inheriting from `StructuredContent`, see more here [Python StructuredContent Classes](../concepts/python-classes.md)), you can import the class and use it for type-safe extraction:

```python
from my_project.finance.finance_struct import Invoice

runner = PipelexMTHDSProtocol()
response = await runner.execute(
    pipe_code="extract_invoice",
    inputs={...},
)
pipe_output = response.pipe_output

# Extract with type safety
invoice = pipe_output.main_stuff_as(content_type=Invoice)
# invoice is now typed as Invoice with full IDE support
```

### Option 2: Inline Structure

If the output concept was defined with [inline structures](../concepts/inline-structures.md) directly in the `.mthds` file, the generated class is not importable. Use the `PipeOutput` accessor methods instead:

```python
runner = PipelexMTHDSProtocol()
response = await runner.execute(
    pipe_code="extract_invoice",
    inputs={...},
)
pipe_output = response.pipe_output

# Access as dictionary-like structure
main_stuff = pipe_output.main_stuff
content = main_stuff.content  # Access the content object

# Or for text outputs
text_content = pipe_output.main_stuff_as_text
text = pipe_output.main_stuff_as_str  # Direct string access
```

## PipeOutput Accessor Methods

The `PipeOutput` class provides specialized methods for accessing different types of content:

### Generic Accessors

**`main_stuff`** - Returns the raw `Stuff` object containing the output:

```python
stuff = pipe_output.main_stuff
```

**`main_stuff_as(content_type)`** - Extracts content as a specific type:

```python
invoice = pipe_output.main_stuff_as(content_type=Invoice)
```

### List Accessors

**`main_stuff_as_list(item_type)`** - Returns a `ListContent` wrapper with typed items:

```python
invoice_list = pipe_output.main_stuff_as_list(item_type=Invoice)
# Access: invoice_list.items
```

**`main_stuff_as_items(item_type)`** - Returns the items directly as a Python list:

```python
invoices = pipe_output.main_stuff_as_items(item_type=Invoice)
# Direct list of Invoice objects
```

### Native Content Type Accessors

For native Pipelex concepts, use these convenience properties:

**`main_stuff_as_text`** - Returns `TextContent` object:

```python
text_content = pipe_output.main_stuff_as_text
```

This works only if the main_stuff is (or refines) a `TextContent` concept.

**`main_stuff_as_str`** - Returns the text as a string:

```python
text = pipe_output.main_stuff_as_str
```

This works only if the main_stuff is (or refines) a `TextContent` concept.

**`main_stuff_as_image`** - Returns `ImageContent` object:

```python
image = pipe_output.main_stuff_as_image
```

This works only if the main_stuff is (or refines) a `ImageContent` concept.

**`main_stuff_as_text_and_image`** - Returns `TextAndImagesContent` object:

```python
content = pipe_output.main_stuff_as_text_and_image
```

This works only if the main_stuff is (or refines) a `TextContent` and a `ImageContent` concept.

**`main_stuff_as_number`** - Returns `NumberContent` object:

```python
number = pipe_output.main_stuff_as_number
```

This works only if the main_stuff is (or refines) a `NumberContent` concept.

**`main_stuff_as_yes_no`** - Returns `YesNoContent` object:

```python
verdict = pipe_output.main_stuff_as_yes_no.yes_no  # a bool
```

This works only if the main_stuff is (or refines) a `YesNoContent` concept.

**`main_stuff_as_html`** - Returns `HtmlContent` object:

```python
html = pipe_output.main_stuff_as_html
```

This works only if the main_stuff is (or refines) a `HtmlContent` concept.

**`main_stuff_as_mermaid`** - Returns `MermaidContent` object:

```python
mermaid = pipe_output.main_stuff_as_mermaid
```

This works only if the main_stuff is (or refines) a `MermaidContent` concept.

## Working Memory Access

The `PipeOutput` object also provides access to the complete working memory:

```python
working_memory = pipe_output.working_memory
```

This allows you to access intermediate results from multi-step pipelines. See [Working Memory](working-memory.md) for details.

## Related Documentation

- [Working Memory](working-memory.md) - Understanding data flow between pipes
- [Executing Pipelines](executing-pipelines.md) - How to run pipelines
- [Inline Structures](../concepts/inline-structures.md) - Defining structures in `.mthds` files
- [Python StructuredContent Classes](../concepts/python-classes.md) - Defining structures in Python



## Understanding Multiplicity

# Understanding Multiplicity

!!! tip "MTHDS Standard Reference"
    Multiplicity is part of the MTHDS open standard. For the authoritative language specification, see [Multiplicity](https://mthds.ai/latest/language/multiplicity/) on mthds.ai. This page documents Pipelex-specific behavior and usage.

Multiplicity in Pipelex defines how many items a particular stuff can comprise in a particular context. This applies to any of the pipe input variables and also to the output of the pipe. This idea is fundamental to building flexible AI methods that can handle both single items and collections.

This guide explains the philosophy behind multiplicity in Pipelex and how to use it effectively in your pipelines.

## The Philosophy: Why Concepts Are Always Singular

In Pipelex, concepts are always defined in the singular form. You define a `Keyword`, not `Keywords`. You define an `Invoice`, not `Invoices`. This isn't just a naming convention—it's a fundamental design principle.

### Concepts Represent Semantic Entities

A concept represents a semantic entity—a meaningful piece of knowledge with clear boundaries. When you define a concept, you're describing what something *is*, not how many of them you might have:

```toml
[concept]
Keyword = "A significant word or term extracted from text"
ProductIdea = "A concept for a new product or service"
SourceDocument = "A written or printed record containing information"
```

Each of these definitions describes a single, coherent entity. The essence of what makes something a "Keyword" doesn't change whether you have one keyword or a hundred.

### Lists Are Circumstantial, Not Essential

The number of items you're working with is a circumstantial detail of your method, not part of the concept's identity:

- A pipe that extracts keywords from text might find 3 keywords or 30—but each is still a `Keyword`
- A pipe that generates product ideas might produce 5 ideas or 10—but each remains a `ProductIdea`
- A pipe that processes documents might handle 1 document or a batch of 100—but each is still a `SourceDocument`

By keeping concepts singular and using multiplicity to express quantity, Pipelex maintains a clean semantic model that's both flexible and intuitive.

## Output Multiplicity

Output multiplicity controls how many items a pipe produces. You can specify this using bracket notation in the `output` field.

### The Three Output Multiplicity Modes

**1. Single output (default)**

When no brackets are used in the output, the pipe produces a single item:

```toml
[concept]
Summary = "A concise overview of content"

[pipe.summarize]
type = "PipeLLM"
description = "Create a summary of the document"
inputs = { document = "Text" }
output = "Summary"
prompt = """
Summarize this document concisely:

@document
"""
```

This is the default behavior and represents the most common case.

**2. Variable output (bracket notation `[]`)**

Use empty brackets in the output to let the LLM decide how many items to generate:

```toml
[concept.LineItem]
description = "A single line item from an invoice"

[concept.LineItem.structure]
description = "Description of the item or service"
quantity = { type = "number", description = "Quantity purchased" }
unit_price = { type = "number", description = "Price per unit" }
total = { type = "number", description = "Total price for this line" }

[pipe.extract_line_items]
type = "PipeLLM"
description = "Extract all line items from an invoice"
inputs = { invoice_text = "Text" }
output = "LineItem[]"
prompt = """
Extract all line items from this invoice:

@invoice_text

For each line item, extract the description, quantity, unit price, and total amount.
"""
```

The pipe will extract however many line items appear in the invoice. A simple invoice might have 2 line items, while a detailed purchase order might have 50.

**Common use cases for variable output:**

- Extract entities from text (unknown count in advance)
- Generate as many alternatives as needed
- List all items that match criteria
- Identify all occurrences of a pattern

**3. Fixed output (bracket notation `[N]`)**

Use a number in brackets to generate an exact number of items:

```toml
[concept]
Headline = "A catchy title for content"

[pipe.generate_headline_options]
type = "PipeLLM"
description = "Generate headline alternatives"
inputs = { article_text = "Text" }
output = "Headline[5]"
prompt = """
Read this article:

@article_text

Generate 5 different headline options for this article.
Make each one unique and compelling.
"""
```

The pipe will always produce exactly 5 headlines.

**Common use cases for fixed output:**

- Generate N alternative versions for A/B testing
- Create a fixed set of options for user selection
- Produce a specific number of variations for comparison
- Match external requirements (e.g., "always provide 3 recommendations")

## Input Multiplicity

Input multiplicity specifies whether a pipe expects a single item or multiple items as input. This allows you to design pipes that explicitly require lists or explicitly require single items.

### Syntax for Input Multiplicity

Input multiplicity is specified using bracket notation in the `inputs` dictionary:

```toml
# Standard syntax (single item, the default)
inputs = { document = "Text" }

# Variable list (indeterminate number of items)
inputs = { documents = "Text[]" }

# Fixed count (exactly N items)
inputs = { comparison_items = "Text[2]" }
```

### The Three Input Multiplicity Modes

**1. Single item (default)**

When you use the standard syntax, the pipe expects exactly one item. This is the default behavior:

```toml
[concept]
Report = "A detailed analytical document"

[pipe.analyze_report]
type = "PipeLLM"
description = "Analyze a single report"
inputs = { report = "Report" }
output = "Analysis"
prompt = """
Analyze this report in detail:

@report
"""
```

!!! note "Default Single Item Behavior"
    When no brackets are used, the input expects a single item. Only use brackets when you need multiple items (`[]`) or a specific count (`[N]`).

**2. Variable list (bracket notation `[]`)**

Use empty brackets `[]` to specify that the pipe expects a list with an indeterminate number of items:

```toml
[concept]
SourceDocument = "A written or printed record"
Summary = "A concise overview of multiple documents"

[pipe.summarize_all_documents]
type = "PipeLLM"
description = "Create a unified summary of multiple documents"
inputs = { documents = "SourceDocument[]" }
output = "Summary"
prompt = """
Analyze all of these documents:

@documents

Create a single unified summary that captures the key points across all documents.
"""
```

!!! info "When You Actually Need `[]` Notation"
    The `[]` notation is only required for **advanced use cases**:
    
    1. **Batching over items**: When using `batch_over` in a `PipeSequence` to process each item separately
    2. **Looping in templates**: When you need to iterate over items using Jinja2 syntax (`{% for item in items %}`) in `PipeLLM`, `PipeCompose`, or `PipeCondition` prompts
    
    For most cases where you simply pass multiple items to a pipe that processes them all together, you don't need to declare the input with `[]`. The pipe will receive the list and process it as a whole.

**3. Fixed count (bracket notation `[N]`)**

Use a number in brackets `[N]` to specify that the pipe expects exactly that many items:

```toml
[concept]
Comparison = "A detailed comparison analysis"

[pipe.compare_two_images]
type = "PipeLLM"
description = "Compare exactly two images side by side"
inputs = { images = "Image[2]" }
output = "Comparison"
prompt = """
Compare these two images in detail:

@images

Describe their similarities, differences, and relative strengths.
"""
```

## Practical Use Cases

### Use Case 1: Batch Processing with Variable Input

Process an unknown number of invoices, extracting structured data from each:

```toml
[concept]
InvoiceImage = "An image of an invoice document"
InvoiceData = "Structured invoice information"

[pipe.extract_single_invoice]
type = "PipeLLM"
description = "Extract data from one invoice"
inputs = { invoice_image = "InvoiceImage" }
output = "InvoiceData"
prompt = """
Extract all fields from this invoice:

@invoice_image
"""

[pipe.process_invoice_batch]
type = "PipeSequence"
description = "Process multiple invoices"
inputs = { invoice_images = "InvoiceImage[]" }
output = "InvoiceData[]"
steps = [
    { pipe = "extract_single_invoice", batch_over = "invoice_images", batch_as = "invoice_image", result = "all_invoice_data" }
]
```

### Use Case 2: Generate Fixed Alternatives for Testing

Create exactly 3 subject line variations for A/B testing:

```toml
[concept]
EmailContent = "The body text of an email"
SubjectLine = "An email subject line"

[pipe.generate_subject_lines]
type = "PipeLLM"
description = "Generate 3 subject line options"
inputs = { email_body = "EmailContent" }
output = "SubjectLine[3]"
prompt = """
Email content:
@email_body

Generate $_nb_output compelling subject lines for this email.
Each should use a different persuasion technique.
"""
```

### Use Case 3: Comparative Analysis

Compare exactly two products side by side:

```toml
[concept]
ProductDescription = "A description of a product's features"
Comparison = "A comparative analysis of products"

[pipe.compare_products]
type = "PipeLLM"
description = "Compare two products"
inputs = { products = "ProductDescription[2]" }
output = "Comparison"
prompt = """
Compare these two products:

@products

Provide a balanced comparison of features, benefits, and potential drawbacks.
"""
```

### Use Case 4: Entity Extraction with Unknown Count

Extract all company names mentioned in a document:

```toml
[concept]
Article = "A news or information article"
CompanyName = "The name of a company or organization"

[pipe.extract_companies]
type = "PipeLLM"
description = "Extract all company names from an article"
inputs = { article = "Article" }
output = "CompanyName[]"
prompt = """
Read this article:

@article

Extract all company and organization names mentioned in the article.
Only include entities that are explicitly named.
"""
```

## Best Practices

### When to Use Variable Output (`[]`)

Use variable output multiplicity when:

- The number of outputs depends on the content being analyzed
- You're extracting or identifying items (entities, keywords, issues)
- The count isn't known until after processing
- You want the LLM to use its judgment about completeness

### When to Use Fixed Output (`[N]`)

Use fixed output multiplicity when:

- You need a specific number for downstream processes
- You're generating alternatives for comparison or selection
- External requirements dictate a fixed count
- You want consistent batch sizes for processing

### When to Use Variable Input (Empty Brackets `[]`)

Use variable input multiplicity when:

- The pipe should handle batches of unknown size
- You're aggregating or summarizing multiple items
- The method involves collecting items before processing
- You want maximum flexibility in how the pipe is called

### When to Use Fixed Input (Brackets with Number `[N]`)

Use fixed input multiplicity when:

- The operation inherently requires a specific count (e.g., comparison of 2 items)
- You want to enforce validation at the pipeline level
- The prompt logic depends on an exact number of inputs
- You're preventing misuse by requiring the right number of items

## How Multiplicity Interacts with ListContent

When a pipe produces multiple outputs, Pipelex automatically wraps them in a `ListContent` container. This container maintains the type information:

```python
from pipelex.pipeline.runner import PipelexMTHDSProtocol

# Execute a pipe with multiple outputs
runner = PipelexMTHDSProtocol()
response = await runner.execute(
    pipe_code="extract_line_items",
    inputs={"invoice_text": "Your invoice text here..."}
)
pipe_output = response.pipe_output

# Get the list of line items
line_items_list = pipe_output.main_stuff_as_items(item_type=LineItem)
# Result: [LineItem(...), LineItem(...), LineItem(...)]

# Or work with the ListContent directly
line_items_container = pipe_output.main_stuff_as_list(item_type=LineItem)
# Result: ListContent[LineItem] with additional metadata
```

Similarly, when providing multiple inputs, you can use lists:

```python
from pipelex.pipeline.runner import PipelexMTHDSProtocol
from pipelex.core.stuffs.text_content import TextContent

runner = PipelexMTHDSProtocol()
response = await runner.execute(
    pipe_code="summarize_all_documents",
    inputs={
        "documents": [
            TextContent(text="Document 1 content..."),
            TextContent(text="Document 2 content..."),
            TextContent(text="Document 3 content..."),
        ]
    }
)
pipe_output = response.pipe_output
```

## Summary

Multiplicity in Pipelex gives you precise control over how many items flow through your pipelines:

- **Concepts stay singular** to maintain clean semantics
- **Output multiplicity** (bracket notation in `output` field) controls generation
- **Input multiplicity** (bracket notation in `inputs` definition) enforces expectations
- **Variable multiplicity** (`[]`) for unknown counts
- **Fixed multiplicity** (`[N]`) for exact requirements

By understanding and using multiplicity effectively, you can build pipelines that handle both single items and collections with clarity and type safety.



## Understanding Optionality

# Understanding Optionality

Real methods meet values that may legitimately not exist: a nickname that was never provided, a document section that wasn't found, a branch of a condition that decided to produce nothing. Optionality makes that a first-class, declared part of your method's contract — instead of a crash, a fabricated placeholder, or a silent `null`.

This guide explains the presence markers (`?` and `!`), what happens at run time when a value is absent, and the static validation that proves your method handles every possible absence before it ever runs.

## The Presence Markers

A concept reference in a pipe's `inputs` or `output` can carry a presence marker:

| Declaration | Meaning |
|---|---|
| `notes = "Note"` | **Plain** (default): the pipe needs the value. If it is absent, the pipe is skipped (see [lifting](#the-runtime-trichotomy-skip-run-or-fail)). |
| `notes = "Note?"` | **Optional** (`?`): the pipe runs even when the value is absent and handles absence itself. On an output, the pipe may resolve it as a *recorded absence* instead of a value. |
| `notes = "Note!"` | **Force** (`!`, inputs only): the pipe asserts the value must be present — an absent value fails the run loudly with `OptionalValueAbsentError`, carrying the full provenance of where absence came from. |

Two grammar rules keep the model coherent:

- **Markers never combine with multiplicity.** `Note[]?` is invalid (`optional_marker_invalid`): a plural slot is never absent — when nothing is found, it is the **empty list**. Absence is a singular-value notion.
- **Outputs take only `?`.** A force marker on an output has no meaning (a pipe cannot assert its own output into existence).

Concept definitions, `refines`, and structure-field references never carry markers — presence is a property of a pipe's contract, not of a concept.

## The Runtime Trichotomy: Skip, Run, or Fail

At run time, a value slot resolves to a value **or a recorded absence** — never a silent hole. When a pipe is about to run and one of its inputs is absent, the marker decides what happens:

1. **Plain input absent → the pipe is skipped (lifted).** The pipe never runs; a `skipped` absence is recorded for its output, chaining to the upstream absence that caused the skip. Downstream pipes then see that output as absent and the same rule applies — absence propagates until something handles it.
2. **Optional (`?`) input absent → the pipe runs.** The pipe declared it can handle the absence — for template-rendering pipes that means guarding the reference (see [Guarding templates](#guarding-templates-over-optional-inputs)).
3. **Force (`!`) input absent → the run fails loudly.** `OptionalValueAbsentError` names the variable, the pipe, and the provenance chain back to where absence entered the flow.

A skipped pipe with a **plural** output resolves it as a real empty list (per the plural rule above), with a ledger note for observability — so downstream list consumers keep working.

### Absence records and provenance

Absence is carried as data: an `AbsenceRecord` with the variable name, a `kind` (`declared_absent` — a `?` output resolved absent; `skipped` — the producer was lifted; `not_provided` — the caller omitted an optional method input), a human-readable reason, the producing pipe, and the upstream record it chains to. Provenance is captured when absence is produced, not reconstructed at failure time — so a `!` failure or a run report can always tell you *why* a value is missing.

The invariant everywhere: **a pipe run always resolves its declared output — a value, or a recorded absence.** An absent main output is a successful run; the CLI and the API deliver an explicit absence document (`{"absent": true, ...}`), never an empty result.

## Optional Method Inputs

A `?` on your method's entry-pipe input makes it omittable by the caller: leaving it out of `inputs.json` (or the execute request) records a `not_provided` absence instead of raising the missing-input error. Plain and `!` inputs stay required at the boundary.

```toml
[pipe.brief_flow]
type        = "PipeSequence"
description = "Write a brief; enrich it when a hint is provided"
inputs      = { topic = "Topic", hint = "Hint?" }
output      = "Brief"
steps = [
  { pipe = "enrich_hint", result = "enriched" },
  { pipe = "write_brief", result = "brief" },
]
```

Here a caller may provide only `topic`. The `enrich_hint` step (which consumes `hint` plainly) is skipped, and `write_brief` — which takes the enrichment as `Enriched?` — runs with it absent.

## Guarding Templates Over Optional Inputs

A template that renders an optional input must guard the reference — otherwise validation rejects it (`optional_input_unguarded`). Three guard idioms are recognized:

- **The optional block sigil `@?`** — inserts the input's content when present, renders nothing when absent:

```toml
prompt = """
Write a brief about this topic: $topic

@?enriched
"""
```

- **A presence block** — `{% if enriched %} ... {% endif %}` (any reference inside the block is guarded, including `$enriched` and attribute access).
- **An inline presence conditional** — `{{ enriched.text if enriched else "no enrichment" }}`.

The guard-lint covers every pipe that renders authored templates over its inputs: PipeLLM (prompt and system prompt), PipeCompose (template mode), PipeCondition (expression), PipeSearch, and PipeImgGen.

## Controllers Under Absence

Each controller has a defined behavior when a value inside its flow resolves absent:

- **PipeSequence** — steps consuming an absent value plainly are lifted (skipped), and the absence propagates step to step. The sequence's own declared output must account for it: if the last producing step can be lifted, the sequence output must be declared `?` (otherwise `optional_not_handled`).
- **PipeCondition** — the `continue` outcome resolves the declared output as a **declared-absent** record (memory otherwise unchanged). A condition that can reach `continue` must declare its output `?` (`optional_output_required`). See [PipeCondition](pipe-controllers/PipeCondition.md).
- **PipeParallel** — combining under absence: a `Composite` output omits the absent component (with a ledger note); a structured output absorbs an absent branch into a non-required field as that field's declared default; a **required** field fed by a maybe-absent branch is rejected statically (`optional_branch_required_field`). See [PipeParallel](pipe-controllers/PipeParallel.md).
- **PipeBatch** — absent branch results are **compacted** out of the aggregated list (a list cannot hold a hole). See [PipeBatch](pipe-controllers/PipeBatch.md).

## The Static Safety Net

You never discover an unhandled absence at run time. Validation walks every controller flow and computes, per slot, whether it is guaranteed present or maybe-absent — and requires every possible absence to reach an explicit sink (a `?` boundary, a guard, an absorbing field). The optionality error types:

| Error | Meaning |
|---|---|
| `optional_marker_invalid` | Grammar misuse: a marker combined with multiplicity, or `!` on an output. |
| `optional_not_handled` | A maybe-absent value escapes through a non-optional boundary. The message names the absence origin, the propagation path, and the fixes. |
| `optional_output_required` | A `continue`-reachable PipeCondition without a `?` output. |
| `optional_input_unguarded` | An unguarded template reference to a declared-optional input. |
| `optional_branch_required_field` | A maybe-absent parallel branch feeding a required structure field. |

Two more optionality facts surface on the **valid** report (`pipelex validate`, the agent CLI, and the API):

- **`liftable_pipes`** — every pipe that may be silently skipped at run time, with the slots whose absence triggers the skip and where that absence can come from. Implicit lifting is acceptable *because* it is visible at build time.
- **`warnings`** — advisory lints that never flip the verdict. The first is `optional_force_redundant`: a `!` whose slot is guaranteed present in every analyzed flow — the assertion can never fire, so the marker is dead weight.

## Best Practices

- **Declare `?` at the boundary you mean it.** An optional method input is a statement to your callers; an optional output is a statement to your consumers. Inside the flow, prefer letting plain pipes lift over sprinkling `?` everywhere — the skip semantics usually express "no input, no output" exactly.
- **Guard every optional reference in templates** with `@?` when you just want the content inserted, or `{% if %}` when the surrounding prose changes with presence.
- **Reserve `!` for genuine assertions** — places where your flow analysis says the value could be absent but you know better and want a loud failure rather than a skip. If the lint says it's redundant, remove it.
- **Check `liftable_pipes` when validating**: it is the list of "this may silently not run" — confirm each entry is intended.

## Related Documentation

- [Understanding Multiplicity](understanding-multiplicity.md) — the other axis of a slot's shape; plural absence is the empty list
- [PipeCondition](pipe-controllers/PipeCondition.md) — the `continue` outcome and its migration notes
- [PipeParallel](pipe-controllers/PipeParallel.md) — combining under absence
- [PipeBatch](pipe-controllers/PipeBatch.md) — compaction of absent branch results
- [Execution Graph Tracing](../../under-the-hood/execution-graph-tracing.md) — skipped nodes, `skip_reason`, and optional data edges


## Overview

# Pipe Operators

!!! tip "MTHDS Standard Reference"
    Pipe operators are part of the MTHDS open standard. For the authoritative language specification, see [Pipes & Operators](https://mthds.ai/latest/language/pipes-operators/) on mthds.ai. This page documents Pipelex-specific behavior and usage.

Pipe operators are the fundamental building blocks in Pipelex, representing a single, focused task. They are the "verbs" of your pipeline that perform the actual work.

Each operator specializes in a specific kind of action, from interacting with Large Language Models to executing custom Python code. You combine these operators using [Pipe Controllers](../pipe-controllers/index.md) to create complex methods.

## Core Operators

Here are the primary pipe operators available in Pipelex:

-   [**`PipeLLM`**](./PipeLLM.md): The core operator for all interactions with Large Language Models (LLMs), including text generation, structured data extraction, and vision tasks.
-   [**`PipeStructure`**](./PipeStructure.md): Turns free-form text into structured, typed data via a single LLM call — ideal when the text comes from an upstream extraction, search, or generation step.
-   [**`PipeExtract`**](./PipeExtract.md): Performs Optical Character Recognition (OCR) on images and PDF documents to extract text and embedded images, and fetches and extracts content from web pages.
-   [**`PipeImgGen`**](./PipeImgGen.md): Generates images from a text prompt using models like GPT Image, Flux, or other image generation models.
-   [**`PipeSearch`**](./PipeSearch.md): Searches the web using a configurable search provider and returns structured results with an answer and source citations.
-   [**`PipeFunc`**](./PipeFunc.md): An escape hatch that allows you to execute any custom Python function, giving you maximum flexibility.
-   [**`PipeCompose`**](./PipeCompose.md): Composes outputs deterministically from working memory — renders Jinja2 templates for formatted reports or complex prompts, or constructs structured objects by mapping fields from inputs, without an LLM.

## Overview

Pipelex provides the following pipe operators:

- `PipeLLM`: For LLM-based text generation and processing
- `PipeCompose`: For composing text (Jinja2 templates) or structured objects (construct mode) from working memory data
- `PipeExtract`: For optical character recognition and document processing
- `PipeFunc`: For executing custom functions
- `PipeImgGen`: For AI-powered image generation
- `PipeSearch`: For web search with structured results
- `PipeStructure`: For turning free-form text into structured data

## PipeLLM

Core operator for LLM-based text generation and processing.

### Key Features

- Text generation
- Structured output generation
- Multiple output modes
- System prompt customization
- LLM configuration

## PipeCompose

Composes outputs deterministically from working memory, without an LLM.

### Key Features

- Template mode: Jinja2 rendering to Text or Html outputs
- Construct mode: structured objects assembled by field mapping
- Field methods: references, templates, fixed values, nested constructs
- Native wrapper unwrapping when copying whole inputs into native fields

## PipeExtract

Processes images and PDFs using Optical Character Recognition, and fetches and extracts content from web pages.

### Key Features

- PDF processing
- Image processing
- Text extraction
- Image extraction
- Page view generation

## PipeFunc

Executes custom functions within the pipeline.

### Key Features

- Custom function execution
- Working memory integration
- Multiple output types
- Function registry integration

## PipeImgGen

Generates and manipulates images.

### Key Features

- Image generation
- Quality control
- Multiple output formats
- Batch processing
- Parameter customization

## PipeSearch

Searches the web and returns structured results with sources.

### Key Features

- Web search via configurable providers
- Structured results with answer and source citations
- Dynamic prompt templates with `$variable` syntax
- Standard and deep search models


## PipeLLM

# PipeLLM

`PipeLLM` is the core operator in Pipelex for leveraging Large Language Models (LLMs). It can be used for a wide range of tasks, including text generation, summarization, classification, and structured data extraction.

## How it works

At its core, `PipeLLM` constructs a detailed prompt from various inputs and templates, sends it to a specified LLM, and processes the output. It can produce simple text or complex structured data (in the form of Pydantic models).

For structured outputs, you have two options:

1. **Direct (default)** — `PipeLLM` asks the LLM to fill the target Pydantic schema in a single call. Fast and cheap; relies on the model's structured-output quality.
2. **Preliminary text** — set `structuring_method = "preliminary_text"`. Pipelex rewrites the pipe at load time into a `PipeSequence` of two pipes: a `PipeLLM` that produces free-form text, then a [`PipeStructure`](./PipeStructure.md) that turns that text into the target concept. Two LLM calls per invocation, but the prose stage is unconstrained by JSON. See [Structuring Method (preliminary_text)](#structuring-method-preliminary-text) below.

If you already have text from elsewhere (a PDF extraction, a search result, an upstream pipe), call [`PipeStructure`](./PipeStructure.md) directly — there's no need to wrap a `PipeLLM` around it.

## Working with Images (Vision Language Models)

`PipeLLM` supports Vision Language Models (VLMs) that can process both text and images. To use images in your prompts:

### Basic Image Input

Images must be declared in the `inputs` section of your pipe definition. The image will be automatically passed to the VLM along with your text prompt.

```toml
[pipe.describe_image]
type = "PipeLLM"
description = "Describe an image"
inputs = { image = "Image" }
output = "VisualDescription"
prompt = """
Describe the provided image in great detail: $image
"""
```

!!! important "Image Variable Tagging"
    It is necessary to tag image variables with `@image` or `$image`, just like with regular objects. This works in both `prompt` and `system_prompt`.

**Flexible Image Inputs**

You can use any concept that refines `Image` as an input, and choose descriptive variable names that fit your use case:

```toml
[pipe.analyze_wedding]
type = "PipeLLM"
description = "Analyze wedding photo"
inputs = { wedding_photo = "Photo" }
output = "PhotoAnalysis"
prompt = """
Analyze this wedding photo and describe the key moments captured: $wedding_photo
"""
```

### Images as Sub-attributes of Structured Content

When working with structured content that contains image fields (like `PageContent` which has a `page_view` field), you need to specify the full path to the image attribute in the `inputs` section:

```toml
[pipe.analyze_page_view]
type = "PipeLLM"
description = "Analyze the visual layout of a page"
inputs = { "page_content.page_view" = "Image" }
output = "LayoutAnalysis"
prompt = """
Analyze the visual layout and design elements of this page: $page_content.page_view
Focus on typography, spacing, and overall composition.
"""
```

In this example:

- `page_content` is the input variable containing a `PageContent` object
- `page_view` is the `ImageContent` field within the `PageContent` structure
- The dot notation `page_content.page_view` tells Pipelex to extract the image from that specific field

### Multiple Images

You can include multiple images in a single prompt by listing them in the inputs:

```toml
[pipe.compare_images]
type = "PipeLLM"
description = "Compare two images"
inputs = { 
    first_image = "Image",
    second_image = "Image"
}
output = "ImageComparison"
prompt = """
Compare these two images and describe their similarities and differences: $first_image and $second_image
"""
```

### Combining Text and Image Inputs

You can mix any stuff and image inputs in the same pipe:

```toml
[pipe.analyze_document_with_context]
type = "PipeLLM"
description = "Analyze a document page with additional context"
inputs = { 
    context = "Text",
    document.page_view = "Image"
}
output = "DocumentAnalysis"
prompt = """
Given this context: $context

Analyze the document page shown in the image and explain how it relates to the provided context: $document.page_view
"""
```

## Working with Documents

`PipeLLM` supports including documents (PDFs, etc.) in prompts. Documents are passed to the LLM along with your text prompt, allowing the model to analyze their content.

### Basic Document Input

Documents must be declared in the `inputs` section of your pipe definition. The document will be automatically passed to the LLM along with your text prompt.

```toml
[pipe.summarize_document]
type = "PipeLLM"
description = "Summarize a document"
inputs = { document = "Document" }
output = "DocumentSummary"
prompt = """
Summarize the key points from this document:

@document
"""
```

!!! important "Document Variable Tagging"
    It is necessary to tag document variables with `@document` or `$document`, just like with regular objects. This works in both `prompt` and `system_prompt`.

**Flexible Document Inputs**

You can use any concept that refines `Document` as an input, and choose descriptive variable names that fit your use case:

```toml
[concept.FinancialReport]
description = "A financial report"
refines = "Document"

[pipe.analyze_report]
type = "PipeLLM"
description = "Analyze financial report"
inputs = { financial_report = "FinancialReport" }
output = "ReportAnalysis"
prompt = """
Analyze this financial report and extract the key metrics: $financial_report
"""
```

### Multiple Documents

You can include multiple documents in a single prompt by listing them in the inputs:

```toml
[pipe.compare_documents]
type = "PipeLLM"
description = "Compare two documents"
inputs = { first_doc = "Document", second_doc = "Document" }
output = "DocumentComparison"
prompt = """
Compare these two documents and describe their similarities and differences: $first_doc and $second_doc
"""
```

For a variable number of documents, use the list syntax:

```toml
[pipe.merge_documents]
type = "PipeLLM"
description = "Merge multiple documents"
inputs = { documents = "Document[]" }
output = "MergedContent"
prompt = """
Combine the information from all these documents into a single coherent summary: $documents
"""
```

### Combining Text, Images, and Documents

You can mix text, image, and document inputs in the same pipe:

```toml
[pipe.analyze_with_context]
type = "PipeLLM"
description = "Analyze a document with context"
inputs = {
    context = "Text",
    screenshot = "Image",
    reference_doc = "Document"
}
output = "ContextualAnalysis"
prompt = """
Given this context: $context

And this screenshot for reference: $screenshot

Analyze the document and explain how it relates to the context: $reference_doc
"""
```

## Configuration

`PipeLLM` is configured in your pipeline's `.mthds` file.

### MTHDS Parameters

| Parameter                   | Type                | Description                                                                                                                                                                  | Required |
| --------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `type`                      | string              | The type of the pipe: `PipeLLM`                                                                          | Yes      |
| `description`               | string              | A description of the LLM operation.                                                                           | Yes      |
| `inputs`                    | dictionary          | The input concept(s) for the LLM operation, as a dictionary mapping input names to concept codes. For images within structured content, use dot notation (e.g., `"page.image_argurment"`)
| `output`                    | string              | The output concept produced by the LLM operation with multiplicity notation using brackets (e.g., `"Text"`, `"Text[]"`, `"Text[3]"`).                                                | Yes      |
| `model`                       | string or table     | Specifies the LLM choice by name, setting, or preset to use.              | No       |
| `model_to_structure`                       | string or table     | LLM choice used whenever this `PipeLLM` produces a structured output. Applies both to direct structured generation and to the structuring step when `structuring_method = "preliminary_text"` is set. | No       |
| `system_prompt`             | string              | A system-level prompt to guide the LLM's behavior (e.g., "You are a helpful assistant"). Supports the same variable syntax as `prompt`, including image and document references.  | No       |
| `prompt`           | string              | A template for the user prompt. Use `$` for inline variables (e.g., `$topic`). Use `@input_name` as a block sigil on its own line to insert the content of an entire input — inline `@var` raises `TemplateSigilSyntaxError` when `var` collides with a declared input. For a declared-optional input (`?`), use `@?input_name` — it inserts the content when present and renders nothing when absent, and counts as a guard for the optionality lint (a bare `$var`/`@var` reference to an optional input is rejected as `optional_input_unguarded`; see [Understanding Optionality](../understanding-optionality.md)). Image and document variables follow the same rules. Escape with `@@` or `$$` to emit a literal `@` or `$` (e.g., `@@font-face` inside a `<style>` block). | No       |
| `structuring_method`        | string              | `"direct"` (default) generates the structured output in one LLM call. `"preliminary_text"` expands the pipe at load time into a `PipeLLM` (text) + [`PipeStructure`](./PipeStructure.md) sequence — two LLM calls. Cannot be combined with a `Text` output. | No       |

### LLM Setting Fields

When `model` is specified as a table (inline LLM setting), it accepts the following fields:

| Field | Type | Description |
|-------|------|-------------|
| `model` | string | Model name or alias (e.g., `"claude-4.5-sonnet"`, `"@default-premium"`) |
| `temperature` | float | Sampling temperature (0.0 – 1.0) |
| `max_tokens` | integer | Maximum output tokens |
| `reasoning_effort` | string | Reasoning depth: `"none"`, `"minimal"`, `"low"`, `"medium"`, `"high"`, `"xhigh"`, `"max"`. Not supported for structured generation. |
| `reasoning_budget` | integer | Explicit token budget for reasoning. Mutually exclusive with `reasoning_effort`. Supported by Anthropic and Google only. |
| `description` | string | Human-readable description (for presets) |

!!! tip "More on Reasoning"
    For provider-specific behavior and model-specific examples, see [Reasoning Controls](../../../under-the-hood/reasoning-controls.md).

### Output Multiplicity

Specify output multiplicity using bracket notation in the `output` field:

- **Single output** (default): `output = "Text"` - generates exactly one item
- **Variable output**: `output = "Text[]"` - lets the LLM decide how many items to generate
- **Fixed output**: `output = "Text[5]"` - generates exactly 5 items

Examples:

```toml
# Single output (default)
output = "Summary"

# Variable - extract all keywords found
output = "Keyword[]"

# Fixed - generate exactly 3 alternatives
output = "Headline[3]"
```

!!! info "Learn More About Multiplicity"
    For a comprehensive guide on output multiplicity, input multiplicity, and the philosophy behind how Pipelex handles single items versus collections, see [Understanding Multiplicity](../understanding-multiplicity.md).

## Structuring Method (preliminary_text) { #structuring-method-preliminary-text }

By default, `PipeLLM` asks the LLM to produce the target structured output in a single call. For some tasks — especially complex extractions from images or documents where the model thinks better in prose than in JSON — you can opt into a two-stage approach:

```toml
[pipe.review_restaurant]
type = "PipeLLM"
description = "Write a structured review of a restaurant from a transcript"
inputs = { transcript = "Text" }
output = "RestaurantReview"
structuring_method = "preliminary_text"
prompt = """
Write a thorough review of this restaurant based on the transcript:

@transcript
"""
```

### What happens at load time

When Pipelex parses a `.mthds` file with `structuring_method = "preliminary_text"`, it rewrites the pipe **before any execution** into a [`PipeSequence`](../pipe-controllers/PipeSequence.md):

1. **Step 1 — `<pipe_code>__draft_text`**: a `PipeLLM` that inherits your original `inputs`, `system_prompt`, `prompt`, and `model`. Its output is always a single `Text`. This is the "write the prose" call.
2. **Step 2 — `<pipe_code>__structure`**: a [`PipeStructure`](./PipeStructure.md) that takes the draft text and produces your declared output (preserving multiplicity — `Foo`, `Foo[]`, or `Foo[N]`). It uses `model_to_structure` if set, otherwise the default LLM choice for object generation.

The wrapping `PipeSequence` keeps your original `pipe_code`, so anything that calls or references your pipe (other pipes, `main_pipe`, the run API) keeps working unchanged.

### Trade-offs

- **Two LLM calls per invocation** instead of one — slower and more expensive.
- **Better quality on complex schemas**, especially with vision models: the prose stage isn't constrained by the JSON schema, then a focused structuring call extracts the typed result.
- **Independent model choice for the structuring step** via `model_to_structure` — pair an expensive vision model for step 1 with a cheap fast model for step 2.
- **Output cannot be `Text`** — there's nothing to structure. Pipelex rejects this at load time.

### When to choose what

- **Stick with `direct`** for most structured outputs — modern models handle them well and one call is always cheaper.
- **Switch to `preliminary_text`** when you're extracting from images or PDFs and a single-call structured output keeps missing fields, or when your schema is deep and the model produces better content in prose first.
- **Use [`PipeStructure`](./PipeStructure.md) directly** when the text already exists (extracted, searched, generated upstream).

## Examples

### Simple Text Generation Example

This pipe takes no input and writes a poem.

```toml
[pipe.write_poem]
type = "PipeLLM"
description = "Write a short poem"
output = "Text"
model = "$writing-creative"
prompt = """
Write a four-line poem about pipes.
"""
```

### Text-to-Text Example

This pipe summarizes an input text, using a `prompt` to inject the input.

```toml
[pipe.summarize_text]
type = "PipeLLM"
description = "Summarize a text"
inputs = { text = "TextToSummarize" }
output = "TextSummary"
prompt = """
Please provide a concise summary of the following text:

@text

The summary should be no longer than 3 sentences.
"""
```

### Vision (VLM) Example

This pipe takes an image of a table and uses a VLM to extract the content as an HTML table.

```toml
[pipe.extract_table_from_image]
type = "PipeLLM"
description = "Extract table data from an image"
inputs = { image = "TableScreenshot" }
output = "TableData"
prompt = """
Extract the table data from this image and format it as a structured table: $image
"""
```

### Structured Data Extraction Example

This pipe extracts a list of `Expense` items from a block of text.

```toml
[concept.Expense]
structure = "Expense" # Assumes a Pydantic model 'Expense' is defined

[pipe.process_expense_report]
type = "PipeLLM"
description = "Process an expense report"
inputs = { report = "ExpenseReport" }
output = "ProcessedExpenseReport"
prompt = """
Analyze this expense report and extract the following information:
- Total amount
- Date
- Vendor
- Category
- Line items

@report
"""
```

In this example, `Pipelex` will instruct the LLM to return a list of objects that conform to the `Expense` structure.

### Reasoning Example

This pipe uses extended reasoning to analyze a complex topic in depth:

```toml
[pipe.deep_analysis]
type = "PipeLLM"
description = "Analyze a complex topic with extended reasoning"
inputs = { topic = "Text" }
output = "Analysis"
model = { model = "claude-4.5-sonnet", temperature = 0.1, reasoning_effort = "high" }
prompt = """
Analyze the following topic in depth, considering multiple perspectives:

@topic
"""
```

You can also reference a reasoning preset:

```toml
[pipe.deep_analysis]
type = "PipeLLM"
description = "Analyze a complex topic with extended reasoning"
inputs = { topic = "Text" }
output = "Analysis"
model = "$deep-analysis"
prompt = """
Analyze the following topic in depth, considering multiple perspectives:

@topic
"""
```

## Related Documentation

- [Hello World Example](../../../cookbook/hello-world.md) - Simple introductory example using PipeLLM
- [Invoice Extraction Example](../../../cookbook/extract-invoice.md) - Complete invoice processing pipeline
- [Write Tweet Example](../../../cookbook/write-tweet.md) - Multi-step tweet generation workflow
- [Table Extraction Example](../../../cookbook/extract-table.md) - Extract and correct tables from images
- [Gantt Extraction Example](../../../cookbook/extract-gantt.md) - Extract Gantt chart data from documents

## PipeStructure

# PipeStructure

`PipeStructure` is the operator that turns text into structured data. It takes a single `Text`-compatible input, asks an LLM to extract the requested structure, and produces a typed object (or list of objects) matching your output concept.

Use `PipeStructure` whenever the source of the text is not the LLM call itself — for example when text comes from a PDF extraction, a web search result, a user message, or an upstream `PipeLLM` step that intentionally produced free-form prose.

## How it works

1. The operator reads the single declared input from the working memory and renders the built-in `structuring_prompt` template against it.
2. It appends a JSON-schema description of the target concept (the same helper `PipeLLM` uses for object generation) so the model knows which fields to fill.
3. It picks an LLM setting using the same precedence as `PipeLLM` for object generation: explicit `model` on the pipe → `llm_choice_overrides.for_object` → `llm_choice_defaults.for_object`.
4. It calls `make_object` (or `make_object_list` when the output multiplicity is a list) and emits the result as the pipe's output.

`PipeStructure` does not accept user-controlled prompt templates. The structuring prompt lives in `pipelex.toml` under `[cogt.llm_config.generic_templates] structuring_prompt`; the only variable rendered into it is `text`, fed automatically from the declared input.

## Configuration

`PipeStructure` is configured in your pipeline's `.mthds` file.

### MTHDS Parameters

| Parameter | Type | Description | Required |
| --- | --- | --- | --- |
| `type` | string | The type of the pipe: `PipeStructure` | Yes |
| `description` | string | A description of what this structuring step produces. | Yes |
| `inputs` | dictionary | Exactly one input. Its concept must be `Text` or refine `Text` (e.g. `Document` is not allowed; a domain concept that `refines = "Text"` is). | Yes |
| `output` | string | The target structured concept, with optional multiplicity (`Foo`, `Foo[]`, `Foo[3]`). Cannot be `Text` or `Text`-compatible. | Yes |
| `model` | string or table | The LLM choice used for the structuring call. Defaults to the `for_object` LLM choice (override → default). | No |

### LLM Setting Fields

When `model` is given as a table, it accepts the same fields as `PipeLLM`'s inline LLM setting (`model`, `temperature`, `max_tokens`, `reasoning_effort`, `reasoning_budget`, `description`). See [PipeLLM › LLM Setting Fields](./PipeLLM.md#llm-setting-fields).

!!! note "No images, no documents"
    `PipeStructure` is intentionally narrow: it takes one `Text` input. To structure an image or a PDF page, run an upstream extraction step (e.g. `PipeExtract` or a vision `PipeLLM`) and feed its text output into `PipeStructure`.

### Output Multiplicity

Use bracket notation in the `output` field to control how many items the LLM should return:

- `output = "Review"` — exactly one item.
- `output = "Review[]"` — the model decides how many.
- `output = "Review[3]"` — exactly three.

See [Understanding Multiplicity](../understanding-multiplicity.md) for the full picture.

## Examples

### Structure free-form text into a typed object

```toml
[concept.RestaurantReview]
description = "A structured restaurant review extracted from prose"

[concept.RestaurantReview.structure]
restaurant_name = { type = "text", description = "Name of the restaurant" }
overall_rating  = { type = "integer", description = "Overall rating from 1 to 5" }
highlights      = { type = "list", item_type = "text", description = "Standout positives" }
complaints      = { type = "list", item_type = "text", description = "Issues mentioned" }

[pipe.structure_review]
type = "PipeStructure"
description = "Turn a free-form review into a RestaurantReview"
inputs = { review_text = "Text" }
output = "RestaurantReview"
```

### Structure a list of objects

```toml
[pipe.structure_review_batch]
type = "PipeStructure"
description = "Extract one or more reviews from a transcript"
inputs = { transcript = "Text" }
output = "RestaurantReview[]"
```

### After a document extraction step

`PipeStructure` shines as the second stage of a sequence whose first stage produces text from a non-text source.

```toml
[pipe.invoice_to_record]
type = "PipeSequence"
description = "Read an invoice PDF and turn it into an InvoiceRecord"
inputs = { invoice_pdf = "Document" }
output = "InvoiceRecord"
steps = [
  { pipe = "extract_invoice_text", result = "invoice_text" },
  { pipe = "structure_invoice",    result = "invoice_record" },
]

[pipe.extract_invoice_text]
type = "PipeLLM"
description = "Read the invoice PDF and produce a faithful textual transcript"
inputs = { invoice_pdf = "Document" }
output = "Text"
prompt = """
Read this invoice and produce a faithful textual transcript of every line item, total, and metadata:
@invoice_pdf
"""

[pipe.structure_invoice]
type = "PipeStructure"
description = "Turn the invoice transcript into an InvoiceRecord"
inputs = { invoice_text = "Text" }
output = "InvoiceRecord"
```

### Pick a different model for the structuring step

```toml
[pipe.structure_review_premium]
type = "PipeStructure"
description = "Use a stronger model for tricky structurings"
inputs = { review_text = "Text" }
output = "RestaurantReview"
model = "@default-premium"
```

## When to reach for `PipeStructure`

- You already have text and you want it as a typed object.
- You want a deterministic boundary between "writing prose" and "filling a schema" — for example to make the prose call cheaper, or to swap the structuring model independently.
- You want the same structuring step reused across several upstream sources (extraction, search, generation).

For the common case where the text is freshly produced by an LLM and you want a single declaration, see [`structuring_method = "preliminary_text"` on PipeLLM](./PipeLLM.md#structuring-method-preliminary-text), which expands at build time into a `PipeSequence` of `PipeLLM` (text) + `PipeStructure`.

## Related Documentation

- [PipeLLM](./PipeLLM.md) — direct structured generation in a single call, plus the `preliminary_text` shorthand.
- [PipeExtract](./PipeExtract.md) — produces text from PDFs and images, a natural upstream step.
- [Understanding Multiplicity](../understanding-multiplicity.md) — single vs list outputs.


## PipeExtract

# PipeExtract

The `PipeExtract` operator extracts structured content from documents. For PDFs and images, it performs OCR to extract text, embedded images, and full-page renderings. For web pages, it fetches and extracts page content.

## How it works

`PipeExtract` takes a single input, which must be a `Document` or an `Image` (or a concept that refines one of them). The document URL can be a file path, a storage URL, or a web page URL. It processes the input and produces a list of `PageContent` objects. Each `PageContent` object encapsulates all the information extracted from one page.

The output is always a list. If the input is a single image, the output still contains one `Page` item.

### The `PageContent` Structure

The `PageContent` object has the following structure:

-   `text_and_images`: Contains the main extracted content.
    -   `text`: The recognized text from the page as a `TextContent`.
    -   `images`: A list of any images found embedded within the page.
-   `page_view`: An `ImageContent` object representing a full visual rendering of the page.

## Configuration

`PipeExtract` is configured in your pipeline's `.mthds` file.

### Extraction Models and Backend System

PipeExtract uses the unified inference backend system to manage extraction models. This means you can:

- Use different extraction providers (Mistral OCR, Linkup Fetch, local PDF extraction via internal backend, etc.)
- Configure extraction models through the same backend system as LLMs and image generation models
- Use extraction presets for consistent configurations across your pipelines
- Route extraction requests to different backends based on your routing profile

Common extraction model handles:

- `mistral-ocr`: Mistral's OCR model for high-quality text and image extraction
- `pypdfium2-extract-pdf`: Local PDF text extraction (no API calls required)
- `linkup-fetch`: Web page content extraction from URLs

Common model aliases:

- `@default-extract-web-page`: Default for web page extraction (Linkup Fetch)
- `@default-extract-document`: Default for document extraction (Azure Document Intelligence)
- `@default-text-from-pdf`: Fast local PDF text extraction (pypdfium2)

Extraction presets are defined in your model deck configuration and can include parameters like `max_nb_images` and `image_min_size`.

### MTHDS Parameters

| Parameter                   | Type    | Description                                                                                                                              | Required |
| --------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `type`                      | string  | The type of the pipe: `PipeExtract`                                                                          | Yes      |
| `description`               | string  | A description of the extraction operation.                                                                   | Yes      |
| `inputs`                    | Fixed  | The value must be of concept `Document` or `Image` (or a concept that refines one of them). For web page extraction, the Document holds a web URL.                                                     | Yes       |
| `output`                    | string  | The output concept produced by the extraction operation. Use `Page[]`.                                  | Yes      |
| `max_page_images`           | integer or null | Maximum number of images to extract from pages: `null` (or omit) for unlimited, `0` for no images, or a positive integer to limit. Defaults to the value in your Extract model preset. | No       |
| `page_views`                | boolean | If `true`, a high-fidelity image of each page will be included in the `page_view` field. Defaults to `false`.                              | No       |
| `page_views_dpi`            | integer | The resolution (in Dots Per Inch) for the generated page views when processing a PDF. Defaults to `72`.                                 | No       |
| `page_image_captions`       | boolean | If `true`, the OCR service may attempt to generate captions for the images found. *Note: This feature depends on the OCR provider.*        | No       |
| `render_js`                 | boolean | For web-page extraction: if `true`, the extraction backend renders JavaScript before fetching the page content. Defaults to `false`. *Note: Only honored by backends that support headless rendering.* | No       |
| `include_raw_html`          | boolean | For web-page extraction: if `true`, the extracted `Page` populates its `raw_html` field with the fetched page's HTML. Defaults to `false`. | No       |
| `model`                       | string  | The Extract model choice by name, setting, or preset to use (e.g., `"mistral-document-ai-2505"`, `"@default-extract-document"`). Defaults to the model specified in the global config. | No       |

### Example: Processing a PDF

This example defines a pipe that takes a PDF, extracts text and full-page images, and outputs them as a list of pages.

```toml
[concept]
ScannedDocument = "A document that has been scanned as a PDF"

[concept.ExtractedPages]
description = "A list of pages extracted from a document by OCR"
refines = "Page"

[pipe.extract_text_from_document]
type = "PipeExtract"
description = "Extract text from a scanned document"
inputs = { document = "ScannedDocument" }
output = "Page[]"
page_views = true
page_views_dpi = 200
```

The output of `PipeExtract` must be `Page[]`.

To use this pipe, first load a PDF into the `ScannedDocument` concept. After the pipe runs, the output contains a list of `PageContent` objects, where each object has the extracted text and a 200 DPI image of the corresponding page.

### Example: Extracting a Web Page

This example defines a pipe that extracts content from a web page URL.

```toml
[pipe.extract_web_article]
type        = "PipeExtract"
description = "Extract content from a web page"
inputs      = { article_url = "Document" }
output      = "Page[]"
model       = "@default-extract-web-page"
```

Pass a web URL as the `article_url` input when running the pipe. PipeExtract fetches the page and extracts its content into `Page` objects, following the same pattern as document extraction.

## Related Documentation

- [Generic Document Extraction Example](../../../cookbook/extract-generic.md) - Extract markdown from complex PDFs using vision
- [Invoice Extraction Example](../../../cookbook/extract-invoice.md) - Complete invoice processing pipeline
- [Document Extraction Feature](../../../features/document-extraction.md) - Overview of document extraction capabilities


## PipeImgGen

# PipeImgGen

The `PipeImgGen` operator is used to generate images from a text prompt using a specified image generation model.

## How it works

`PipeImgGen` takes a required `prompt` **string template** (plus an optional `negative_prompt`) and a set of parameters, then calls an underlying image generation service (like GPT Image or Flux) to create one or more images. It does **not** consume a dedicated "prompt concept" — the prompt is a template you write directly on the pipe.

The variables you declare in `inputs` are **injected into that template** at run time:

- **Text inputs** (concept `Text`, or anything refining it) are interpolated into the prompt string — reference them with `$var` (or `@var` / `{{ var }}`).
- **Image inputs** (concept `Image`, single or a list) are referenced in the prompt the same way. Each referenced image is pulled from working memory, replaced in the text by an `[Image N]` placeholder token, and passed alongside the rendered text to the generator. This is the **same vision pattern** used for image inputs to `PipeLLM`, and it is what enables image-to-image, reference-image, and image-editing generation (e.g. Flux / GPT Image with reference images), bounded by the model's `max_prompt_images`.

The pipe can be configured to generate a single image or a list of images. Its only concept constraints are that declared image inputs must be `Image`-compatible and the `output` must be `Image`-compatible.

## Configuration

`PipeImgGen` is configured in your pipeline's `.mthds` file.

### The `prompt` Field is Required

**IMPORTANT:** `PipeImgGen` requires an explicit `prompt` field. When using dynamic inputs, the `prompt` field must reference the input variable using the `$` prefix.

**Wrong - Missing required `prompt` field:**

```toml
[pipe.render_slide]
type = "PipeImgGen"
inputs = { img_prompt = "Text" }
output = "Image"
model = "$gen-image"
# ERROR: Missing required field 'prompt'
```

**Correct - `prompt` field references the input:**

```toml
[pipe.render_slide]
type = "PipeImgGen"
inputs = { img_prompt = "Text" }
output = "Image"
prompt = "$img_prompt"
model = "$gen-image"
```

### Image Generation Models and Backend System

PipeImgGen uses the unified inference backend system to manage image generation models. This means you can:

- Use different image generation providers (OpenAI GPT Image, FAL models like Flux, etc.)
- Configure image generation models through the same backend system as LLMs and OCR models
- Use image generation presets for consistent configurations across your pipelines
- Route image generation requests to different backends based on your routing profile

Common image generation model handles:

- `default-general`: General-purpose image generation model (alias for nano-banana)
- `default-premium`: Premium image generation model (alias for nano-banana-2)
- `default-small`: Small, fast image generation model (alias for gpt-image-1-mini)
- `best-gpt`: Best OpenAI image generation model (alias for gpt-image-2)
- `best-gemini`: Best Gemini image generation model (alias for nano-banana-2)

Image generation presets are defined in your model deck configuration and can include parameters like `quality`, `guidance_scale`, and `safety_tolerance`.

### MTHDS Parameters

| Parameter               | Type            | Description                                                                                                                   | Required |
| ----------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------- |
| `type`                  | string          | The type of the pipe: `PipeImgGen`                                                                          | Yes      |
| `description`           | string          | A description of the image generation operation.                                                                           | Yes      |
| `inputs`                | dictionary      | The input concept(s) for the image generation operation, as a dictionary mapping input names to concept codes. Required when the prompt references variables. | No       |
| `output`                | string          | The output concept produced by the image generation operation. Use bracket notation for multiple images: `"Image[3]"` generates exactly 3 images.                                                | Yes      |
| `model`           | string          | The choice of image generation model name, setting, or preset to use (e.g., `"gpt-image-1"`). Defaults to the model specified in the global config.    | No       |
| `prompt`                | string          | The image generation prompt. Can be a static string or reference input variables using `$` prefix (e.g., `"$description"` or `"A sketch of: $subject"`). | Yes      |
| `negative_prompt`       | string          | Optional negative prompt when supported by the selected model or provider.                                                    | No       |
| `aspect_ratio`          | string          | The desired aspect ratio of the image. Valid values: `"square"`, `"landscape_4_3"`, `"landscape_3_2"`, `"landscape_16_9"`, `"landscape_21_9"`, `"landscape_4_1"`, `"landscape_8_1"`, `"portrait_3_4"`, `"portrait_2_3"`, `"portrait_9_16"`, `"portrait_9_21"`, `"portrait_1_4"`, `"portrait_1_8"`. The banner ratios (`landscape_4_1`, `landscape_8_1`, `portrait_1_4`, `portrait_1_8`) are only supported by Gemini 3.1 image models.                                                              | No       |
| `size`                  | string          | Portable size intent: either a **size tier** (`"1k"`, `"2k"`, `"4k"`) or an **exact pixel size** like `"2048x1152"`. A tier composes with `aspect_ratio`; an exact size implies its own ratio, so setting both an exact `size` and `aspect_ratio` is a validation error. When unset, the model's provider default applies. See [Image size and portability](#image-size-and-portability). | No       |
| `seed`                  | integer or "auto" | A seed for the random number generator to ensure reproducibility. `"auto"` uses a random seed.                                | No       |
| `background`            | string          | Optional background setting when supported by the selected provider.                                                          | No       |
| `output_format`         | string          | Optional output image format when supported by the selected provider.                                                         | No       |
| `is_raw`                | boolean         | Request a raw generation mode when supported by the selected provider.                                                        | No       |

### Image size and portability

The `size` parameter carries a size intent that stays portable when you switch the pipe's `model` between providers. It accepts two forms:

- **A size tier** (`"1k"`, `"2k"`, `"4k"`): "produce this pixel class at my chosen `aspect_ratio`". A tier promises a pixel *class*, not identical dimensions across providers — `size = "2k"` at 16:9 yields 2752×1536 on `nano-banana-2` (Google's published grid) and 3072×1728 on `gpt-image-2` (computed from OpenAI's constraints). Both are 2K-class images; that is the abstraction. The `"0.5k"` tier token is reserved and currently rejected by every model.
- **An exact pixel size** (`"2048x1152"`): "produce exactly these pixels". Deterministic everywhere it runs. Because an exact size implies its own ratio, combining it with `aspect_ratio` is a blueprint validation error. On tier-grid models (Gemini), an exact size runs only if it exactly matches a cell of the model's published grid — a near-miss is a validation error suggesting the nearest valid cells, never a silent snap.

When `size` is unset, no size intent is sent: Gemini-routed jobs omit the `image_size` parameter entirely, and OpenAI models fall back to their fixed 1K-class preset for the chosen aspect ratio. Either way you stay at the provider's default resolution and cost — Pipelex never silently upgrades.

Per-model support:

| Model | Size tiers | Exact size |
| --- | --- | --- |
| `nano-banana` (Gemini 2.5 Flash Image) | `1k` | matching grid cells only |
| `nano-banana-pro` (Gemini 3 Pro Image) | `1k`, `2k`, `4k` | matching grid cells only |
| `nano-banana-2` (Gemini 3.1 Flash Image) | `1k`, `2k`, `4k` (all ratios incl. banners) | matching grid cells only |
| `nano-banana-2-lite` (Gemini 3.1 Flash Lite) | `1k` | matching grid cells only |
| `gpt-image-2` | `1k`, `2k` | arbitrary within OpenAI's constraints (multiples of 16, ≤ 3:1 ratio, 0.65–8.29 MP) |
| `gpt-image-1` / `-1-mini` / `-1.5` (legacy) | `1k` | the fixed OpenAI sizes only |
| Flux / Flux 1.1 Ultra / SDXL / Qwen | `1k` (no-op) | not supported |

An unsatisfiable request is a **hard validation error**, never a warn-and-ignore: it fails at blueprint-load time when the pipe's `model` resolves to a concrete model with rules, or at runtime before any API call when the choice is a preset or alias resolved later. Worked examples of switching `model` on an otherwise unchanged pipe:

- `nano-banana-2` + `aspect_ratio = "landscape_16_9"` + `size = "2k"` → switch to `gpt-image-2`: runs, 2K-class dimensions. Switch to `flux-pro/v1.1`: validation error (Flux cannot produce the 2K class).
- `gpt-image-2` + `size = "2048x2048"` → switch to `nano-banana-2`: runs (exact grid hit: square at 2K). With `size = "2000x2000"` instead: validation error suggesting `2048x2048`.
- SDXL + `size = "1k"` → runs on every model (worst case as a no-op).
- `nano-banana-2` + `size = "4k"` → switch to `gpt-image-2`: validation error (4K exceeds its pixel cap) — an honest refusal, not a silent downgrade.

!!! note "Cost"

    Size tiers change the bill: providers charge proportionally more for larger outputs (Google's output tokens scale with the tier, OpenAI's prices scale with the size), so `2k` and `4k` generations cost several times a `1k` generation. Leaving `size` unset keeps the model at its provider-default size and cost.

### Example: Generating a single image from a static prompt

This pipe generates one image of a futuristic car without requiring any input.

```toml
[pipe.generate_car_image]
type = "PipeImgGen"
description = "Generate a futuristic car image"
output = "Image"
prompt = "A sleek, futuristic sports car driving on a neon-lit highway at night."
model = "$gen-image"
aspect_ratio = "landscape_16_9"
```

### Example: Generating an image from a dynamic prompt

This pipe takes a text prompt as input and generates an image.

```toml
[pipe.generate_from_description]
type = "PipeImgGen"
description = "Generate an image from a description"
inputs = { description = "Text" }
output = "Image"
prompt = "$description"
model = "$gen-image"
```

### Example: Dynamic prompt with prefix text

You can combine static text with dynamic variables in the prompt:

```toml
[pipe.generate_sketch]
type = "PipeImgGen"
description = "Generate a black and white sketch from a description"
inputs = { subject = "Text" }
output = "Image"
prompt = "Create a black and white pencil sketch of: $subject"
model = "$gen-image"
```

### Example: Generating multiple images

This pipe takes a text prompt as input and generates three variations of the image using bracket notation.

```toml
[pipe.generate_logo_variations]
type = "PipeImgGen"
description = "Generate three logo variations from a prompt"
inputs = { img_prompt = "Text" }
output = "Image[3]"
prompt = "$img_prompt"
model = "$gen-image"
aspect_ratio = "square"
```

### Example: Image-to-image with a reference image

Declare an `Image` input and reference it in the prompt. At run time the referenced image is pulled from working memory, replaced in the text by an `[Image N]` placeholder, and passed to the generator alongside the rendered text — enabling image-to-image, reference-image, and editing generation.

```toml
[pipe.restyle_photo]
type = "PipeImgGen"
description = "Restyle a reference photo as an oil painting"
inputs = { ref = "Image" }
output = "Image"
prompt = "Repaint this photo as an oil painting in the style of Van Gogh: $ref"
model = "$gen-image"
```

You can also reference several images by declaring a list input. Each referenced image becomes its own `[Image N]` token, up to the model's `max_prompt_images`:

```toml
[pipe.blend_references]
type = "PipeImgGen"
description = "Blend multiple reference images into one scene"
inputs = { refs = "Image[]" }
output = "Image"
prompt = "Combine the styles of these reference images into a single coherent scene: $refs"
model = "$gen-image"
```

Image references also work through a dotted path to a nested image field (e.g. `$page.page_view` when `page` is a `Page`), and through any of the `$var`, `@var`, or `{{ var }}` syntaxes.

### Inputs and outputs

The `output` of `PipeImgGen` has to be a concept compatible with the native `Image` concept — a concept is compatible if it refines `Image`. Its inputs are ordinary variables injected into the `prompt` template: `Text`-compatible variables are interpolated as text, and `Image`-compatible variables are injected as reference images (as shown above). There is no dedicated prompt concept anywhere in this flow.

To use the multiple-images example above, you would reference an ordinary `Text` stuff (e.g., "A minimalist logo for a coffee shop, featuring a stylized mountain and a coffee bean") from the `prompt` template. After running, the output will be a list containing three generated `ImageContent` objects.

## Related Documentation

- [Image Generation Feature](../../../features/image-generation.md) - Overview of image generation capabilities
- [Cloud Storage](../../../features/cloud-storage.md) - Store generated images in cloud storage



## PipeSearch

# PipeSearch

The `PipeSearch` operator searches the web using a configurable search provider and returns structured results with an answer and source citations.

## How it works

`PipeSearch` takes a search prompt (which can include `$variable` template references) and sends it to a web search provider (such as Linkup). The provider returns a synthesized answer along with a list of sources.

The output is a `SearchResult` (or a concept that refines it), which contains:

-   `answer`: The synthesized answer text from the search
-   `sources`: A list of sources, each with a `title`, `url`, and optional `snippet`

## Configuration

`PipeSearch` is configured in your pipeline's `.mthds` file.

### Search Models and Backend System

PipeSearch uses the unified inference backend system to manage search providers. This means you can:

- Use different search providers (e.g., Linkup)
- Choose between standard and deep search models, with presets for each
- Route search requests through the same backend system as LLMs and other operators

Common search presets:

- `$standard`: Standard web search with fast results
- `$deep`: Deep web search for more thorough results

### MTHDS Parameters

| Parameter     | Type       | Description                                                                                                                                         | Required |
| ------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `type`        | string     | The type of the pipe: `PipeSearch`                                                                                                                  | Yes      |
| `description` | string     | A description of the search operation.                                                                                                              | Yes      |
| `inputs`      | dictionary | The input concept(s) for the search query, as a dictionary mapping input names to concept codes. Required when the prompt references variables.     | No       |
| `output`      | string     | The output concept produced by the search. `SearchResult` is the default shape, but PipeSearch can also target a structured output concept.         | Yes      |
| `model`       | string     | The search model preset to use (e.g., `"$standard"`, `"$deep"`). Defaults to the model specified in the global config.                | No       |
| `prompt`      | string     | The search query. Can be a static string or reference input variables using `$` prefix (e.g., `"What is $topic?"`).                                 | Yes      |
| `include_images` | boolean | Whether to include images in search results. Overrides the preset setting.                                                                        | No       |
| `max_results`    | integer | Maximum number of results to return. Overrides the preset setting.                                                                                | No       |
| `from_date`      | string  | Start date filter in YYYY-MM-DD format. Only return results from this date onwards.                                                               | No       |
| `to_date`        | string  | End date filter in YYYY-MM-DD format. Only return results up to this date.                                                                        | No       |
| `include_domains` | list of strings | Restrict search to these domains only.                                                                                                    | No       |
| `exclude_domains` | list of strings | Exclude results from these domains.                                                                                                       | No       |

### Example: Static search query

This pipe performs a fixed search query without any inputs.

```toml
[pipe.search_ai_news]
type = "PipeSearch"
description = "Search for the latest AI news"
output = "SearchResult"
model = "$standard"
prompt = "What are the latest developments in artificial intelligence?"
```

### Example: Dynamic search with input variable

This pipe takes a topic as input and searches the web for information about it.

```toml
[pipe.search_topic]
type = "PipeSearch"
description = "Search the web for information about a topic"
inputs = { topic = "Text" }
output = "SearchResult"
model = "$standard"
prompt = "What is $topic?"
```

### Example: Deep search

Use a deep search preset for more thorough results.

```toml
[pipe.deep_research]
type = "PipeSearch"
description = "Perform deep research on a topic"
inputs = { topic = "Text" }
output = "SearchResult"
model = "$deep"
prompt = "What are the main details about $topic?"
```

### Example: Filtering by date and domain

Use date filters and domain restrictions to narrow search results.

```toml
[pipe.search_recent_from_sources]
type = "PipeSearch"
description = "Search specific sources for recent news"
inputs = { topic = "Text" }
output = "SearchResult"
model = "$standard"
prompt = "What are the latest developments about $topic?"
from_date = "2026-01-01"
include_domains = ["reuters.com", "apnews.com", "bbc.com"]
```

When the output concept is compatible with `SearchResult`, the output contains the synthesized `answer` and a list of `sources` with their titles, URLs, and snippets. PipeSearch can also feed a structured output concept when your method needs a more specialized downstream shape.

## Related Documentation

- [Web Search Feature](../../../features/web-search.md) - Overview of web search capabilities


## PipeCompose

# PipeCompose

The `PipeCompose` operator composes data from your pipeline's working memory into new outputs. It supports two modes:

1. **Template mode**: Render templates to produce `Text`-like output
2. **Construct mode**: Build structured objects by mapping fields from inputs

In both modes, `PipeCompose` produces a single output stuff: the `output` concept cannot carry multiplicity (`output = "Report[]"` is rejected at validation).

## Template Mode

Template mode uses [Jinja2 templates](https://jinja.palletsprojects.com/) to dynamically generate text by combining data from working memory. This is ideal for creating formatted reports, HTML content, or constructing complex, multi-part prompts for LLMs.

### How Template Mode Works

`PipeCompose` takes all the data currently in the `WorkingMemory` and uses it as the context for rendering a Jinja2 template. The resulting text is then saved back to the working memory as a new `Text` or `Html` output. In template mode, the `output` concept must refine the native `Text` or `Html` concept — use construct mode for structured outputs.

Template mode supports two syntax variants:

- **Simple**: `template = "Hello $name"` — an inline string, defaults to `basic` category
- **Rich**: A `[pipe.name.template]` section with `template`, `category`, and optionally `templating_style`

### Template Context

The Jinja2 template has access to all the "stuffs" currently in the working memory. You can access them by the names they were given in previous pipeline steps. For example, if a previous step produced an output named `user_profile`, you can access its attributes in the template like `{{ user_profile.name }}` or `{{ user_profile.email }}`.

### Template Mode Configuration

| Parameter       | Type              | Description                                                                 | Required |
| --------------- | ----------------- | --------------------------------------------------------------------------- | -------- |
| `type`          | string            | The type of the pipe: `PipeCompose`                                         | Yes      |
| `description`   | string            | A description of the operation                                              | Yes      |
| `inputs`        | table             | Input variables needed for the template                                     | No       |
| `output`        | string            | The concept for the output                                                  | Yes      |
| `template`      | string or section | An inline template string, or a `[pipe.name.template]` section (see below)  | Yes*     |

*Template mode requires `template`. When using the rich form (`[pipe.name.template]` section), the following sub-fields are available:

| Sub-field          | Type   | Description                                                                      | Required |
|--------------------|--------|----------------------------------------------------------------------------------|----------|
| `template`         | string | The Jinja2 template string                                                       | Yes      |
| `category`         | string | Template category: `basic`, `markdown`, `html`, `mermaid`                        | Yes      |
| `templating_style` | table  | Style options: `{ tag_style = "...", text_format = "..." }`                      | No       |
| `extra_context`    | table  | Additional key-value pairs merged into the template rendering context             | No       |

### Template Mode Examples

**Simple inline template:**

```toml
[pipe.compose_greeting]
type = "PipeCompose"
description = "Compose a greeting message"
inputs = { user = "User" }
output = "Text"
template = "Hello $user.name, welcome to our platform!"
```

**Markdown template with category:**

```toml
[pipe.compose_report]
type = "PipeCompose"
description = "Format data as a markdown report"
inputs = { summary = "Text", items = "Item[]" }
output = "Text"

[pipe.compose_report.template]
category = "markdown"
template = """
# Report

$summary

{% for item in items %}
- {{ item.name }}: {{ item.value }}
{% endfor %}
"""
```

**HTML template with templating style:**

```toml
[pipe.format_html_report]
type = "PipeCompose"
description = "Format data as HTML"
inputs = { summary = "Text", items = "Item[]" }
output = "Html"

[pipe.format_html_report.template]
category = "html"
templating_style = { tag_style = "xml", text_format = "html" }
template = """
<h1>Report</h1>
<p>{{ summary }}</p>
<ul>
{% for item in items %}
  <li>{{ item.name }}: {{ item.value }}</li>
{% endfor %}
</ul>
"""
```

## Construct Mode

Construct mode builds structured objects by mapping fields from inputs. Use this when you need to assemble a complex output concept from multiple inputs without using an LLM.

### How Construct Mode Works

Instead of rendering a template, construct mode creates a structured object by specifying how each field should be populated. Fields can be:

- **Referenced from inputs**: Copy a value from working memory
- **Templated**: Generate a string using template interpolation
- **Fixed**: Use a static value

### Construct Mode Configuration

| Parameter     | Type   | Description                                               | Required |
| ------------- | ------ | --------------------------------------------------------- | -------- |
| `type`        | string | The type of the pipe: `PipeCompose`                       | Yes      |
| `description` | string | A description of the operation                            | Yes      |
| `inputs`      | table  | Input variables needed for the construct                  | No       |
| `output`      | string | The structured concept to output                          | Yes      |
| `construct`   | section| Field mappings (see below)                                | Yes*     |

*Either `template` or `construct` must be provided, but not both.

### Construct Field Methods

Each field in the `[pipe.name.construct]` section can use one of these methods:

| Method | Syntax | Description |
|--------|--------|-------------|
| Reference | `{ from = "input.field" }` | Copy value from input variable or nested field |
| Template | `{ template = "text with $var" }` | Generate string using template interpolation |
| Fixed | `"value"`, `123`, `true`, or `["a", "b"]` | Use a static value directly (scalars and lists) |
| Nested | a table with sub-fields (no `from`/`template` key) | Recursively compose a nested structured object |

A nested construct is written as a table whose keys are the sub-object's own field names — each sub-field uses any of the four methods, recursively:

```toml
[pipe.assemble_invoice.construct]
customer = { name = { from = "order.customer_name" }, tier = "standard" }
```

A `from` reference also accepts one modifier, `list_to_dict_keyed_by`: when the target field is a dict, it converts the referenced list into a dict keyed by the named attribute of each item:

```toml
[pipe.index_products.construct]
products_by_sku = { from = "products", list_to_dict_keyed_by = "sku" }
```

The referenced value must be a list, and every item must carry the key attribute with a string value — otherwise the composer raises an error.

### Copying Whole Inputs Into Native Fields

The `from` reference is not limited to dotted paths like `"customer.name"` — it can name a whole input variable. When the referenced input is a native stuff (`Text`, `Number`, `YesNo`, `Date`, `Time`, or a list of them) and the target field is native-typed, the composer automatically converts the content wrapper into the field's native value. This works for required and optional fields alike.

Conversion matrix:

| Source input | Native target field | Composed value |
|---|---|---|
| `Text` | `type = "text"` | the text string |
| `Number` | `type = "number"` | the number |
| `YesNo` | `type = "boolean"` | the boolean |
| `Date` | `type = "date"` | the date |
| `Time` | `type = "time"` | the time of day |
| `Text[]` | `type = "list"`, `item_type = "text"` | the list of strings |
| `Number[]` | `type = "list"`, `item_type = "number"` | the list of numbers |
| `YesNo[]` | `type = "list"`, `item_type = "boolean"` | the list of booleans |
| `Date[]` | `type = "list"`, `item_type = "date"` | the list of dates |
| `Time[]` | `type = "list"`, `item_type = "time"` | the list of times of day |

When the target field expects a content object rather than a native value (e.g. a field typed with a concept), the object is kept as-is — the conversion only fires when the field expects the native type.

One fidelity guard: a `Date` stuff that carries a time of day cannot be copied into a bare `date` field — that would silently drop the time and its UTC offset, so the composer raises an error instead. Target a `Date`-typed field to keep the full timestamp. The same guard applies per item when copying a `Date[]` into a list of `date` items.

`Time` needs no such guard, and the asymmetry is deliberate: a `Time` is a single value that carries its UTC offset inside the time itself, so copying it into a `time` field is lossless and nothing can be dropped. In the other direction, a whole `Date` does not convert into a `time` field at all — that would drop the date. Use a dotted path instead, `{ from = "deadline.time" }`, to take the time of day out of a `Date`.

Worked example — assembling a report from whole stuffs produced by earlier steps:

```toml
[concept.ScreeningReport]
description = "The final screening report"

[concept.ScreeningReport.structure]
match_score = { type = "number", description = "The match score", required = true }
rejection_email = { type = "text", description = "The rejection email — optional" }
interview_questions = { type = "list", item_type = "text", description = "Questions to ask — optional" }

[pipe.assemble_report]
type = "PipeCompose"
description = "Assemble the screening report from previously generated pieces"
inputs = { score = "Number", email = "Text", questions = "Text[]" }
output = "ScreeningReport"

[pipe.assemble_report.construct]
match_score = { from = "score" }
rejection_email = { from = "email" }
interview_questions = { from = "questions" }
```

Here `email` is a whole `Text` stuff copied into an optional `text` field, `questions` is a whole `Text[]` stuff copied into an optional `list` of `text` items, and `score` is a whole `Number` stuff copied into a required `number` field. Each lands as its native value (`str`, `list[str]`, `float`).

### Construct Mode Example

```toml
[pipe.assemble_invoice]
type = "PipeCompose"
description = "Assemble invoice from order and customer data"
inputs = { order = "Order", customer = "Customer" }
output = "Invoice"

[pipe.assemble_invoice.construct]
invoice_number = { template = "INV-$order.id" }
customer_name = { from = "customer.name" }
customer_email = { from = "customer.email" }
line_items = { from = "order.items" }
total = { from = "order.total" }
status = "pending"
version = 1
```

In this example:

- `invoice_number` is generated from a template using the order ID
- `customer_name`, `customer_email`, `line_items`, and `total` are copied from inputs
- `status` and `version` are fixed values

## Choosing Between Modes

| Use Case | Mode |
|----------|------|
| Generate text reports, emails, prompts | Template |
| Create HTML content | Template |
| Assemble structured data from multiple sources | Construct |
| Combine fields from different inputs into one object | Construct |
| Need to map/rename fields | Construct |

## Related Documentation

- [Working Memory](../working-memory.md) - How pipes access and share data
- [Provide Inputs](../provide-inputs.md) - Passing inputs to pipes and pipelines


## PipeFunc

# PipeFunc

The `PipeFunc` operator provides an essential escape hatch, allowing you to execute arbitrary Python code from within a pipeline. This is useful for custom data manipulation, complex logic, or integrating with external services not supported by other operators.

## How it works

`PipeFunc` operates by calling a Python function that has been registered with Pipelex's central function registry.

1.  **Decorator Required**: Functions must be decorated with `@pipe_func()` to be discovered and registered (since v0.12.0).
2.  **Automatic Discovery**: Functions with the `@pipe_func()` decorator are automatically discovered from anywhere in your project when Pipelex starts up.
3.  **Function Signature**: Eligible functions are registered using their function name (or a custom name) as the registry key.
4.  **Execution**: When the `PipeFunc` pipe is executed, it looks up your function by name and calls it, passing in the current `working_memory`.
5.  **Returning Data**: The function returns data, which `PipeFunc` places back into the working memory, associated with the pipe's `output` concept.

## Function Eligibility Requirements

For a function to be automatically registered and available to `PipeFunc`, it **must** meet all of the following criteria:

!!! warning "Function Eligibility Requirements"

- **Must be decorated with** `@pipe_func()` (required since v0.12.0)
- **Can be async or sync**: `async def` functions are awaited directly; plain `def` functions run in a worker thread (via `asyncio.to_thread`) so they don't block the pipeline
- **Must have exactly 1 parameter** named `working_memory`
- **Parameter type must be** `WorkingMemory`
- **Return type must be** a subclass of `StuffContent` (or a generic type like `ListContent[SomeType]`)
- **Must be discoverable** (not in excluded directories like `.venv`, `__pycache__`, etc.)

### Return values

Your Python function can return:

-   A `StuffContent` object (e.g., `TextContent`, `ImageContent`, or a custom `StructuredContent` model).
-   A `ListContent` containing `StuffContent` objects.

## How to Create a Function

To make a Python function available to `PipeFunc`:

1. Add the `@pipe_func()` decorator to your function
2. Place the function anywhere in your project (it will be auto-discovered)
3. Ensure it meets all eligibility requirements

!!! warning "Module Execution During Auto-Discovery"
    When Pipelex discovers functions with `@pipe_func()`, it imports the module containing them. **Any code at the module level (outside functions/classes) will be executed during import.** This can have unintended side effects.
    
    **Best practice:** Keep your `@pipe_func()` functions in dedicated modules with minimal module-level code, or ensure module-level code is safe to execute during discovery.

Here is an example of an eligible function:

```python
# in any Python file in your project (e.g., my_project/custom_functions.py)

from pipelex.system.registries.func_registry import pipe_func
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.stuffs.text_content import TextContent

@pipe_func()  # Required decorator for auto-discovery
async def concatenate_texts(working_memory: WorkingMemory) -> TextContent:
    """
    Retrieves two text stuffs, concatenates them, and returns a new text stuff.
    """
    # Get data from working memory using the names from previous steps
    text1 = working_memory.get_stuff_as_str("text_a")
    text2 = working_memory.get_stuff_as_str("text_b")

    concatenated = f"{text1} -- {text2}"

    return TextContent(text=concatenated)
```

The function will be automatically registered with the name `concatenate_texts` (the function name) when Pipelex starts up.

### Custom Registration Name

You can optionally specify a custom name for registration:

```python
@pipe_func(name="custom_concat")
async def concatenate_texts(working_memory: WorkingMemory) -> TextContent:
    # Implementation...
    pass
```

Then use `function_name = "custom_concat"` in your `.mthds` file.

### Function names share one flat name space

Registration names are unqualified — there is no module or package prefix — and every library directory Pipelex scans registers into the same process-wide registry. Two functions registered under the same name is therefore an error: Pipelex raises a `FuncRegistryError` naming both origins instead of letting scan order decide which one your `.mthds` files actually call.

Registering the *same* function again — a directory scanned twice, a module reached by two paths — is a no-op, so overlapping scan roots are fine.

If you hit a collision, give one of the two functions a distinct registration name:

```python
@pipe_func(name="invoices_summarize")
async def summarize(working_memory: WorkingMemory) -> TextContent:
    # Implementation...
    pass
```

Prefixing names with the domain they serve is the simplest way to keep them distinct as a project grows.

## Configuration

Once the function is registered, you can use it in your `.mthds` file.

### MTHDS Parameters

| Parameter       | Type   | Description                                                                 | Required |
| --------------- | ------ | --------------------------------------------------------------------------- | -------- |
| `type`          | string | The type of the pipe: `PipeFunc`                                                                          | Yes      |
| `description`   | string | A description of the function operation.                                                                   | Yes      |
| `function_name` | string | The unique name used to register the Python function (e.g., "combine_two_texts"). | Yes      |
| `output`        | string | The concept to associate with the function's return value.                  | Yes      |

### Example

This MTHDS snippet shows how to use the `concatenate_texts` function defined above. It assumes two previous pipes have produced outputs named `text_a` and `text_b`.

```toml
[pipe.combine_them]
type = "PipeFunc"
description = "Combine two text inputs using a custom Python function"
function_name = "concatenate_texts"
output = "ConcatenatedText"
```

## Related Documentation

- [Advanced Customizations](../../../advanced/index.md) - Extend Pipelex with custom logic
- [Working Memory](../working-memory.md) - How pipes access and share data


## Overview

# Pipe Controllers

!!! tip "MTHDS Standard Reference"
    Pipe controllers are part of the MTHDS open standard. For the authoritative language specification, see [Pipes & Controllers](https://mthds.ai/latest/language/pipes-controllers/) on mthds.ai. This page documents Pipelex-specific behavior and usage.

Pipe controllers are the orchestrators of a Pipelex pipeline. While [Pipe Operators](../pipe-operators/index.md) perform the work, pipe controllers define the method and manage the execution logic. They allow you to run other pipes in sequence, in parallel, or conditionally.

## Core Controllers

Here are the primary pipe controllers available in Pipelex:

-   [**`PipeSequence`**](./PipeSequence.md): The most fundamental controller. It runs a series of pipes one after another, passing the results from one step to the next.
-   [**`PipeParallel`**](./PipeParallel.md): Executes multiple independent pipes at the same time, significantly speeding up methods where tasks don't depend on each other.
-   [**`PipeBatch`**](./PipeBatch.md): Performs a "map" operation. It takes a list of items and runs the same pipe on every single item in parallel.
-   [**`PipeCondition`**](./PipeCondition.md): Adds branching logic (`if/else`) to your pipeline. It evaluates an expression and chooses which pipe to run next based on the result.

## Overview

Pipelex provides the following pipe controllers:

- `PipeSequence`: For chaining multiple pipes in sequence
- `PipeParallel`: For running different pipes in parallel
- `PipeBatch`: For running one pipe over a batch of inputs
- `PipeCondition`: For conditional execution based on an evaluated expression

## PipeSequence

Run multiple pipes in sequence.

### Key Features

- Sequential execution
- Working memory management
- Sub-pipe handling
- Pipeline composition

## PipeCondition

Enables conditional execution based on an evaluated expression.

### Key Features

- Expression-based routing
- Default fallback paths
- Jinja2 template support
- Error handling


## PipeSequence

# PipeSequence

The `PipeSequence` controller is used to execute a series of pipes one after another. It is the fundamental building block for creating linear methods where the output of one step becomes the input for the next.

## How it works

A `PipeSequence` defines a list of `steps`. Each step calls another pipe and gives a name to its output. The working memory is passed from one step to the next, accumulating results along the way.

-   The `input` of the `PipeSequence` is passed to the first pipe in the sequence.
-   The `output` of each intermediate step is named via the `result` key and becomes available in the working memory for all subsequent steps.
-   The final `output` of the `PipeSequence` is the output produced by the very last step in the sequence.

## Configuration

`PipeSequence` is configured in your pipeline's `.mthds` file.

### MTHDS Parameters

| Parameter  | Type            | Description                                                                                                    | Required |
| ---------- | --------------- | -------------------------------------------------------------------------------------------------------------- | -------- |
| `type`      | string          | The type of the pipe: `PipeSequence`                                                                          | Yes      |
| `description` | string          | A description of the sequence operation.                                                                          | Yes      |
| `inputs`    | dictionary  | The input concept(s) for the *first* pipe in the sequence, as a dictionary mapping input names to concept codes.                                                     | No       |
| `output`   | string          | The output concept produced by the *last* pipe in the sequence.                                                | Yes      |
| `steps`    | array of tables | An ordered list of the pipes to execute. Each table in the array defines a single step.                          | Yes      |

### Step Configuration

Each entry in the `steps` array is a table with the following keys:

| Key      | Type   | Description                                                        | Required |
| -------- | ------ | ------------------------------------------------------------------ | -------- |
| `pipe`   | string | The name of the pipe to execute for this step.                     | Yes      |
| `result` | string | The name to give to this step's output in the working memory. When omitted, the output is stored only in the unnamed `main_stuff` slot (the default output), so later steps can pick it up as their implicit input but cannot reference it by a dedicated name. | No       |
| `nb_output` | integer | Request a fixed number of outputs from this step's pipe. Cannot be combined with `multiple_output`. | No       |
| `multiple_output` | boolean | Request a variable number of outputs from this step's pipe (the model decides how many). Cannot be combined with `nb_output`. | No       |
| `batch_over` | string | The name of a list in the working memory to batch this step over, running the pipe once per item. Must be provided together with `batch_as`. See [Understanding Multiplicity](../understanding-multiplicity.md). | No       |
| `batch_as` | string | The name each item takes in the working memory during a `batch_over` run. Must differ from `batch_over` (e.g. `batch_over = "items"`, `batch_as = "item"`). | No       |

!!! important "Output Concept Matching"
    The output concept of the `PipeSequence` has to match the output of the last pipe in the sequence.

### Example

Let's imagine a pipeline that first extracts text from an image, then summarizes that text, and finally translates the summary into French.

```toml
[pipe.extract_text_from_image]
type = "PipeExtract"
description = "Extract text from an image"
inputs = { image = "Image" }
output = "Page[]"
model = "@default-extract-image"

[pipe.summarize_text]
type = "PipeLLM"
description = "Summarize text"
inputs = { extracted_text = "Page[]" }
output = "Text"

[pipe.translate_to_french]
type = "PipeLLM"
description = "Translate text to French"
inputs = { english_summary = "Text" }
output = "Text"


[pipe.image_to_french_summary]
type = "PipeSequence"
description = "Extract, summarize, and translate text from an image"
inputs = { image = "Image" }
output = "Text"
steps = [
    { pipe = "extract_text_from_image", result = "extracted_text" },
    { pipe = "summarize_text", result = "english_summary" },
    { pipe = "translate_to_french", result = "french_summary" },
]
```

## Related Documentation

- [Invoice Extraction Example](../../../cookbook/extract-invoice.md) - Complete invoice processing pipeline using PipeSequence
- [Write Tweet Example](../../../cookbook/write-tweet.md) - Multi-step tweet generation workflow
- [Table Extraction Example](../../../cookbook/extract-table.md) - Extract and correct tables from images
- [Gantt Extraction Example](../../../cookbook/extract-gantt.md) - Extract Gantt chart data from documents

## PipeParallel

# PipeParallel

The `PipeParallel` controller executes multiple pipes simultaneously, then combines their results into a single output. This is highly effective for running independent tasks concurrently in isolated branches.

## How it works

`PipeParallel` runs a list of sub-pipes in concurrent branches.

1.  **Isolation**: Before execution, `PipeParallel` creates a deep copy of the current `WorkingMemory` for each branch. This means every parallel pipe starts with the exact same state, but they run in complete isolation—a change in one branch will not affect another.
2.  **Concurrent Execution**: All specified pipes are executed at the same time using `asyncio.gather`.
3.  **Combination**: After all branches have finished, their results are always combined into a single object of the pipe's declared `output` concept, keyed by each branch's `result` name. That combined object is the pipe's main output.
4.  **Optional branch exposure**: If `add_each_output` is `true`, each branch's individual result is also added to the working memory under the name specified in its `result` key, so downstream pipes can consume branches directly.

## The output concept

The declared `output` of a `PipeParallel` is the concept the branch results are combined into. It must be one of:

-   **`Composite`** — the native untyped composition: the combined object simply holds each branch result under its `result` name. Use this when you don't want to declare a bespoke concept.
-   **A structured concept** whose fields correspond to the branch `result` names: every required field of the structure must match a branch `result` name, and every `result` name must match a declared field. Branch output *types* are checked too: when a field is typed with a content class, the branch feeding it must produce that concept or one refining it (e.g. a concept that refines `Text` fits a `TextContent` field).

Anything else is rejected at validation time: scalar native concepts (`Text`, `Image`, ...), `Dynamic`, `Anything`, and multiplicity suffixes (`Foo[]`, `Foo[3]`) are all invalid — a parallel combination is a named composite, never a scalar or a list (a list aggregation is [`PipeBatch`](PipeBatch.md)'s shape).

### Combining under absence

A branch result may legitimately be **absent** at run time: the branch pipe declares an optional output (`?`), or the branch is lifted (skipped) because a plain input is fed by one of the parallel's own optional (`?`) inputs. The combine handles absence by output kind:

-   **`Composite` output**: the absent component is simply omitted from the combined object (an absence note is kept in the ledger for observability).
-   **Structured output, non-required field**: the absence is absorbed as the field's default (`null` unless the field declares another default).
-   **Structured output, required field**: this is rejected **statically** at validation time (`optional_branch_required_field`) — a required field cannot be fed by a maybe-absent branch. Make the field non-required, or sink the absence upstream with a `?` input on the branch path.

## Configuration

`PipeParallel` is configured in your pipeline's `.mthds` file.

### MTHDS Parameters

| Parameter         | Type          | Description                                                                                                                                                                    | Required |
| ----------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- |
| `type`            | string        | The type of the pipe: `PipeParallel`                                                                          | Yes      |
| `description`     | string        | A description of the parallel operation.                                                                           | Yes      |
| `inputs`    | dictionary  | The input concept(s) for the parallel operation, as a dictionary mapping input names to concept codes.                                                     | No       |
| `output`   | string          | The concept the branch results are combined into: `Composite` or a structured concept whose fields match the branch `result` names.                                                | Yes      |
| `branches`        | array of tables| An array defining the pipes to run in parallel. Each table is a sub-pipe definition.                                                                                           | Yes      |
| `add_each_output` | boolean       | If `true`, also adds the output of each parallel pipe to the working memory individually under its `result` name. Defaults to `false`.                                                                       | No       |

### Parallel Step Configuration

Each entry in the `branches` array is a table with the following keys:

| Key      | Type   | Description                                                                              | Required |
| -------- | ------ | ---------------------------------------------------------------------------------------- | -------- |
| `pipe`   | string | The name of the pipe to execute for this branch.                                         | Yes      |
| `result` | string | The name for this branch's output. Must be unique within the `PipeParallel` definition. | Yes      |

### Example: Extracting different details from a text

Imagine you have a product description and you want to extract the product features and the product sentiment at the same time.

```toml
[concept.ProductDescription]
description = "A product description"
refines = "Text"

[concept.ProductFeatures]
description = "The features of a product"
refines = "Text"

[concept.ProductSentiment]
description = "The sentiment expressed about a product"
refines = "Text"

[concept.ProductAnalysis]
description = "Combined product analysis"

[concept.ProductAnalysis.structure]
features  = { type = "concept", concept_ref = "ProductFeatures", description = "Product features", required = true }
sentiment = { type = "concept", concept_ref = "ProductSentiment", description = "Product sentiment", required = true }

[pipe.extract_features]
type = "PipeLLM"
description = "Extract features from text"
inputs = { description = "ProductDescription" }
output = "ProductFeatures"

[pipe.analyze_sentiment]
type = "PipeLLM"
description = "Analyze sentiment of text"
inputs = { description = "ProductDescription" }
output = "ProductSentiment"

# The PipeParallel definition
[pipe.analyze_product_in_parallel]
type = "PipeParallel"
description = "Extract features and sentiment at the same time"
inputs = { description = "ProductDescription" }
output = "ProductAnalysis" # The concept the branch results are combined into
add_each_output = true
branches = [
    { pipe = "extract_features", result = "features" },
    { pipe = "analyze_sentiment", result = "sentiment" },
]
```

How this works:

1.  The `analyze_product_in_parallel` pipe starts. It receives a `ProductDescription`.
2.  Two parallel branches are created, both with access to the `ProductDescription`.
3.  The `extract_features` pipe runs in one branch, and the `analyze_sentiment` pipe runs in the other, simultaneously.
4.  After both complete, `PipeParallel` collects the results. The output from `extract_features` is named `features`, and the output from `analyze_sentiment` is named `sentiment`.
5.  A new structured object of type `ProductAnalysis` is created and populated with the results, like `{"features": ..., "sentiment": ...}`. This object becomes the main output of the `analyze_product_in_parallel` pipe.
6.  Because `add_each_output` is `true`, `features` and `sentiment` are also available individually in the working memory for downstream pipes.

If you don't want to declare a `ProductAnalysis` concept, set `output = "Composite"` instead: the combined object then holds the same `{"features": ..., "sentiment": ...}` shape as an untyped composition.

## Related Documentation

- [Pipeline Orchestration](../../../features/pipeline-orchestration.md) - Overview of pipeline orchestration capabilities
- [Native Concepts](../../concepts/native-concepts.md) - Includes the `Composite` concept


## PipeBatch

# PipeBatch

The `PipeBatch` controller provides a powerful "map" operation for your pipelines. It takes a list of items as input and runs the same pipe on each item in the list, executing the branches concurrently for efficiency.

This is the ideal controller for processing collections of documents, images, or any other data records where the same logic needs to be applied to each one independently.

## How it works

`PipeBatch` orchestrates a parallel, per-item execution of a single "branch pipe".

1.  **Input List**: It identifies an input list from the working memory.
2.  **Branching**: For each item in the input list, it creates a new, isolated execution branch.
3.  **Isolation & Injection**: Each branch gets a deep copy of the `WorkingMemory`. The specific item for that branch is injected into this memory with a defined name.
4.  **Concurrent Execution**: The specified `branch_pipe_code` runs across the branches concurrently — in bounded chunks, by default at most `max_concurrency` branches at a time. Each branch pipe operates only on its own item.
5.  **Aggregation**: After all branches have completed, `PipeBatch` collects the individual output from each one and aggregates them into a single new list. This list becomes the final output of the `PipeBatch` pipe.

## Concurrency

`PipeBatch` does not spawn every branch at once. Branches run in bounded concurrent chunks, capped by the `max_concurrency` setting under `[pipelex.pipeline_execution_config]` (default `8`). This keeps a large batch — one pipe over thousands of items — from overwhelming memory, the asyncio event loop, and provider rate limits.

To restore unbounded fan-out (every branch started at once), set `max_concurrency = "unbounded"`.

The setting is read **once, when the run's parameters are built**, and then frozen onto the run (`PipeRunParams.batch_max_concurrency`) — every `PipeBatch` in that run, at any depth, uses the value that was in effect when the run started. Editing the config while a run is in flight does not reshape it. This matters most on a durable-execution backend, where the bound is also the chunk size that decides where the backend's task boundaries fall between branch dispatches: a value re-read mid-run could make a replay group its dispatches differently from the recorded history.

Results always preserve input order regardless of the concurrency bound. If a branch fails, the failure propagates and the first error by input index wins.

For durable, rate-limited execution of very large batches, run the pipeline on the Temporal track.

## Compaction Under Absence

When a branch's result resolves as a recorded absence — typically because the branch pipe is a `PipeCondition` whose outcome was `continue`, or the branch was skipped (lifted) on an absent optional value — that branch contributes **no item** to the aggregated output list. The list is compacted: a list cannot hold a hole, and absent results are dropped rather than replaced by placeholders. Order is preserved among the results that did produce a value, and each dropped branch is logged with its absence reason.

This is the batch arm of the "route or skip" pattern: batch over items with a condition branch pipe that builds a result for the items that qualify and `continue`s past the rest — the output is the compacted list of qualifying results. See [Understanding Optionality](../understanding-optionality.md) for the full absence model.

## Configuration

`PipeBatch` is configured in your pipeline's `.mthds` file.

### MTHDS Parameters

| Parameter          | Type         | Description                                                                                                                                      | Required |
| ------------------ | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | -------- |
| `type`             | string       | The type of the pipe: `PipeBatch`                                                                          | Yes      |
| `description`      | string       | A description of the batch operation.                                                                           | Yes      |
| `inputs`           | dictionary   | The input concept(s) for the batch operation, as a dictionary mapping input names to concept codes.                                                     | Yes       |
| `output`           | string       | The output concept produced by the batch operation.                                                | Yes      |
| `branch_pipe_code` | string       | The name of the single pipe to execute for each item in the input list.                                                                          | Yes      |
| `input_list_name`  | string       | The name of the input list to iterate over. Must match one of the keys in `inputs`. Typically a plural noun (e.g. `articles`).                   | Yes      |
| `input_item_name`  | string       | The name that an individual item from the list will have inside its execution branch — this is how the branch pipe finds its input. Must differ from `input_list_name` and from every key in `inputs`. Typically the singular form of the list name (e.g. `article`). | Yes      |

### Example: Summarizing a list of articles

Suppose you have a list of articles and you want to generate a summary for each one.

```toml
# The pipe that knows how to summarize one article
[pipe.summarize_one_article]
type = "PipeLLM"
description = "Summarize a single article"
inputs = { article = "ArticleText" }
output = "ArticleSummary"
prompt = "Please provide a one-sentence summary of the following article:\n\n@article"

# The PipeBatch definition
[pipe.summarize_all_articles]
type = "PipeBatch"
description = "Summarize a batch of articles in parallel"
inputs = { articles = "ArticleText[]" }  # This is the list to iterate over
output = "ArticleSummary[]" # This will be the list of summaries
branch_pipe_code = "summarize_one_article"
input_list_name = "articles" # Name of the input list to iterate over
input_item_name = "article" # Name of an item within the branch
```

How this works:
1.  The `summarize_all_articles` pipe receives a list of `ArticleText` items under the name `articles`. Let's say it contains 10 articles.
2.  `PipeBatch` creates 10 parallel branches.
3.  In branch #1, it takes the first article from the `articles` list, puts it into the branch's isolated working memory, and gives it the name `article` (as specified by `input_item_name`).
4.  The `summarize_one_article` pipe is then executed in branch #1. It looks for an input named `article`, finds the injected article, and produces a summary.
5.  Steps 3 and 4 run concurrently across branches, up to `max_concurrency` at a time (8 by default), until all 10 articles are processed.
6.  Once all `summarize_one_article` pipes are done, `PipeBatch` collects the 10 `ArticleSummary` outputs and bundles them into a single `ArticleSummary[]` list. This list is the final result.

## Related Documentation

- [Understanding Multiplicity](../understanding-multiplicity.md) - How Pipelex handles lists and batch processing
- [Understanding Optionality](../understanding-optionality.md) - Presence markers, absence records, and compaction


## PipeCondition

# PipeCondition

The `PipeCondition` controller adds branching logic to your pipelines. It evaluates an expression and, based on the string result, chooses which subsequent pipe to execute from a map of possibilities.

## How it works

`PipeCondition` is a routing mechanism. Its execution flow is as follows:

1.  **Evaluate an Expression**: It takes an expression and renders it using Jinja2, with the full `WorkingMemory` available as context. This evaluation results in a simple string.
2.  **Look Up in the Outcome Map**: The resulting string is used as a key to find a corresponding outcome in `outcomes` — the name of a pipe to execute, or a special outcome (`continue` / `fail`).
3.  **Use Default**: If the key is not found in `outcomes`, the `default_outcome` is used.
4.  **Execute Chosen Pipe**: The chosen pipe is then executed. It receives the exact same `WorkingMemory` and inputs that were passed to the `PipeCondition` operator. The output of the chosen pipe becomes the output of the `PipeCondition` itself.

## Configuration

`PipeCondition` is configured in your pipeline's `.mthds` file.

### MTHDS Parameters

| Parameter                      | Type           | Description                                                                                                                                              | Required                       |
| ------------------------------ | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| `type`                         | string         | The type of the pipe: `PipeCondition`                                                                          | Yes                            |
| `description`                   | string         | A description of the condition operation.                                                                          | Yes                            |
| `inputs`                       | dictionary     | The input concept(s) for the condition, as a dictionary mapping input names to concept codes.                                                     | Yes                            |
| `output`                       | string         | The output concept produced by the selected pipe.                                                | Yes                            |
| `expression`                   | string         | A simple Jinja2 expression. `{{ ... }}` are automatically added. Good for simple variable access like `"my_var.category"`.                                | Yes (or `expression_template`)   |
| `expression_template`            | string         | A full Jinja2 template string. Use this for more complex logic, like `{% if my_var.value > 10 %}high{% else %}low{% endif %}`.                           | Yes (or `expression`)          |
| `outcomes`                     | table (dict)   | A mapping where keys are the possible string results of the expression, and values are the names of the pipes to execute — or the special outcomes `continue` / `fail`.                                  | Yes                            |
| `default_outcome`            | string         | The outcome used when the expression result does not match any key in `outcomes`: a pipe name, `continue`, or `fail`.                                                             | Yes                             |
| `add_alias_from_expression_to` | string         | An advanced feature. If provided, an alias named after the string result of the expression evaluation is added to the working memory, pointing to the existing entry named by this value (which must already be in the working memory).               | No                             |

!!! important "Output Concept Matching"
    The output concept of the `PipeCondition` has to match the output of all the pipes in the `outcomes`.

### Example: Simple routing based on category

Here's a basic example showing how PipeCondition routes based on input data:

```toml
domain = "routing_example"
description = "Example of PipeCondition routing"

[concept.CategoryInput]
description = "Input with a category field"

[concept.CategoryInput.structure]
category = "The category of the input"

# Define the PipeCondition first
[pipe.route_by_category]
type = "PipeCondition"
description = "Route based on category field"
inputs = { input_data = "CategoryInput" }
output = "native.Text"
expression = "input_data.category"
default_outcome = "fail"

[pipe.route_by_category.outcomes]
small = "process_small"
medium = "process_medium"
large = "process_large"

# Define the pipes that PipeCondition can route to
[pipe.process_small]
type = "PipeLLM"
description = "Handle small category"
output = "native.Text"
prompt = """
Output this only: "small"
"""

[pipe.process_medium]
type = "PipeLLM"
description = "Handle medium category"
output = "native.Text"
prompt = """
Output this only: "medium"
"""

[pipe.process_large]
type = "PipeLLM"
description = "Handle large category"
output = "native.Text"
prompt = """
Output this only: "large"
"""
```

How this works:
1. `PipeCondition` receives input data with a `category` field (e.g., `{category: "small"}`)
2. It evaluates the expression `"input_data.category"` which results in the string `"small"`
3. It looks up `"small"` in the `outcomes` and finds the corresponding pipe: `"process_small"`
4. The `process_small` pipe is executed with the same working memory
5. The output from `process_small` becomes the output of the entire `PipeCondition`

### Example: Routing with default fallback

```toml
[pipe.route_with_fallback]
type = "PipeCondition"
description = "Route with default handling"
inputs = { classification = "DocumentType" }
output = "ProcessedDocument"
expression = "classification.type"
default_outcome = "process_unknown"

[pipe.route_with_fallback.outcomes]
invoice = "process_invoice"
receipt = "process_receipt"

[pipe.process_invoice]
type = "PipeLLM"
description = "Process invoice documents"
inputs = { classification = "DocumentType" }
output = "ProcessedDocument"
prompt = """
Process this invoice document...
"""

[pipe.process_receipt]
type = "PipeLLM"
description = "Process receipt documents" 
inputs = { classification = "DocumentType" }
output = "ProcessedDocument"
prompt = """
Process this receipt document...
"""

[pipe.process_unknown]
type = "PipeLLM"
description = "Handle unknown document types"
inputs = { classification = "DocumentType" }
output = "ProcessedDocument"
prompt = """
Process this unknown document type...
"""
```

## Special outcomes: `continue` and `fail`

Beside pipe names, an outcome (or the `default_outcome`) can be one of two special outcomes:

- **`fail`**: the run fails loudly with an error naming the pipe and the evaluated expression. Use it to make "this should never happen" branches explicit.
- **`continue`**: the condition declares that it produced **no output**. The runtime records an absence for the declared output (with the evaluated expression as the reason) and the run continues as a success — the rest of the working memory is unchanged.

Because `continue` resolves the declared output as absent, a `continue`-reachable condition (any outcome mapped to `continue`, or `default_outcome = "continue"`) **must declare its output optional** — `output = "Constraint?"`. Validation enforces this statically (`optional_output_required`), so the no-output path is always visible to the type system. The same visibility rule applies at the condition's boundary: if a mapped outcome pipe declares an optional output, the condition must declare `?` too (`optional_not_handled`).

### What happens downstream of `continue`

An absent output propagates like any recorded absence:

- A downstream pipe consuming it as a **plain** input is skipped (lifted), and its own output is recorded absent in turn.
- A downstream pipe declaring the input **optional** (`?`) runs and handles both arms.
- A downstream pipe declaring the input **forced** (`!`) fails with a typed error carrying the full provenance chain.
- Inside a `PipeBatch`, an absent branch result is dropped from the aggregated list (compaction): batching a "keep or skip" condition over items yields a list of only the kept results.
- Inside a `PipeParallel`, an absent branch result is omitted from the combined output (a non-required structured field absorbs it as its default — `null` unless the field declares another default).

!!! warning "Breaking change: `continue` no longer passes the previous output through"
    `continue` used to deliver the current main stuff by passing it through. It now resolves the condition's output as **absent** instead. The previous value is not lost — it stays in the working memory under its own name, so a downstream pipe consumes it explicitly by that name (declared `?` when it may be absent). If you need "use the new value if produced, otherwise the previous one", that coalescing idiom is planned as a follow-up; today, model it with an explicit downstream pipe that declares both inputs.

```toml
[pipe.build_or_skip]
type = "PipeCondition"
description = "Builds a constraint for approved links, skips rejected ones"
inputs = { verified_link = "VerifiedLink" }
output = "Constraint?"
expression = "verified_link.verdict"
default_outcome = "continue"

[pipe.build_or_skip.outcomes]
approved = "build_single_constraint"
rejected = "continue"
```

## Expression Types

### Simple Expression
```toml
expression = "input_data.category"
```

- Direct access to working memory variables
- No template syntax needed (`{{ }}` are automatically added)
- Good for simple field access
- Access to Jinja2 filters and functions

### Complex Expression  
```toml
expression_template = "{% if input_data.score >= 70 %}high{% else %}low{% endif %}"
```

- Full Jinja2 template syntax
- Conditional logic and loops
- Complex transformations
- Multiple variable access

## Features

### Default Routing
```python
default_outcome = "process_unknown"
```

- Fallback pipe when no match is found

### Expression Aliasing
```toml
add_alias_from_expression_to = "category_type"
```

- Creates an alias from the expression result
- Makes the result available in working memory
- Requires the target to exist in working memory beforehand

## Related Documentation

- [Pipeline Orchestration](../../../features/pipeline-orchestration.md) - Overview of pipeline orchestration capabilities
- [Provide Inputs](../provide-inputs.md) - Passing inputs to pipes and pipelines


## Signature Pipes

# Signature Pipes

Every pipe *has* a signature — its contract: the `inputs` it consumes, the `output` it promises, and a `description` of its purpose. A `PipeSignature` is that contract **alone**: a pipe declared by its signature with no implementation behind it. It is the placeholder you reach for when you want to design the shape of a pipeline (or one of its steps) before committing to the operator that will eventually do the work.

Because a signature is a contract rather than a way of running, it sits **outside** the executable pipe taxonomy. That taxonomy has two levels: a `PipeCategory` (`PipeOperator` or `PipeController`) and, beneath it, a concrete `PipeType` (`PipeLLM`, `PipeSequence`, and the rest). A `PipeSignature` is neither — it is not a `PipeType` like `PipeLLM` or `PipeSequence`, and it belongs to no `PipeCategory`. Its runtime form is a dry-run-only shim — it can verify inputs and mock its declared output, but it never executes live. Implementing a signature is not swapping one pipe *type* for another; it is fulfilling a contract with a concrete pipe that satisfies it.

Signatures let an agent (or a human author) build a complete, dry-runnable bundle while leaving some pipes deliberately unimplemented. Strict validation refuses any pipeline that still depends on a signature; lenient mode (`--allow-signatures`) dry-runs them as mocks, so you can keep iterating on the structure of a method before writing every step.

## When to use a `PipeSignature`

- **Sketching a pipeline top-down.** Define the controller (e.g. a `PipeSequence`) and stub each step as a `PipeSignature` so the structure dry-runs end-to-end.
- **Designing the contract first.** Capture inputs and outputs precisely — multiplicity included — before deciding whether the implementation will be a `PipeLLM`, a `PipeFunc`, or another operator.
- **Handing work between agents.** A planner agent emits signatures for the steps; an implementer agent replaces each one with a real operator.

## How a signature behaves at runtime

| Mode | Behavior |
| ---- | -------- |
| **Strict validation** (default) | Signatures are never a validation error: the bundle is valid, but a pipeline that reaches an unimplemented signature is reported as NOT yet runnable. The validation report lists the outstanding placeholders in `pending_signatures` and sets `is_runnable: false`; `validate` / `validate --all` / `validate bundle` / `validate method` then exit non-zero on that verdict (the runnability gate). In strict mode signature pipes are excluded from the dry-run sweep (not mock-run). |
| **Lenient validation** (`--allow-signatures`) | The verdict is unchanged (`pending_signatures` / `is_runnable` are still reported), but the runnability gate is tolerated (exit 0) and signature pipes are mock-run during the sweep — each mints a mock `Stuff` matching its declared `output`, multiplicity included. |
| **Live execution** | Running a pipeline that hits a signature raises `PipeSignatureNotExecutableError` — a signature has no implementation to execute. |

## MTHDS Parameters

A signature has **no `type`** — omitting the type *is* what makes a pipe a signature. A `[pipe.x]` section that declares only the contract below (and nothing else) is a signature. Adding any other field means you are describing an implementation, which must name its `type`; and writing `type = "PipeSignature"` explicitly is an error (`PipeSignature` is not a pipe type — delete the line).

| Parameter           | Type   | Description                                                                                                  | Required |
| ------------------- | ------ | ------------------------------------------------------------------------------------------------------------ | -------- |
| `description`       | string | Human-readable purpose of the contract.                                                                      | Yes      |
| `inputs`            | object | Mapping of input variable names to concept references. Multiplicity is supported (`Doc[]`, `Image[3]`).      | No       |
| `output`            | string | The concept the signature promises to produce. Multiplicity is supported.                                    | Yes      |
| `signature_for`     | string | Optional hint naming the downstream pipe type (e.g. `"PipeLLM"`) the signature stands in for. Tooling-only.  | No       |

`signature_for` cannot be set to `"PipeSignature"` — a signature standing in for a signature is nonsensical and is rejected at load time.

## Examples

### A single contract pipe

The simplest signature: a pipe that promises to summarize a document, with no implementation.

```toml
domain = "signature_demo"
main_pipe = "summarize_doc"

[concept]
SigDocument = "A document concept used for testing signatures."

[pipe.summarize_doc]
description = "Produces a summary of a document (contract only)."
inputs = { doc = "SigDocument" }
output = "Text"
```

No `type` line — that is what makes `summarize_doc` a signature. Validating this in strict mode fails (the main pipe *is* a signature). Validating it with `--allow-signatures` succeeds and dry-runs to a mock `Text`.

### A signature inside a `PipeSequence`

A common pattern: an existing operator runs first, then a signature stands in for the step that hasn't been implemented yet.

```toml
domain = "signature_mixed"
main_pipe = "process_doc"

[concept]
MixDoc = "A document concept."
MixSummary = "A summary concept."

[pipe.extract_doc]
type = "PipeLLM"
description = "Extract text from a document."
inputs = { doc = "MixDoc" }
output = "Text"
prompt = "Extract text from $doc."

[pipe.summarize_extracted]
description = "Summarize extracted text (contract only)."
inputs = { extracted = "Text" }
output = "MixSummary"

[pipe.process_doc]
type = "PipeSequence"
description = "Extract then summarize."
inputs = { doc = "MixDoc" }
output = "MixSummary"
steps = [
  { pipe = "extract_doc", result = "extracted" },
  { pipe = "summarize_extracted", result = "summary" },
]
```

Strict validation reports the dependency chain (`process_doc → summarize_extracted`) so you know exactly which controllers reach the signature. Lenient validation dry-runs the whole sequence: `extract_doc` mocks a `Text`, then `summarize_extracted` mocks a `MixSummary`.

### Multiplicity in a signature's inputs

Signatures honor the same multiplicity rules as other pipes (see [Understanding Multiplicity](understanding-multiplicity.md)).

```toml
domain = "signature_multiplicity"
main_pipe = "fuse_docs_and_images"

[concept]
FuseDoc = "A document."
FuseImage = "An image."
FuseReport = "A report."

[pipe.fuse_docs_and_images]
description = "Combine docs and exactly 3 images (contract only)."
inputs = { docs = "FuseDoc[]", images = "FuseImage[3]" }
output = "FuseReport"
```

Variable lists (`FuseDoc[]`) and fixed-length lists (`FuseImage[3]`) round-trip through the interpreter unchanged.

## Validating signatures from the CLI

Signatures interact with the [`pipelex validate`](../../tools/cli/validate.md) command via the `--allow-signatures` flag:

```bash
# Strict (default): fails if any reachable pipe is a PipeSignature
pipelex validate process_doc

# Lenient: signatures dry-run as mocks
pipelex validate process_doc --allow-signatures

# Same flag works for bundles and the --all sweep
pipelex validate bundle methods/draft_pipeline.mthds --allow-signatures
pipelex validate --all --allow-signatures
```

When `--all` runs in strict mode, signature pipes themselves are skipped during the iteration (an unreached signature is fine). Strict mode only fires when a non-signature pipe *depends on* a signature.

!!! note "Signatures and definitions can live in separate files"
    A signature and the concrete pipe that implements it do not have to sit in the same `.mthds` file. A same-domain library can be split across sibling files in one directory: a header file forward-declares pipes as `PipeSignature` (plus the concepts those contracts reference), and definition files supply the concrete pipes. Loading the set merges them — a `PipeSignature` and a concrete pipe of the same code reconcile (the concrete satisfies the header), provided their `inputs`/`output` contracts match. Order does not matter.

    On a successful lenient run, `validate bundle` reports `pending_signatures` — the headers still unimplemented across the *whole merged* library — and states explicitly whether the method is runnable (runnable once nothing remains, not yet runnable while headers are pending), so you can see exactly what is left to define. See [Validate Commands](../../tools/cli/validate.md) for the output surface. The same verdict is reported by the protocol-level `validate` (`PipelexMTHDSProtocol.validate`) as `pending_signatures` + `is_runnable` fields on its validation report, so a build driving the runtime through the MTHDS Protocol sees it too.

## Replacing a signature with a real implementation

When you are ready to implement, add a `type` to the pipe (a signature has none) and the operator-specific fields. The pipe code, `description`, `inputs`, and `output` typically carry over unchanged — that is the value of having declared the contract first.

```toml
[pipe.summarize_extracted]
type = "PipeLLM"
description = "Summarize extracted text."
inputs = { extracted = "Text" }
output = "MixSummary"
prompt = "Summarize the following text:\n\n$extracted"
```

Once every signature in the dependency graph is replaced, strict validation succeeds and the pipeline can be run live.

## Related Documentation

- [Understanding Multiplicity](understanding-multiplicity.md) — the bracket notation used for both inputs and outputs.
- [Pipe Operators](pipe-operators/index.md) — operators you can swap a signature out for.
- [Pipe Controllers](pipe-controllers/index.md) — controllers that orchestrate signatures alongside real pipes.
- [CLI Validate](../../tools/cli/validate.md) — the `--allow-signatures` flag and validation surface.


## Optimize Cost & Quality

# LLM Settings Guide

## Overview

Pipelex provides a flexible way to configure and manage your LLM (Large Language Model) integrations through the inference backend configuration system. 

The system provides three main concepts for LLM configuration:

- LLM Handles (Aliases)
- LLM Presets  
- Model Deck

For complete details about the inference backend configuration system, see the [Inference Backend Configuration](../configuration/config-technical/inference-backend-config.md) documentation.

## LLM Handles (Aliases)

An LLM handle can be either:

1. **A direct model name** (like "gpt-4o-mini", "claude-3-sonnet") - automatically available for all models loaded by the inference backend system
2. **An alias** - user-defined shortcuts that map to model names, defined in the `[llm.aliases]` section:

### Example Alias Configurations

```toml
[llm.aliases]
best-claude = "claude-4.1-opus"
best-gemini = "gemini-2.5-pro"
best-mistral = "mistral-large"
base-gpt = "gpt-5"
```

The system first looks for direct model names, then checks aliases if no direct match is found. The system handles model routing through backends automatically.

!!! tip "Alias Naming Convention"
    Defining an alias is always meant to describe what model it is. Never define an alias to describe what it is for or what it's good at. LLM Presets are for that.

## LLM Settings & LLM Presets

LLM Settings combine an LLM handle with specific parameters optimized for particular tasks. They help maintain consistency across similar operations and make it easier to switch between different configurations.

An LLM Preset is simply a name for a LLM Settings that you have predefined in order to use it in various places.

### Example LLM Preset definitions

```toml
[llm.presets]

engineering-structured = {
    model = "@default-premium-structured",
    temperature = 0.2
}

retrieval = {
    model = "@default-large-context-text",
    temperature = 0.1
}
```

### Reasoning Presets

LLM Settings support `reasoning_effort` and `reasoning_budget` parameters for enabling extended reasoning (chain-of-thought / thinking). Here are the built-in reasoning presets:

```toml
[llm.presets]
deep-analysis = { model = "@default-premium", temperature = 0.1, reasoning_effort = "high", description = "Deep reasoning and analysis" }
quick-reasoning = { model = "@default-premium", temperature = 0.3, reasoning_effort = "low", description = "Quick reasoning for simple tasks" }
```

`reasoning_effort` accepts values from `"none"` to `"max"`. For an explicit token budget, use `reasoning_budget` instead (mutually exclusive with `reasoning_effort`). For provider-specific behavior and model examples, see [Reasoning Controls](../under-the-hood/reasoning-controls.md).

### Using LLM Settings in Pipelines

Here's how to use these configurations in your pipelines:

```toml
[pipe.generate_response]
type = "PipeLLM"
description = "Generate a creative response"
inputs = { question = "Question" }
output = "Response"
model = {
    model = "gpt-4-turbo",  # Using inline LLM settings
    temperature = 0.8,
}
prompt = """
Generate a creative response to this question:

@question
"""

[pipe.extract_weather_data]
type = "PipeLLM"
description = "Extract structured weather data from text"
inputs = { text = "Text" }
output = "WeatherData"
model = "$retrieval"  # Using a preset with $ prefix
prompt = """
Extract the weather data from this text:

@text
"""
```

## Model Deck

The Model Deck is your central configuration hub for all LLM-related settings. It's stored in the `.pipelex/inference/deck/` directory and consists of numbered deck files loaded in order:

- `1_llm_deck.toml`: LLM aliases and presets
- `2_img_gen_deck.toml`: Image generation configuration
- `3_extract_deck.toml`: Document extraction configuration
- `4_search_deck.toml`: Search configuration
- `x_custom_extract_deck.toml`: Custom extraction waterfalls/overrides (loaded last)
- `x_custom_llm_deck.toml`: Custom LLM waterfalls/overrides (loaded last)

### Directory Structure

```bash
.pipelex/
└── inference/
    ├── backends.toml              # Backend configurations
    ├── routing_profiles.toml      # Model routing rules
    ├── backends/                  # Individual backend model specs
    │   ├── openai.toml
    │   ├── anthropic.toml
    │   └── ...
    └── deck/                      # Model deck configurations
        ├── 1_llm_deck.toml            # LLM aliases & presets
        ├── 2_img_gen_deck.toml        # Image generation config
        ├── 3_extract_deck.toml        # Document extraction config
        ├── 4_search_deck.toml         # Search config
        ├── x_custom_extract_deck.toml # Custom extraction waterfalls/overrides
        └── x_custom_llm_deck.toml     # Custom LLM waterfalls/overrides
```



## Best Practices

1. **Consistent Naming**: Use clear, descriptive names for handles and presets
1. **Task-Specific Presets**: Create presets optimized for specific skills and tasks
1. **Cost Management**: Consider using different models based on task complexity and cost requirements


## LLM Prompting Style

# LLM Prompting Style Configuration

The `PromptingConfig` class controls how Pipelex handles prompting styles for different LLM targets.

## Configuration Options

```python
class PromptingConfig(ConfigModel):
    default_prompting_style: TemplatingStyle
    prompting_styles: dict[str, TemplatingStyle]
```

### Fields

- `default_prompting_style`: The default prompting style to use when none is specified
- `prompting_styles`: Dictionary mapping LLM targets to their specific prompting styles

## Prompting Styles

Each prompting style defines how prompts are formatted and presented to the LLM. The style can be customized per LLM target to optimize performance and ensure compatibility.

## Example Configuration

```toml
[pipelex.prompting_config]
default_prompting_style = { tag_style = "xml" }

[pipelex.prompting_config.prompting_styles]
openai = { tag_style = "ticks" }
anthropic = { tag_style = "xml" }
mistral = { tag_style = "square_brackets" }
gemini = { tag_style = "xml" }
```

## Usage

The configuration provides a method to get the appropriate prompting style:

```python
def get_prompting_style(self, prompting_target: PromptingTarget | None = None) -> TemplatingStyle | None:
    if prompting_target:
        return self.prompting_styles.get(prompting_target, self.default_prompting_style)
    return None
```

This allows for:

- Target-specific prompting styles
- Fallback to default style when no specific style is defined
- Optional prompting when no target is specified

## Best Practices

- Define a sensible default prompting style
- Configure specific styles for LLMs with unique requirements
- Test prompting styles with each LLM target
- Document any special formatting requirements

## Related Documentation

- [PipeLLM › Structuring Method](./pipes/pipe-operators/PipeLLM.md#structuring-method-preliminary-text) - Two-step text-then-structure via `structuring_method = "preliminary_text"`
- [PipeStructure](./pipes/pipe-operators/PipeStructure.md) - Standalone operator that turns text into a structured concept
- [PipeLLM](./pipes/pipe-operators/PipeLLM.md) - Pipe-level prompting and output controls
- [LLM Integration](../features/llm-integration.md) - High-level LLM capability overview


# Reference

## Overview

# Pipelex CLI Documentation

The `pipelex` CLI is the runtime and project-configuration CLI for Pipelex. Use it to initialize config, validate methods, run pipes, inspect the runtime state, and generate supporting files.

## Overview

The Pipelex CLI is organized into several command groups:

| Command | Description |
|---------|-------------|
| [**init**](init.md) | Initialize Pipelex configuration |
| [**update**](update.md) | Refresh the model deck to match the installed pipelex version |
| [**validate**](validate.md) | Validate configuration and pipelines |
| [**fix**](fix.md) | Apply deterministic safe fixes to a bundle and re-validate |
| [**show**](show.md) | Inspect configuration, pipes, and AI models |
| [**run**](run.md) | Execute pipelines |
| [**build**](build/index.md) | Generate pipelines, runners, and structures |

## Global flags

These flags work on any `pipelex` command and are position-agnostic — place them anywhere in the invocation:

- `--traceback` - print the full Rich-rendered stack trace before the friendly one-line error. Off by default. Both `pipelex run --traceback pipe ...` and `pipelex run pipe ... --traceback` work.
- `--no-logo` - suppress the startup logo banner. Handy for clean logs and to save tokens in agent output.

## Related CLI Surface

Package manifest management currently lives in the `mthds` CLI:

- [**package**](pkg.md) — `mthds package init`, `mthds package list`, and `mthds package validate`

## Usage Tips

1. **Initial Setup**

    - Run `pipelex init` to create configuration files and select your backends
    - Configure your AI providers in `.pipelex/inference/backends.toml`

2. **Development Workflow**

    - Write or generate pipelines in `.mthds` files
    - Validate with `pipelex validate pipe your_pipe_code` or `pipelex validate bundle your_bundle.mthds` during development
    - When validation errors carry a `💡 Suggested fix:` line, apply them automatically with `pipelex fix bundle your_bundle.mthds`
    - Run `pipelex validate pipe --all` before committing changes

3. **Running Pipelines**

    - Use `pipelex show pipe --all` to see available pipes
    - Use `pipelex show pipe pipe_code` to inspect pipe details
    - Run with `pipelex run pipe pipe_code`, add the required inputs using `--inputs`

4. **Configuration Management**

    - Use `pipelex show config` to verify current settings
    - Use `pipelex show backends` to check inference backend setup
    - Use `pipelex show models backend_name` to see available models

## Related Documentation

- [Configure AI Providers](../../get-started/configure-ai-providers.md) - Set up LLM backends
- [Design and Run Pipelines](../../building-methods/pipes/index.md) - Pipeline development guide
- [Packages](../../building-methods/packages.md) - `METHODS.toml` manifests and exports



## Init

# Init Commands

Initialize Pipelex configuration files and related setup flows.

## Initialize Configuration

```bash
pipelex init [FOCUS]
pipelex init --local [FOCUS]
```

By default, `pipelex init` writes to the global config directory at `~/.pipelex/`. Use `--local` to create a project-level `.pipelex/` directory at the detected project root. Credentials and Gateway service terms always remain in the global `~/.pipelex/` directory regardless of `--local`.

!!! note "Config updates not yet supported"
    The `pipelex init` command always performs a full reset of the configuration. Incremental config updates will be supported in a future release.

**Arguments:**

- `FOCUS` - What to initialize (optional):
    - `all` (default) - Initialize everything
    - `agreement` - Review or accept Pipelex Gateway terms
    - `config` - Only configuration files
    - `credentials` - Prompt for missing credentials only
    - `inference` - Only inference backend setup
    - `routing` - Only routing profile setup
    - `telemetry` - Only telemetry configuration

**Examples:**

```bash
# Initialize global config (recommended for first-time setup)
pipelex init

# Initialize project-local config
pipelex init --local

# Initialize only configuration files
pipelex init config

# Prompt only for missing credentials
pipelex init credentials

# Reconfigure inference backends
pipelex init inference

# Reconfigure telemetry settings
pipelex init telemetry

# Review gateway terms
pipelex init agreement
```

## What Gets Initialized

This command creates or resets a Pipelex config directory with:

- **pipelex.toml** - Main configuration file for logging, reporting, etc.
- **inference/** - AI backend and routing configuration
    - `backends.toml` - Backend provider settings
    - `routing_profiles.toml` - Model routing rules
    - `backends/` - Individual backend configuration files
    - `deck/` - AI model aliases and presets
- **telemetry.toml** - Telemetry and observability settings

!!! note "Offline cache priming"
    When Pipelex Gateway is enabled, `pipelex init` also primes an on-disk copy of the gateway's remote config at `~/.pipelex/cache/remote_config.json`. This lets later setup, validation, and dry-runs work even when the remote config service is briefly unreachable. If you run `pipelex init` while offline, it prints a yellow warning and skips priming — re-run it online to enable offline dry-runs.

## Interactive Setup Flow

When you run `pipelex init`, Pipelex can guide you through:

1. **Config reset** - Recreate the selected config files
2. **Backend selection** - Choose which AI providers to enable
3. **Credential prompts** - Fill in missing keys when relevant
4. **Routing configuration** - Set up how models are routed to backends
5. **Telemetry setup** - Configure observability and analytics

## Non-Interactive Init (`pipelex-agent init`)

For automated or agent-driven setups, use the `pipelex-agent` CLI:

```bash
pipelex-agent init [--config/-c JSON] [--global/-g]
```

**Target directory:**

- **Default:** project-level `.pipelex/` at the detected project root (looks for `.git`, `pyproject.toml`, etc.). Errors out if no project root is found.
- **`--global`/`-g`:** forces `~/.pipelex/`.

**Config JSON schema:**

```json
{
  "backends": ["openai", "anthropic"],
  "primary_backend": "openai",
  "accept_gateway_terms": true
}
```

All fields are optional:

| Field | Type | Description |
|-------|------|-------------|
| `backends` | `list[str]` | Backend keys to enable (e.g. `openai`, `anthropic`, `pipelex_gateway`). Omit to keep template defaults. |
| `primary_backend` | `str` | Required only when 2+ backends are selected and `pipelex_gateway` is not among them. |
| `accept_gateway_terms` | `bool` | Required when `pipelex_gateway` is in backends. |

Telemetry is not configured via `--config`: init seeds a `telemetry.toml` from a template (a global init writes an active one; a project init drops a commented-out one).

**Examples:**

```bash
# Initialize with OpenAI backend (project-level)
pipelex-agent init --config '{"backends": ["openai"]}'

# Initialize globally with gateway
pipelex-agent init -g --config '{"backends": ["pipelex_gateway"], "accept_gateway_terms": true}'
```

## Related Configuration

- [Configure AI Providers](../../get-started/configure-ai-providers.md)
- [Inference Backend Configuration](../../configuration/config-technical/inference-backend-config.md)
- [Telemetry Configuration](../../configuration/config-practical/telemetry-config.md)



## Update

# Update Command

Refresh the installed model deck so it matches the kit shipped with your current `pipelex` version.

```bash
pipelex update [OPTIONS]
pipelex update --local [OPTIONS]
```

When a new `pipelex` release ships changes to the model deck — new models, new aliases, retired entries — your previously installed deck under `~/.pipelex/inference/deck/` will fall behind. `pipelex update` brings it up to date without nuking your overrides.

## When to run it

Pipelex prints a one-line yellow advisory on most CLI invocations when it detects that the installed deck is older than the running package, or when no deck manifest is present yet (it is suppressed for `login`, `init`, `doctor`, `update`, and `which`):

```text
⚠ Pipelex model deck may be out of date — run pipelex update to refresh
```

Run `pipelex update` to clear the warning. To silence it permanently for a session or environment:

```bash
export PIPELEX_NO_DECK_NOTICE=1
```

## What it does

Pipelex tracks the deck install state in a small JSON manifest at `~/.pipelex/inference/deck/.kit_manifest.json` (or `.pipelex/inference/deck/.kit_manifest.json` for project-local installs). The manifest stores the kit version that produced the install and a SHA-256 of each managed deck file at install time.

On `update`, Pipelex compares three states — what the kit ships, what is on disk, what the manifest recorded — and produces a per-file plan:

| Status | Meaning | Action |
|---|---|---|
| `up-to-date` | Installed file matches the kit | none |
| `new` | Kit ships a new managed file not yet installed | install from kit |
| `behind` | Installed file unchanged from manifest, kit has newer content | overwrite from kit |
| `locally modified` | Installed file was edited after install — kit version differs | back up + overwrite |
| `removed upstream` | Kit no longer ships this file | back up + remove |

Locally-modified files are preserved as `<file>.bak.<UTC-timestamp>` before the kit version is written in their place. Removed files get the same backup treatment before deletion. After the run, Pipelex writes a fresh manifest stamped with the new kit version.

## Managed files vs. user overrides

Pipelex draws a sharp line between deck files it owns and deck files you own.

- **Managed (pipelex-owned)** — `1_llm_deck.toml`, `2_img_gen_deck.toml`, `3_extract_deck.toml`, `4_search_deck.toml`. These are refreshed by `pipelex update`. Local edits are preserved with a `.bak.<timestamp>` backup but will not survive future updates.
- **User overrides** — any file in the deck directory whose name starts with `x_custom_` (e.g. `x_custom_llm_deck.toml`, `x_custom_extract_deck.toml`). Pipelex never tracks, hashes, copies, or removes these. Add new ones whenever you need to override aliases, presets, or default choices for a backend.

The deck loader merges all `*.toml` files under the deck directory in alphabetical order via deep-merge, so your `x_custom_*.toml` always wins over the numbered defaults.

## Options

| Flag | Description |
|---|---|
| `--local` / `-l` | Update the project-local `.pipelex/` deck instead of the global `~/.pipelex/` |
| `--yes` / `-y` | Apply updates non-interactively (skip the confirmation prompt) |
| `--dry-run` | Show the plan without modifying any file |
| `--no-backup` | Do not create `.bak` files for locally-modified deck files |

## Examples

```bash
# Preview what would change
pipelex update --dry-run

# Interactive update of the global deck
pipelex update

# Non-interactive update — useful for CI or scripts
pipelex update --yes

# Project-local deck only
pipelex update --local --yes

# Skip the .bak backups (advanced — your edits will be lost permanently)
pipelex update --yes --no-backup
```

## Migration: existing installs without a manifest

If your deck was installed before the `update` command existed, there will be no `.kit_manifest.json` next to your deck files. The first `pipelex update` run reports each managed file as `locally modified` (no provenance proof) and asks before overwriting. Two ways to migrate cleanly:

1. **Trust the kit** — run `pipelex update --yes` to take the upstream version of every managed file. Anything you want to keep can be moved into an `x_custom_*.toml` afterwards using the `.bak.<timestamp>` files as a reference.
2. **Move overrides first** — copy any custom aliases, presets, or defaults from your numbered files into a new `x_custom_<area>_deck.toml`, then run `pipelex update --yes`. Your customizations will live in the file Pipelex never touches.

After either path, subsequent updates land cleanly without prompting on the unchanged files.

## Diagnostics

`pipelex doctor` includes a **Model Deck** section that reports the same per-file status as `update`. Run `pipelex doctor --fix` to be offered an interactive `pipelex update` as part of the standard fix flow.

## Related

- [Init Commands](init.md) — first-time setup that materializes the deck and its manifest
- [LLM Providers & Models Configuration](../../configuration/config-technical/inference-backend-config.md) — what the deck files actually configure


## Validate

# Validate Commands

Validate your pipeline definitions and configuration for correctness.

The `validate` command has three subcommands depending on what you want to validate:

```bash
pipelex validate pipe ...      # Validate a pipe from your project's library
pipelex validate bundle ...    # Validate a bundle file or directory
pipelex validate method ...    # Validate an installed method package
```

!!! tip "Shorthand"
    `pipelex validate --all` and `pipelex validate <pipe_code>` are shortcuts that default to the `pipe` subcommand. You can omit `pipe` for convenience.

## Validate Pipe

```bash
pipelex validate pipe <PIPE_CODE>
pipelex validate pipe --all
```

Validates and dry-runs a specific pipe from your imported packages, or all pipes at once.

**Arguments:**

- `PIPE_CODE` - The pipe code to validate (omit when using `--all`)

**Options:**

- `--all`, `-a` - Validate all discovered pipes
- `--library-dir`, `-L` - Directory to search for pipe definitions. Can be specified multiple times.
- `--allow-signatures` - Accept `PipeSignature` placeholders in the dependency graph (lenient mode). See [Signature Pipes](../../building-methods/pipes/signature-pipes.md).

**Examples:**

```bash
# Validate a specific pipe
pipelex validate analyze_cv_matching
pipelex validate write_weekly_report

# Validate all pipes
pipelex validate --all

# Explicit subcommand form also works
pipelex validate pipe --all

# Validate with custom library directories
pipelex validate my_pipe -L ./pipelines
pipelex validate --all -L ./pipelines -L ./shared_pipes

# Allow PipeSignature placeholders during dry-run
pipelex validate my_draft_pipe --allow-signatures
pipelex validate --all --allow-signatures
```

## Validate Bundle

```bash
pipelex validate bundle <PATH>
```

Validates all pipes defined in a bundle file (`.mthds`) or a pipeline directory. When a directory is given, the bundle file is auto-detected inside it.

**Arguments:**

- `PATH` - Path to a `.mthds` bundle file or a directory containing one

**Options:**

- `--library-dir`, `-L` - Directory to search for additional pipe definitions. Can be specified multiple times.
- `--allow-signatures` - Accept `PipeSignature` placeholders in the dependency graph (lenient mode). See [Signature Pipes](../../building-methods/pipes/signature-pipes.md).

**Examples:**

```bash
# Validate a bundle file
pipelex validate bundle my_pipeline.mthds
pipelex validate bundle pipelines/invoice_processor.mthds

# Validate a pipeline directory (auto-detects the bundle file)
pipelex validate bundle pipelines/invoice_processor/

# Validate with additional library directories
pipelex validate bundle my_bundle.mthds -L ./shared_pipes

# Allow PipeSignature placeholders during dry-run
pipelex validate bundle methods/draft_pipeline.mthds --allow-signatures
```

!!! note
    When validating a bundle, ALL pipes in that bundle are validated, not just the main pipe.

## Validate Method

```bash
pipelex validate method <NAME>
```

Validates all pipes in an installed method package.

**Arguments:**

- `NAME` - The name of the installed method to validate

**Options:**

- `--pipe PIPE_CODE` - Validate only a specific pipe within the method
- `--library-dir`, `-L` - Directory to search for additional pipe definitions. Can be specified multiple times.

**Examples:**

```bash
# Validate an installed method
pipelex validate method invoice_extractor

# Validate a specific pipe within a method
pipelex validate method invoice_extractor --pipe extract_amounts
```

## Suggested Fixes

When a validation error has a deterministic safe fix, the error output includes a `💡 Suggested fix:` line describing the change, and the report ends with the exact command to apply every suggested fix automatically:

```text
💡 1 of these errors can be fixed automatically — run: pipelex fix bundle my_pipeline.mthds
```

See [Fix Commands](fix.md) for `pipelex fix bundle`, including the `--diff` preview.

## What Validation Checks

All validation commands check:

- Syntax correctness of `.mthds` files
- Concept and pipe definitions are valid
- Input/output connections are correct
- All referenced pipes and concepts exist
- Dry-run execution succeeds without errors, which implies the logic is correct and the pipe can be run

## Related Configuration

- [Dry Run Configuration](../../configuration/config-pipeline-validation/dry-run-config.md)


## Fix

# Fix Commands

Apply deterministic, safe fixes to a bundle and re-validate until it is valid.

```bash
pipelex fix bundle ...    # Fix a bundle file or directory in place
```

When `pipelex validate` reports errors that have a deterministic fix, each one carries a `💡 Suggested fix:` line and the output ends with the exact `pipelex fix bundle` command to run. The fix command runs a validate → apply safe fixes → re-validate loop until the bundle is valid, out of fixes, or the iteration cap is reached — and names every change it makes.

## Fix Bundle

```bash
pipelex fix bundle <PATH>
```

Fixes a bundle file (`.mthds`) or a pipeline directory in place. When a directory is given, the bundle file is auto-detected inside it — exactly the same resolution as `pipelex validate bundle`, so fix patches exactly what validate judged.

**Arguments:**

- `PATH` - Path to a `.mthds` bundle file or a directory containing one

**Options:**

- `--library-dir`, `-L` - Directory to search for additional pipe definitions. Can be specified multiple times. Files under these directories are inside the fix write scope; files loaded any other way are never written.
- `--diff` - Preview: show the changes as a unified diff without writing any file. Exit codes keep the same verdict semantics, so `--diff` answers "would it converge?"
- `--select` - Only apply the named fix rule code (repeatable)
- `--ignore` - Skip the named fix rule code (repeatable; mutually exclusive with `--select`)
- `--max-iterations` - Maximum fix-apply rounds before reporting non-convergence
- `--allow-signatures` - Accept `PipeSignature` placeholders in the dependency graph (lenient mode). See [Signature Pipes](../../building-methods/pipes/signature-pipes.md).

**Examples:**

```bash
# Fix a bundle file in place
pipelex fix bundle my_pipeline.mthds

# Preview the changes without writing anything
pipelex fix bundle my_pipeline.mthds --diff

# Fix a pipeline directory (auto-detects the bundle file)
pipelex fix bundle pipelines/invoice_processor/

# Fix with additional library directories (also widens the write scope)
pipelex fix bundle my_bundle.mthds -L ./shared_pipes

# Only apply one fix rule
pipelex fix bundle my_bundle.mthds --select match-sequence-output
```

## Exit Codes

- `0` - The bundle is valid after the run (fixed, or already valid)
- `1` - Negative verdict: the bundle is still invalid, or valid but not runnable (pending `PipeSignature` placeholders without `--allow-signatures`)
- `2` - No verdict: bad path, ambiguous bundle directory, invalid rule filter, or an unexpected failure

## What Gets Fixed

Only fixes classified as SAFE are applied — deterministic corrections derived from the structured validation errors, never from guesswork. Errors without a safe deterministic fix are left in place and reported as remaining errors. The available fix rule codes (for `--select`/`--ignore`) are listed in the error message when you pass an unknown code.

## Related Documentation

- [Validate Commands](validate.md) - Where suggested fixes are surfaced
- [Error Model — suggested fixes](../../under-the-hood/error-model.md#suggested_fix-structured-deterministic-fixes) - The `SuggestedFix` wire model, the fix planner, and the patch-op vocabulary


## Run

# Run Commands

Execute a pipeline with optional inputs and outputs.

The `run` command has three subcommands depending on where your pipeline is defined:

```bash
pipelex run pipe ...      # Run a pipe from your project's library
pipelex run bundle ...    # Run a pipeline from a bundle file or directory
pipelex run method ...    # Run an installed method package
```

## Run Pipe

```bash
pipelex run pipe <PIPE_CODE> [OPTIONS]
```

Runs a pipe by code from your project's pipe library.

**Arguments:**

- `PIPE_CODE` - The pipe code to run

**Options:**

- `--inputs`, `-i` - Path to a JSON or TOML file containing inputs (discriminated by file extension — see [Input File Formats](#input-file-formats))
- `--output-dir`, `-o` - Directory to save outputs (defaults to `results/`)
- `--save-main-stuff` / `--no-save-main-stuff` - Whether to save the main output to a file
- `--save-working-memory` / `--no-save-working-memory` - Whether to save the full working memory
- `--working-memory-path` - Custom path for the working memory output file
- `--save-csv` - Write the main stuff to this literal CSV path (**not** under `--output-dir`; absolute or `~`/relative paths all work). Requires a flat list output. See [CSV Input & Output](../../building-methods/pipes/csv-input-and-output.md)
- `--no-pretty-print` - Skip pretty printing the main output
- `--graph` / `--no-graph` - Enable/disable execution graph visualization
- `--graph-full-data` / `--graph-no-data` - Include full data in the graph visualization
- `--dry-run` - Dry-run the pipeline without calling AI providers
- `--mock-inputs` - Use mock inputs for the pipeline (requires `--dry-run`)
- `--library-dir`, `-L` - Directory to search for pipe definitions. Can be specified multiple times.

**Examples:**

```bash
# Run a pipe by code
pipelex run pipe hello_world

# Run with inputs from JSON file
pipelex run pipe write_weekly_report --inputs weekly_report_data.json

# Run with custom output directory
pipelex run pipe hello_world --output-dir my_output/

# Run without saving or pretty printing
pipelex run pipe my_pipe --no-save-main-stuff --no-pretty-print

# Write a flat list output to CSV (literal path, not under --output-dir)
pipelex run pipe summarize_people --inputs people.json --save-csv summaries.csv

# Dry-run (no AI calls)
pipelex run pipe my_pipe --dry-run

# Run with custom library directories
pipelex run pipe my_pipe -L ./pipelines -L ./shared_pipes
```

## Run Bundle

```bash
pipelex run bundle <PATH> [OPTIONS]
```

Runs a pipeline from a bundle file (`.mthds`) or a pipeline directory. When a directory is given, the bundle file is auto-detected inside it — and so is an inputs file (`inputs.json` or `inputs.toml`) when `--inputs` is not passed (see [Input File Formats](#input-file-formats)).

**Arguments:**

- `PATH` - Path to a `.mthds` bundle file or a directory containing one

**Options:**

- `--pipe PIPE_CODE` - Run a specific pipe from the bundle (defaults to the bundle's main pipe)
- `--inputs`, `-i` - Path to a JSON or TOML file containing inputs (discriminated by file extension — see [Input File Formats](#input-file-formats))
- `--output-dir`, `-o` - Directory to save outputs
- `--save-main-stuff` / `--no-save-main-stuff` - Whether to save the main output
- `--save-working-memory` / `--no-save-working-memory` - Whether to save the full working memory
- `--working-memory-path` - Custom path for the working memory output file
- `--save-csv` - Write the main stuff to this literal CSV path (**not** under `--output-dir`; absolute or `~`/relative paths all work). Requires a flat list output. See [CSV Input & Output](../../building-methods/pipes/csv-input-and-output.md)
- `--no-pretty-print` - Skip pretty printing the main output
- `--graph` / `--no-graph` - Enable/disable execution graph visualization
- `--graph-full-data` / `--graph-no-data` - Include full data in the graph visualization
- `--dry-run` - Dry-run the pipeline without calling AI providers
- `--mock-inputs` - Use mock inputs for the pipeline (requires `--dry-run`)
- `--library-dir`, `-L` - Directory to search for additional pipe definitions. Can be specified multiple times.

**Examples:**

```bash
# Run a bundle file (uses its main_pipe)
pipelex run bundle my_bundle.mthds

# Run a pipeline directory
pipelex run bundle pipelines/invoice_processor/

# Run a specific pipe from a bundle
pipelex run bundle my_bundle.mthds --pipe extract_invoice

# Run with inputs
pipelex run bundle my_bundle.mthds --inputs invoice_data.json

# Run with execution graph
pipelex run bundle my_bundle.mthds --graph
```

## Run Method

```bash
pipelex run method <NAME> [OPTIONS]
```

Runs a pipeline from an installed method package.

**Arguments:**

- `NAME` - The name of the installed method to run

**Options:**

- `--pipe PIPE_CODE` - Run a specific pipe within the method (defaults to the method's main pipe)
- `--inputs`, `-i` - Path to a JSON or TOML file containing inputs (discriminated by file extension — see [Input File Formats](#input-file-formats))
- `--output-dir`, `-o` - Directory to save outputs
- `--save-main-stuff` / `--no-save-main-stuff` - Whether to save the main output
- `--save-working-memory` / `--no-save-working-memory` - Whether to save the full working memory
- `--working-memory-path` - Custom path for the working memory output file
- `--save-csv` - Write the main stuff to this literal CSV path (**not** under `--output-dir`; absolute or `~`/relative paths all work). Requires a flat list output. See [CSV Input & Output](../../building-methods/pipes/csv-input-and-output.md)
- `--no-pretty-print` - Skip pretty printing the main output
- `--graph` / `--no-graph` - Enable/disable execution graph visualization
- `--graph-full-data` / `--graph-no-data` - Include full data in the graph visualization
- `--dry-run` - Dry-run the pipeline without calling AI providers
- `--mock-inputs` - Use mock inputs for the pipeline (requires `--dry-run`)
- `--library-dir`, `-L` - Directory to search for additional pipe definitions. Can be specified multiple times.

**Examples:**

```bash
# Run an installed method
pipelex run method invoice_extractor

# Run a specific pipe within a method
pipelex run method invoice_extractor --pipe extract_amounts

# Run with inputs
pipelex run method invoice_extractor --inputs invoice_data.json
```

## Input File Formats

An inputs file is a dictionary whose keys are input variable names, and each value is interpreted **against the pipe's declared signature** — so you provide the values directly (a string, a number, an object) and Pipelex types them as the declared concept. See [Providing Inputs](../../building-methods/pipes/provide-inputs.md) for the full model, including the explicit `{concept, content}` escape hatch. Pipelex accepts **both JSON and TOML**, discriminated by the file extension:

- A `.toml` suffix is parsed as TOML.
- Every other value — `.json`, no extension, anything else — is parsed as JSON.

There is no content sniffing: the extension alone decides. Inline JSON passed to `--inputs` (a value starting with `{`) stays JSON-only.

An input declared optional (`?`) on the entry pipe may be omitted — the run records a `not_provided` absence for it instead of failing (see [Understanding Optionality](../../building-methods/pipes/understanding-optionality.md)).

Both formats produce the same input dictionary for the value types they share, so the [PipelineInputs shapes](../../building-methods/pipes/provide-inputs.md) apply to both — with one structural exception: JSON `null` has no TOML equivalent (TOML has no null type).

### JSON

Provide each value directly — a string, a number, an object — and it is typed as the input's declared concept:

```json
{
  "instructions": "simple string value",
  "priority": 3,
  "client": { "name": "Acme Corp", "country": "France" }
}
```

To override or disambiguate a concept, wrap a value in the explicit `{concept, content}` envelope (`{"concept": "domain_code.ConceptName", "content": {...}}`) — see [Providing Inputs](../../building-methods/pipes/provide-inputs.md#the-explicit-format-escape-hatch).

### TOML

The same inputs in TOML. TOML's multi-line strings (`"""..."""`) make text-heavy inputs far more pleasant to author than escaped JSON:

```toml
instructions = """
Focus on payment terms.
Flag anything that looks unusual.
"""

priority = 3

[client]
name = "Acme Corp"
country = "France"
```

!!! tip "TOML temporal literals are native `Date` / `Time` inputs"
    A top-level TOML date or datetime literal (e.g. `hearing = 2026-09-01` or `departure = 2026-07-07T15:40:00+02:00`) maps directly to the native [`Date`](../../building-methods/concepts/native-concepts.md) concept — the offset is kept when stated. A bare *time-of-day* literal (`opening = 09:00:00`) maps to the native `Time` concept; it never silently becomes a `Date` (a time alone has no date to attach to, so shaping a `Time` into a `Date` slot fails the compatibility check).

### Auto-detection and ambiguity

`run bundle <dir>` auto-detects a default inputs file inside the directory when `--inputs` is omitted: it looks for `inputs.json`, then `inputs.toml`. If **both** exist, the run fails with an ambiguity error asking you to pass `--inputs` explicitly — passing `--inputs` always bypasses auto-detection.

## Output Format

The output JSON contains the complete working memory after pipeline execution, including all intermediate results and the final output.

### Absent main output

A run whose main output resolves as a recorded absence (an optional `?` output that produced nothing — e.g. a `PipeCondition` `continue` outcome, or a skipped producer) is a **successful** run. The CLI prints the absence with its reason, and `--save-main-stuff` writes an explicit absence artifact instead of a value dump: `main_stuff.json` is `{"absent": true, ...}` with the absence record, and `main_stuff.md` is a human-readable summary including the provenance chain (no HTML render or interactive viewer is produced — there is nothing to view). `--save-csv` fails with an explicit error and a non-zero exit code, the same way it does for any main output that is not a flat list — there is no tabular value to save.

## Related Documentation

- [Executing Pipelines](../../building-methods/pipes/executing-pipelines.md)
- [Providing Inputs to Pipelines](../../building-methods/pipes/provide-inputs.md)
- [CSV Input & Output](../../building-methods/pipes/csv-input-and-output.md)
- [Design and Run Pipelines](../../building-methods/pipes/index.md)


## Show

# Show Commands

Inspect your Pipelex configuration, pipes, and available AI models.

## Show Configuration

```bash
pipelex show config
```

Displays the current Pipelex configuration loaded from all sources (default config overridden by user config).

**Examples:**

```bash
# Display current configuration
pipelex show config
```

!!! note
    This shows the main Pipelex configuration but not the inference backend details. Use `pipelex show backends` for backend configuration.

## List All Pipes

```bash
pipelex show pipe --all
```

Lists all pipes discovered in your project and imported packages, showing their pipe codes and basic information.

**Examples:**

```bash
# List all available pipes
pipelex show pipe --all
```

This includes:

- Internal Pipelex pipes (like the pipe builder)
- Pipes from your project's `.mthds` files
- Pipes that are part of imported packages

## Show Pipe Definition

```bash
pipelex show pipe PIPE_CODE
```

Displays the complete definition of a specific pipe including inputs, outputs, prompts, model settings, and all configuration.

**Arguments:**

- `PIPE_CODE` - The pipe code to inspect

**Examples:**

```bash
# Show pipe definition
pipelex show pipe analyze_cv_matching
pipelex show pipe write_weekly_report
```

## List AI Models

```bash
pipelex show models BACKEND_NAME [OPTIONS]
```

Lists all available models from a configured backend provider by querying the provider's API.

**Arguments:**

- `BACKEND_NAME` - The backend to query (e.g., `openai`, `anthropic`, `mistral`)

**Options:**

- `--flat`, `-f` - Output in flat CSV format for easy copying into other configuration files

**Examples:**

```bash
# List models
pipelex show models openai
pipelex show models mistral

# List models in flat format
pipelex show models anthropic --flat
```

**Use case:** When configuring new models in your deck, use this command to see what models are available from each provider.

## Show Backends

```bash
pipelex show backends [OPTIONS]
```

Displays all configured inference backends and the active routing profile with its routing rules.

**Options:**

- `--all`, `-a` - Show all backends including disabled ones (by default, only enabled backends are shown)

**Examples:**

```bash
# Show enabled backends and routing profile
pipelex show backends

# Show all backends including disabled ones
pipelex show backends --all
```

**What it displays:**

- Table of configured backends with status, endpoint, and model count
- Active routing profile name and description
- Default backend for the profile
- Routing rules mapping model patterns to backends

## Related Configuration

- [Inference Backend Configuration](../../configuration/config-technical/inference-backend-config.md)
- Backend configuration files: `.pipelex/inference/backends.toml`
- Routing configuration: `.pipelex/inference/routing_profiles.toml`



## Pkg

# Package Commands

Package manifest management is currently exposed through the `mthds` CLI, not the `pipelex` CLI.

Use these commands when you need to create, inspect, or validate a `METHODS.toml` file for a method package.

## Overview

```bash
mthds package init
mthds package list
mthds package validate
```

All three commands also accept `-C, --package-dir <path>` to target a package directory other than the current one.

## `mthds package init`

```bash
mthds package init
mthds package init -C ./my-package
```

Create a new `METHODS.toml` manifest interactively.

The command prompts for:

- package `address`
- `version`
- `description`
- optional `authors`
- optional `license`

It writes a manifest with package metadata and an empty `[exports]` section. If `METHODS.toml` already exists, the CLI asks whether to overwrite it.

## `mthds package list`

```bash
mthds package list
mthds package list -C ./my-package
```

Read and display the current `METHODS.toml` manifest from the target directory.

The output includes:

- package address, version, and description
- optional display name, authors, and license
- declared exports by domain

If no `METHODS.toml` exists in the target directory, the command reports an error and suggests running `mthds package init`.

## `mthds package validate`

```bash
mthds package validate
mthds package validate -C ./my-package
```

Validate the `METHODS.toml` manifest in the target directory.

This checks manifest syntax and package metadata fields, then reports the parsed manifest back on success.

## Current Scope

The public package-management commands currently documented in this repo are:

- `mthds package init`
- `mthds package list`
- `mthds package validate`

Older `pipelex pkg ...` examples and dependency-management commands such as `add`, `lock`, `install`, and `update` are not part of the current Python `pipelex` CLI surface and should not be used here.

## Related Documentation

- [Packages](../../building-methods/packages.md) — Package manifest concepts and `METHODS.toml` structure
- [CLI](index.md) — Runtime CLI vs `mthds` CLI overview


## Overview

# Build Commands

Generate pipelines, runner code, Python structures, and JSON templates from natural language descriptions.

## Available Commands

| Command | Description |
|---------|-------------|
| [**build runner**](runner.md) | Generate Python code to run a pipe |
| [**build structures**](structures.md) | Generate the structures (Pydantic models) of your concepts |
| [**build inputs**](inputs.md) | Generate example input JSON for a pipe |
| [**build output**](output.md) | Generate example output representation for a pipe (JSON, Python, or JSON Schema) |


## Runner

# Build Runner

Generate Python code to run a pipe with all necessary imports and example inputs.

## Usage

### From a bundle

```bash
pipelex build runner bundle <PATH> [OPTIONS]
```

**Arguments:**

- `PATH` - Path to a `.mthds` bundle file or a pipeline directory

**Options:**

- `--pipe` - Pipe code to use (optional if the `.mthds` declares a `main_pipe`)
- `--output`, `-o` - Path to save the generated Python file (defaults to target's directory)
- `--library-dirs`, `-L` - Directories to search for pipe definitions. Can be specified multiple times.

### From an installed method

```bash
pipelex build runner method <NAME> [OPTIONS]
```

**Arguments:**

- `NAME` - Name of the installed method

**Options:**

- `--pipe` - Pipe code (overrides method's `main_pipe`)
- `--output`, `-o` - Path to save the generated Python file
- `--library-dirs`, `-L` - Directories to search for pipe definitions. Can be specified multiple times.

## Examples

**Generate runner from a bundle (uses main_pipe):**

```bash
pipelex build runner bundle my_bundle.mthds
```

**Generate runner from a pipeline directory:**

```bash
pipelex build runner bundle pipeline_01/
```

**Specify which pipe to use from a bundle:**

```bash
pipelex build runner bundle my_bundle.mthds --pipe my_pipe
```

**With additional library directories:**

```bash
pipelex build runner bundle my_bundle.mthds -L ./shared_pipes/ -L ./common/
```

**Custom output path:**

```bash
pipelex build runner bundle my_bundle.mthds --output custom_runner.py
```

## What Gets Generated

The generated Python file includes:

1. **All necessary imports** - Imports for Pipelex execution, content types, and any custom structures (`from structures.structures import …`)
2. **Generated Pydantic structures** - A stamped `structures/structures.py` module (plus its `codegen.lock`) emitted by the codegen engine, same as `pipelex build structures`
3. **Input memory setup** - Example input values based on the pipe's input types
4. **Pipeline execution** - Async function that executes the pipeline
5. **Output handling** - Code to extract and display the results
6. **Main execution block** - Pipelex initialization and asyncio setup

## Input Type Handling

- **Native concepts** (Text, Image, Document, etc.) - Automatically generates appropriate content objects
- **Custom concepts** - Recursively generates the structure with example values
- **Structured content** - Creates example data matching the concept's fields

## Using the Generated Runner

After generating the runner file:

1. Open the generated Python file
2. Review and customize the example input values
3. Run the script: `python results/run_{pipe_code}.py`
4. Iterate and adjust as needed

## Related Documentation

- [Build Structures](structures.md) - Generate Pydantic models separately
- [Build Inputs](inputs.md) - Generate example input JSON for a pipe
- [Build Output](output.md) - Generate example output JSON for a pipe
- [Run Command](../run.md) - Run pipes directly from CLI


## Structures

# Build Structures

Generate Pydantic models from your concept definitions.

!!! note "Alias of `codegen types`"
    `pipelex build structures` is an alias of [`pipelex codegen types --target python-structures`](../../../under-the-hood/codegen-projections.md): it resolves your bundles into a normalized library crate and emits a single stamped `structures.py` module plus a `codegen.lock`. Use `pipelex codegen check` to verify the generated files are current without regenerating.

## Usage

```bash
pipelex build structures <TARGET> [OPTIONS]
```

**Arguments:**

- `TARGET` - Either a library directory containing `.mthds` files, or a specific `.mthds` file (its directory is used as the closure)

**Options:**

- `-o`, `--output-dir` - Output directory for the generated module (defaults to `structures/` in target's directory)
- `-L`, `--library-dir` - Additional directory of `.mthds` bundles to load (repeatable)

## Examples

**Generate structures from a library directory:**

```bash
pipelex build structures ./my_pipelines/
```

**Generate structures from a specific bundle file:**

```bash
pipelex build structures ./my_pipeline/bundle.mthds
```

**Generate structures to a specific output directory:**

```bash
pipelex build structures ./my_pipelines/ -o ./generated/
```

## What Gets Generated

One `structures.py` module for the whole closure, plus a `codegen.lock`:

- **Pydantic model classes** - One `StructuredContent` subclass per concept, with types, constraints, and descriptions
- **Bare-when-unique names** - A concept `demo.Report` becomes `class Report`; only a code that collides across domains gets the domain-qualified spelling
- **Stamp + lock** - Each file carries a self-describing codegen stamp (crate fingerprint, engine version, content hash) and is tracked in `codegen.lock`, so `pipelex codegen check` can detect drift offline
- **Declared imprecision** - A concept whose shape cannot be fully derived is surfaced with an explicit caveat, never a silent guess

Generated files are never meant to be edited: to customize a type, subclass it in a sibling module (the generated header shows how) — subclasses survive regeneration.

## Why Use Generated Structures?

- **Use as type hints** - Get IDE autocompletion and type checking in your code
- **Add custom logic** - Subclass the generated classes in your own module
- **Integrate in your app** - Use the models directly in your application
- **Trust the drift check** - `pipelex codegen check` tells you when the models no longer match the method

## Example Output

For a concept defined in a `.mthds` file like:

```toml
[concept.CandidateProfile]
description = "A structured representation of a candidate's professional background."

[concept.CandidateProfile.structure]
skills = { type = "text", description = "Technical and soft skills", required = true }
years_of_experience = { type = "number", description = "Total years of experience" }
education = { type = "text", description = "Educational background", required = true }
work_history = { type = "text", description = "Summary of previous roles", required = true }
```

You get (inside the stamped `structures.py`):

```python
class CandidateProfile(StructuredContent):
    """A structured representation of a candidate's professional background."""

    skills: str = Field(..., description="Technical and soft skills")
    years_of_experience: Optional[float] = Field(default=None, description="Total years of experience")
    education: str = Field(..., description="Educational background")
    work_history: str = Field(..., description="Summary of previous roles")
```

## Related Documentation

- [Codegen projections](../../../under-the-hood/codegen-projections.md) - The engine behind this command (`codegen types`, stamps, lock, offline check)
- [Build Runner](runner.md) - Generate Python runner scripts
- [Build Inputs](inputs.md) - Generate example input JSON for a pipe
- [Build Output](output.md) - Generate example output JSON for a pipe
- [Concepts](../../../building-methods/concepts/define_your_concepts.md) - Understanding concept definitions


## Inputs

# Build Inputs

Generate an example inputs template for a pipe, showing the expected input structure based on the pipe's input types. The template is emitted as JSON by default, or as TOML with `--format toml`.

By default the template is the **light** shape — the values themselves, matching the pipe's declared signature (a bare string for a `Text`-refining input, a bare number for a `Number`-refining one, an object for a structured concept). This is exactly what [`pipelex run`](../run.md) accepts, so a template you generate and fill in runs as-is. Pass `--explicit` to emit the ceremonial `{concept, content}` envelope form instead. See [Providing Inputs](../../../building-methods/pipes/provide-inputs.md) for the full model.

## Usage

### By pipe code

```bash
pipelex build inputs pipe <PIPE_CODE> [OPTIONS]
```

**Arguments:**

- `PIPE_CODE` - The pipe code (e.g. `my_domain.my_pipe`)

**Options:**

- `--library-dir`, `-L` - Directory to search for pipe definitions. Can be specified multiple times.
- `--output`, `-o` - Path to save the generated inputs file (defaults to `results/`, filename `inputs.json` or `inputs.toml` depending on `--format`)
- `--format` - Template format: `json` (default) or `toml`
- `--explicit` - Emit the ceremonial `{concept, content}` envelope form instead of the light values

### From a bundle

```bash
pipelex build inputs bundle <PATH> [OPTIONS]
```

**Arguments:**

- `PATH` - Path to a `.mthds` bundle file or a pipeline directory

**Options:**

- `--pipe` - Pipe code to use (can be omitted if the bundle declares a `main_pipe`)
- `--library-dir`, `-L` - Directory to search for pipe definitions. Can be specified multiple times.
- `--output`, `-o` - Path to save the generated inputs file (defaults to the bundle's directory, filename `inputs.json` or `inputs.toml` depending on `--format`)
- `--format` - Template format: `json` (default) or `toml`
- `--explicit` - Emit the ceremonial `{concept, content}` envelope form instead of the light values

### From an installed method

```bash
pipelex build inputs method <NAME> [OPTIONS]
```

**Arguments:**

- `NAME` - Name of the installed method

**Options:**

- `--pipe` - Pipe code (overrides method's `main_pipe`)
- `--library-dir`, `-L` - Directory to search for pipe definitions. Can be specified multiple times.
- `--output`, `-o` - Path to save the generated inputs file (filename `inputs.json` or `inputs.toml` depending on `--format`)
- `--format` - Template format: `json` (default) or `toml`
- `--explicit` - Emit the ceremonial `{concept, content}` envelope form instead of the light values

## Examples

**Generate inputs for a pipe by code:**

```bash
pipelex build inputs pipe my_domain.my_pipe
```

**Generate inputs from a bundle (uses main_pipe):**

```bash
pipelex build inputs bundle my_bundle.mthds
```

**Generate inputs from a pipeline directory:**

```bash
pipelex build inputs bundle pipeline_01/
```

**Specify which pipe to use from a bundle:**

```bash
pipelex build inputs bundle my_bundle.mthds --pipe my_pipe
```

**Generate inputs for a pipe using a library directory:**

```bash
pipelex build inputs pipe my_domain.my_pipe -L ./my_library/
```

**Custom output path:**

```bash
pipelex build inputs bundle my_bundle.mthds --output custom_inputs.json
```

**Generate a TOML template instead of JSON:**

```bash
pipelex build inputs bundle my_bundle.mthds --format toml
```

When `--format toml` is selected and no explicit `--output` is given, the default filename becomes `inputs.toml` (instead of `inputs.json`).

**Generate the explicit envelope form instead of the light values:**

```bash
pipelex build inputs bundle my_bundle.mthds --explicit
```

## Output Format

The generated file contains all inputs required by the pipe, with example values based on each input's concept type. Both formats carry the same structure — [`pipelex run`](../run.md#input-file-formats) accepts either.

### Light form (default)

The default template is the light shape: each value is what the pipe's signature expects, with no `{concept, content}` envelope. For a pipe declaring a `Text`-refining `question`, a `Number`-refining `priority`, a structured `invoice`, and a `Text`-refining list `tags = "Tag[]"`:

**JSON (`--format json`, default):**

```json
{
  "question": "text_value",
  "priority": 1,
  "invoice": {
    "invoice_number": "invoice_number_value",
    "amount": 0.0
  },
  "tags": ["text_value"]
}
```

**TOML (`--format toml`):** the light TOML template uses inline tables so top-level scalars and structured values coexist, and carries the declared concept for each key as a comment (a declared-multiple input is tagged `[]`):

```toml
# concept: my_domain.Question
question = "text_value"
# concept: my_domain.Priority
priority = 1
# concept: my_domain.Invoice
invoice = {invoice_number = "invoice_number_value", amount = 0.0}
# concept: my_domain.Tag[]
tags = ["text_value"]
```

### Explicit envelope form (`--explicit`)

Pass `--explicit` to reproduce the ceremonial `{concept, content}` envelope for every input — useful when you want the concept written out inline (JSON has no comments), or when authoring an input meant for the explicit escape hatch:

```json
{
  "question": {
    "concept": "my_domain.Question",
    "content": {"text": "text_value"}
  },
  "priority": {
    "concept": "my_domain.Priority",
    "content": {"number": 1}
  },
  "invoice": {
    "concept": "my_domain.Invoice",
    "content": {"invoice_number": "invoice_number_value", "amount": 0.0}
  },
  "tags": {
    "concept": "my_domain.Tag",
    "content": [{"text": "text_value"}]
  }
}
```

In TOML, `--explicit` renders the all-tables layout; a multiple input keeps a single `[key]` envelope whose `content` is an array-of-tables (`[[key.content]]`):

```toml
[question]
concept = "my_domain.Question"

[question.content]
text = "text_value"

# ... priority and invoice tables omitted ...

[tags]
concept = "my_domain.Tag"

[[tags.content]]
text = "text_value"
```

### Multiplicity

When an input is declared multiple (`Tag[]`), the light form wraps the example value in a list (`["text_value"]`); the explicit form keeps a single `{concept, content}` envelope whose `content` is the list (not an array of separate envelopes). Either way, [`pipelex run`](../run.md) accepts the filled-in template.

## Use Cases

- **Understanding pipe requirements** - Quickly see what inputs a pipe expects
- **Creating input templates** - Generate a starting point for your input JSON files
- **API integration** - Know the exact structure to send when calling pipes programmatically

## Related Documentation

- [Build Output](output.md) - Generate example output JSON for a pipe
- [Build Runner](runner.md) - Generate Python code to run a pipe
- [Provide Inputs](../../../building-methods/pipes/provide-inputs.md) - Learn about input formats


## Output

# Build Output

Generate output representation for a pipe, showing the expected output structure based on the pipe's output concept type. Supports multiple output formats including JSON Schema for TypeScript/Zod integration.

## Usage

### By pipe code

```bash
pipelex build output pipe <PIPE_CODE> [OPTIONS]
```

**Arguments:**

- `PIPE_CODE` - The pipe code (e.g. `my_domain.my_pipe`)

**Options:**

- `--library-dir`, `-L` - Directory to search for pipe definitions. Can be specified multiple times.
- `--output`, `-o` - Path to save the generated file (defaults to `results/`)
- `--format`, `-f` - Output format (default: `json`):
    - `json` - JSON object with example placeholder data
    - `python` - Python class instantiation code
    - `schema` - JSON Schema definition, ideal for generating TypeScript interfaces or Zod schemas

### From a bundle

```bash
pipelex build output bundle <PATH> [OPTIONS]
```

**Arguments:**

- `PATH` - Path to a `.mthds` bundle file or a pipeline directory

**Options:**

- `--pipe` - Pipe code to use (can be omitted if the bundle declares a `main_pipe`)
- `--library-dir`, `-L` - Directory to search for pipe definitions. Can be specified multiple times.
- `--output`, `-o` - Path to save the generated file (defaults to bundle's directory)
- `--format`, `-f` - Output format (default: `json`): `json`, `python`, or `schema`

### From an installed method

```bash
pipelex build output method <NAME> [OPTIONS]
```

**Arguments:**

- `NAME` - Name of the installed method

**Options:**

- `--pipe` - Pipe code (overrides method's `main_pipe`)
- `--library-dir`, `-L` - Directory to search for pipe definitions. Can be specified multiple times.
- `--output`, `-o` - Path to save the generated file
- `--format`, `-f` - Output format (default: `json`): `json`, `python`, or `schema`

## Examples

**Generate output for a pipe by code:**

```bash
pipelex build output pipe my_domain.my_pipe
```

**Generate output from a bundle (uses main_pipe):**

```bash
pipelex build output bundle my_bundle.mthds
```

**Generate JSON Schema from a bundle for TypeScript/Zod integration:**

```bash
pipelex build output bundle my_bundle.mthds --format schema
```

**Generate output from a pipeline directory:**

```bash
pipelex build output bundle pipeline_01/
```

**Specify which pipe to use from a bundle:**

```bash
pipelex build output bundle my_bundle.mthds --pipe my_pipe
```

**Generate output for a pipe using a library directory:**

```bash
pipelex build output pipe my_domain.my_pipe -L ./my_library/
```

**Custom output path:**

```bash
pipelex build output bundle my_bundle.mthds --output expected_output.json
```

## Output Formats

### JSON Format (default)

The JSON format generates example placeholder data showing the output structure:

```json
{
  "concept": "my_domain.MyOutputConcept",
  "content": {
    "title": "title_value",
    "key_points": "key_points_value"
  }
}
```

For native concepts like `Text`, `Image`, or `Document`:

```json
{
  "concept": "native.Text",
  "content": {
    "text": "text_value"
  }
}
```

When a pipe's output has multiplicity (returns multiple items), the content is a list:

```json
{
  "concept": "my_domain.Item",
  "content": [
    {
      "name": "name_value",
      "description": "description_value"
    }
  ]
}
```

### Python Format

The Python format generates class instantiation code:

```python
{
  "concept": "my_domain.MyOutputConcept",
  "content": "MyOutputConcept(title=\"title_value\", key_points=\"key_points_value\")"
}
```

### Schema Format

The schema format generates JSON Schema definitions, ideal for generating TypeScript interfaces or Zod schemas:

```json
{
  "concept": "my_domain.MyOutputConcept",
  "content": {
    "type": "object",
    "properties": {
      "title": { "type": "string", "title": "Title" },
      "key_points": { "type": "string", "title": "Key Points" }
    },
    "required": ["title", "key_points"],
    "title": "MyOutputConcept"
  }
}
```

**Array outputs** (e.g., `MyType[5]`) are properly represented as JSON Schema arrays:

```json
{
  "concept": "my_domain.Item",
  "content": {
    "type": "array",
    "items": {
      "type": "object",
      "properties": {
        "name": { "type": "string", "title": "Name" },
        "description": { "type": "string", "title": "Description" }
      },
      "required": ["name", "description"],
      "title": "Item"
    }
  }
}
```

### `native.Anything` Output

When a pipe's output is `native.Anything` (e.g., a `PipeCondition` with mapped pipes that have different output types), the command shows all possible outputs from the mapped pipes.

For JSON format:

```json
{
  "output_option_1": {
    "concept": "my_domain.Result1",
    "content": { "field1": "field1_value" }
  },
  "output_option_2": {
    "concept": "my_domain.Result2",
    "content": { "field2": "field2_value" }
  }
}
```

For schema format:

```json
{
  "schema_option_1": {
    "concept": "my_domain.Result1",
    "content": {
      "type": "object",
      "properties": { "field1": { "type": "string" } },
      "required": ["field1"]
    }
  },
  "schema_option_2": {
    "concept": "my_domain.Result2",
    "content": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": { "field2": { "type": "string" } },
        "required": ["field2"]
      }
    }
  }
}
```

This helps you understand all possible output structures that the pipe could return depending on execution path.

## Use Cases

- **Understanding pipe outputs** - Quickly see what structure a pipe returns
- **API integration** - Know the exact structure to expect when calling pipes programmatically
- **TypeScript/Zod integration** - Use `--format schema` to generate JSON Schema, then convert to TypeScript interfaces or Zod schemas for type-safe frontend integration
- **Type definitions** - Use as reference for defining types in your application code
- **Testing** - Create expected output templates for validation

## Related Documentation

- [Build Inputs](inputs.md) - Generate example input JSON for a pipe
- [Build Runner](runner.md) - Generate Python code to run a pipe
- [Pipe Output](../../../building-methods/pipes/pipe-output.md) - Learn about pipe outputs


## Agent CLI

# Agent CLI (`pipelex-agent`)

The `pipelex-agent` CLI is a machine-oriented companion to the main `pipelex` CLI. It is designed for programmatic consumption by AI agents, IDE extensions, and other automation tools. Output format varies by command — markdown or JSON, raw TOML, or `plxt` passthrough — with no Rich formatting or interactive prompts. Structured commands emit errors to stderr; the error format is controlled by `--error-format`, which defaults to the value of `--format` (so `--format json` flips both, as before the split). `fmt` and `lint` pass through native `plxt` output.

It is consumed by the `mthds-agent` CLI (from the `mthds` npm package) which itself is used by Claude Code skills, the VS Code extension, and can be called directly from the command line.

## Global Options

| Option | Description |
|--------|-------------|
| `--runner` | Select the runner backend to use (`pipelex` or `api`) |
| `--version` | Print the version and exit |

There is no `--log-level` flag: `pipelex-agent` is machine-consumed, so logging is cut off process-wide by design.

## Commands Overview

The agent CLI mirrors the main CLI's subcommand structure for `run`, `validate`, and `inputs`, each with `pipe`, `bundle`, and `method` subcommands. It also provides `fix bundle` for deterministic in-place repairs.

### Run

Execute a pipeline. Success output is markdown by default, or JSON with `--format json`. Errors follow `--error-format` (defaults to `--format`'s value).

```bash
pipelex-agent run pipe <PIPE_CODE> [OPTIONS]
pipelex-agent run bundle <PATH> [OPTIONS]
pipelex-agent run method <NAME> [OPTIONS]
```

**Common options:**

- `--inputs`, `-i` - Path to a JSON or TOML inputs file (discriminated by file extension: `.toml` → TOML, everything else → JSON). Inline JSON (a value starting with `{`) stays JSON-only.
- `--dry-run` - Dry-run without calling AI providers
- `--mock-inputs` - Use mock inputs (requires `--dry-run`)
- `--graph` / `--no-graph` - Enable/disable execution graph (enabled by default)
- `--library-dir`, `-L` - Additional library directory
- `--with-memory` - Include full working memory in output
- `--format` - Success output format: `markdown` (default) or `json`
- `--error-format` - Error output format: `markdown` or `json` (defaults to `--format`'s value)

For `bundle` and `method`, use `--pipe` to target a specific pipe.

!!! note "Stdin inputs stay JSON"
    The agent CLI can also read inputs from stdin (a flat dict or a `working_memory` envelope). Stdin inputs are **JSON-only** — the extension-based TOML discrimination applies to `--inputs` file paths only. Like the main CLI, `run bundle <dir>` auto-detects `inputs.json` / `inputs.toml` when `--inputs` is omitted, erroring if both exist.

### Validate

Validate pipes, bundles, or methods. Success output is markdown by default, or JSON with `--format json`. Errors follow `--error-format` (defaults to `--format`'s value).

```bash
pipelex-agent validate pipe <PIPE_CODE> [OPTIONS]
pipelex-agent validate pipe --all [OPTIONS]
pipelex-agent validate bundle <PATH> [OPTIONS]
pipelex-agent validate method <NAME> [OPTIONS]
```

**Common options:**

- `--library-dir`, `-L` - Additional library directory
- `--allow-signatures` - Accept [`PipeSignature`](../../building-methods/pipes/signature-pipes.md) placeholders in the dependency graph (lenient mode)
- `--format` - Success output format: `markdown` (default) or `json`
- `--error-format` - Error output format: `markdown` or `json` (defaults to `--format`'s value)

For `bundle`, additional options are available:

- `--pipe` - Require a specific pipe in the bundle
- `--graph`, `-g` - Generate an execution graph visualization
- `--graph-format`, `-f` - Graph output format (`mermaidflow`, `reactflow`, or `both`)
- `--direction` - Graph layout direction

!!! note "Signature pipes"
    `pipelex-agent validate` is strict by default — same as `pipelex validate`. A bundle whose dependency graph reaches a `PipeSignature` is rejected unless you pass `--allow-signatures`, which dry-runs signatures as mocks.

    On a successful run, the envelope also carries `pending_signatures` — the library-wide list of pipes still declared as `PipeSignature` (unimplemented forward declarations), each namespaced by `pipe_ref` (`domain.code`). In JSON it is a `pending_signatures` array, in markdown a "Pending signatures" section. A top-down build reads it to see exactly which headers remain to implement. The envelope also carries a derived `is_runnable` boolean (`true` ⇔ `pending_signatures` is empty), and the markdown states the runnability verdict in plain English — runnable when complete, NOT yet runnable above the "Pending signatures" section otherwise. `validate bundle`, `validate method`, and `validate pipe --all` carry them and gate on them: without `--allow-signatures`, the command exits non-zero when `is_runnable` is false. Bare `validate pipe <code>` omits them, and a `--pipe` slice surfaces them for information without gating.

!!! note "Advisory warnings on validate"
    Whole-bundle and whole-library validate surfaces (`validate bundle`, `validate method`, `validate pipe --all`) also carry a `warnings` array — advisory optionality lints on a VALID bundle that never flip the verdict or the exit code. Each entry has the **same shape as a validation error item** (`category`, `error_type`, `pipe_code`, `domain_code`, `variable_names`, `message`) — this is a different shape from the `init`/`doctor` setup `warnings` (`{type, message}`) documented under Output Contract below. The first occupant is the useless-`!` lint (`optional_force_redundant`): a `!` (force) input whose slot is guaranteed present in every analyzed flow, so the assertion can never fire. In markdown, warnings render as a "Warnings" section. The array is empty when there is nothing to report; single-pipe `validate pipe` omits it (no flow context to lint in).

### Fix

Apply deterministic safe fixes to a bundle, re-validating after each round until the bundle is valid, no applicable fix remains, or the iteration limit is reached.

```bash
pipelex-agent fix bundle <PATH> [OPTIONS]
```

**Options:**

- `--library-dir`, `-L` - Additional library directory; files in explicitly supplied directories may be fixed when an error identifies them as the source
- `--allow-signatures` - Accept `PipeSignature` placeholders during validation
- `--max-iterations` - Limit the number of validate/apply rounds
- `--select` - Apply only the named fix rule; repeat for multiple rules
- `--ignore` - Skip the named fix rule; repeat for multiple rules (`--select` and `--ignore` are mutually exclusive)
- `--format` - Success output format: `markdown` (default) or `json`
- `--error-format` - Error output format: `markdown` or `json` (defaults to `--format`'s value)

The result reports `iterations`, `fixes_applied`, `files_written`, and any `remaining_errors`. A fully valid result exits 0; a completed fix attempt that remains invalid exits 1; argument, setup, or unexpected failures exit 2.

### Inputs

Generate an example inputs template for a pipe, bundle, or method. By default the template is the **light** signature-driven shape (bare values matching the pipe's declared concepts), which is exactly what `run` accepts; `--explicit` emits the ceremonial `{concept, content}` envelope form instead.

```bash
pipelex-agent inputs pipe <PIPE_CODE> [OPTIONS]
pipelex-agent inputs bundle <PATH> [OPTIONS]
pipelex-agent inputs method <NAME> [OPTIONS]
```

**Common options:**

- `--library-dir`, `-L` - Additional library directory
- `--format` - Template serialization: `json` (default) or `toml`
- `--explicit` - Emit the ceremonial `{concept, content}` envelope form instead of the light values

For `bundle` and `method`, use `--pipe` to target a specific pipe.

!!! note "`inputs --format` is `json|toml`, not `markdown|json`"
    Unlike `run`/`validate`, the `inputs` command's `--format` selects the **template serialization**, not a presentation style. `json` (the default) emits the structured JSON success envelope; `toml` prints the raw TOML template straight to stdout (a pipe with no inputs prints a TOML comment line, which loads back as an empty dict). This mirrors the raw-TOML output of the `concept` and `pipe` commands. `inputs` has no `--error-format` — its errors stay JSON.

!!! note "`--explicit` and concept hints"
    The light `--format toml` template carries the declared concept for each key as a `# concept: ...` comment; the light `--format json` template (the default) cannot (JSON has no comments), so pass `--explicit` when you want the concept written out inline. The JSON success envelope shape (`success` / `pipe_code` / `inputs`) is unchanged — only the `inputs` payload flips between the light values and the envelope form.

### Flat Commands

These commands do not have subcommands:

| Command | Description |
|---------|-------------|
| `fmt` | Format a `.mthds`/`.toml`/`.plx` file in-place (delegates to `plxt`) |
| `lint` | Lint a `.mthds`/`.toml`/`.plx` file for errors (delegates to `plxt`) |
| `concept` | Convert a JSON concept spec into raw TOML (stdout) |
| `pipe` | Convert a JSON pipe spec into raw TOML (stdout) |
| `models` | List available model presets, aliases, and waterfalls (`--format markdown\|json` success, default: markdown; `--error-format` for errors, defaults to `--format`'s value) |
| `doctor` | Check config, credentials, and model health (`--format markdown\|json` success, default: markdown; `--error-format` for errors, defaults to `--format`'s value) |

## Output Contract

Commands use different stdout formats depending on their purpose:

- **Markdown or JSON**: `run`, `validate`, `fix`, `init`, `models`, `check-model`, `doctor` — markdown by default, JSON with `--format json`. Error format follows `--error-format` (defaults to `--format`'s value, so `--format json` flips both).
- **JSON or raw TOML**: `inputs` — structured JSON via `agent_success()` by default (`--format json`), or the raw TOML template printed directly to stdout with `--format toml`
- **Raw TOML**: `concept`, `pipe` — TOML text printed directly to stdout
- **Passthrough**: `fmt`, `lint` — raw `plxt` output

**JSON success** — written to stdout:

```json
{
  "success": true,
  "target_dir": "/path/to/.pipelex",
  "backends_enabled": ["openai"]
}
```

JSON commands return the result object directly. They are not wrapped in a `status` or `data` envelope.

**Warnings** — non-fatal setup conditions are surfaced on a top-level `warnings` array of the success envelope, so consumers don't have to parse stderr. The field is absent when there is nothing to report. Each entry is an object with a `type` and a `message`:

```json
{
  "success": true,
  "target_dir": "/path/to/.pipelex",
  "warnings": [
    { "type": "RemoteConfigStale", "message": "Using a cached gateway config; run `pipelex init` while online to refresh." }
  ]
}
```

`RemoteConfigStale` is emitted when the gateway is enabled but the remote config service is unreachable and Pipelex falls back to its on-disk cache (offline mode).

Do not confuse this setup-warning shape with the `warnings` array on the `validate` envelope — validate warnings are advisory lint items that reuse the validation-error item shape (see "Advisory warnings on validate" above).

**Error** — written to stderr:

```json
{
  "error": true,
  "error_type": "specific_error_type",
  "message": "Human-readable description",
  "error_domain": "input",
  "hint": "Suggested fix or next step",
  "retryable": false
}
```

`fmt` and `lint` are raw passthroughs to the `plxt` binary and bypass the JSON output contract, producing native `plxt` output instead.

## Graph Visualization

Graph visualization is available through the `validate bundle` subcommand with the `--graph` flag, and through `run` subcommands where it is enabled by default. Use `--no-graph` on `run` to disable it.


## Overview

# Configuration

## Overview

Pipelex uses a TOML-based configuration system with **shipped defaults** plus **project-level overrides**.

- **Shipped defaults**: Pipelex ships default values that are maintained in the Pipelex repository (contributors will see them in the repo root `pipelex.toml`). This is the baseline used by the installed package.
- **Project overrides**: a project that *uses* Pipelex typically customizes behavior via files created in `.pipelex/`.

You can create the configuration files by running:

```bash
pipelex init config          # creates the global config at ~/.pipelex/
pipelex init config --local  # creates the project config at {project_root}/.pipelex/
```

!!! important "Configuration Setup Notes"
    1. By default `pipelex init config` targets the **global** `~/.pipelex/` directory; pass `--local` to create the project-level `.pipelex/` instead.
    2. `pipelex init config` creates a **template** configuration file with sample settings. It does not include all possible configuration options - it's meant as a starting point.
    3. Running `pipelex init config` will **overwrite** your existing `pipelex.toml` file without warning. Make sure to backup your configuration before running this command.
    4. Credentials (`~/.pipelex/.env`) and the Gateway service-agreement file always remain in the global `~/.pipelex/` directory regardless of `--local` — only config, inference, and telemetry files are written to the project `.pipelex/`.

For a complete list of all possible configuration options, refer to the configuration group documentation below.

## Where to edit configuration in a project

The main project configuration files are:

- `.pipelex/pipelex.toml`: project customization (logging, reporting, feature flags, etc.)
- `.pipelex/telemetry.toml`: custom telemetry destinations
- `.pipelex/inference/…`: inference backends, routing profiles, and model presets

## Overrides (advanced)

In addition to the base `pipelex.toml`, Pipelex applies override files from **inside** each `.pipelex/` directory (both the global `~/.pipelex/` and the project's `.pipelex/`) for machine- and environment-specific settings:

1. `pipelex_local.toml`
2. `pipelex_{environment}.toml` (example: `pipelex_dev.toml`) — selected by the `PIPELEX_ENV` environment variable (see [Selecting the environment](#selecting-the-environment))
3. `pipelex_{run_mode}.toml` (example: `pipelex_normal.toml`; unit tests use `tests/pipelex_unit_test.toml`, the one override file read outside a `.pipelex/` directory)
4. `pipelex_override.toml` (recommended to gitignore)
5. `pipelex_temporary_override.toml` (recommended to gitignore)

!!! info "Contributor details"
    For the full “where defaults live” and “how config is merged” explanation, see [Configuration Internals](../contribute/configuration-defaults-and-overrides.md).

## Configuration Structure

The configuration is organized into four main sections:

1. `[pipelex]` - Core Pipelex settings
2. `[cogt]` - Cognitive tools and LLM settings
3. `[plugins]` - Plugin-specific configurations
4. `[migration]` - Migration helpers for renamed settings

Each section contains multiple subsections for specific features and functionalities.

## Configuration Override System

Pipelex uses a sophisticated configuration override system that loads and merges configurations in a specific order. This allows for fine-grained control over settings in different environments and scenarios.

The exact loading sequence is (later wins, per leaf key):

1. Base configuration from the installed Pipelex package (`pipelex.toml`)
2. Global base configuration (`~/.pipelex/pipelex.toml`)
3. Global override sequence, from `~/.pipelex/`: `pipelex_local.toml`, `pipelex_{environment}.toml`, `pipelex_{run_mode}.toml`, `pipelex_override.toml`, `pipelex_temporary_override.toml`
4. Your project's base configuration (`{project_root}/.pipelex/pipelex.toml`)
5. Project override sequence, from the project's `.pipelex/`: the same five files as step 3
6. Programmatic overrides passed in code, if any

Notes on the override sequence:

- `pipelex_{environment}.toml` (example: `pipelex_dev.toml`) is selected by the `PIPELEX_ENV` environment variable (see [Selecting the environment](#selecting-the-environment))
- `pipelex_{run_mode}.toml` — example run modes: normal, unit_test; under unit testing the run-mode overlay is sourced exclusively from `./tests/pipelex_{run_mode}.toml`
- Each subsequent configuration file in this sequence can override settings from the previous ones, and project-level files override the global `~/.pipelex/` layer

### Override File Naming

- Base config: `pipelex.toml`
- Local overrides: `pipelex_local.toml`
- Environment overrides: `pipelex_dev.toml`, `pipelex_staging.toml`, `pipelex_prod.toml`, etc.
- Run mode overrides: `pipelex_normal.toml`, `tests/pipelex_unit_test.toml`, etc.
- Final overrides: `pipelex_override.toml`

NB: The run_mode unit_test is used for testing purposes.

### Selecting the environment

The `pipelex_{environment}.toml` overlay is picked at runtime from the `PIPELEX_ENV` environment variable.

| Value     | Overlay file loaded   |
| --------- | --------------------- |
| `local`   | `pipelex_local.toml` *(also loaded as the local override layer; see above)* |
| `dev`     | `pipelex_dev.toml`     |
| `staging` | `pipelex_staging.toml` |
| `prod`    | `pipelex_prod.toml`    |

If `PIPELEX_ENV` is unset, Pipelex defaults to `dev`. Any other value raises an error at startup — the accepted values are defined by the `RunEnvironment` enum in `pipelex/system/runtime.py`.

The selected environment is also stamped on OpenTelemetry spans as `deployment.environment`, so it doubles as the environment label for traces and metrics.

Set it the way you set any other env var — for example, in your shell, your `.env` file, your CI configuration, or your container runtime:

```bash
export PIPELEX_ENV=staging
```

### Best Practices for Overrides

1. Use the base `pipelex.toml` for default settings
2. Use `pipelex_local.toml` for machine-specific settings
3. Use environment files for environment-specific settings (dev, staging, prod)
4. Use run mode files for normal or unit_test configurations
5. Use `pipelex_override.toml` sparingly, only for temporary overrides (add to .gitignore)

## Best Practices

1. **Version Control**: Include your base `pipelex.toml` in version control
2. **Environment Overrides**: Use environment-specific files for sensitive or environment-dependent settings
3. **Documentation**: Comment any custom settings for team reference
4. **Validation**: Run `pipelex validate --all` after making configuration changes
5. **Gitignore**: Add local and sensitive override files to `.gitignore`


## Dry Run

# Dry Run Configuration

The `DryRunConfig` class controls how Pipelex behaves during dry runs.

## Configuration Options

```python
class DryRunConfig(ConfigModel):
    text_gen_truncate_length: int
    nb_list_items: int
    nb_extract_pages: int
    image_urls: List[str]
    allowed_to_fail_pipes: List[str] = Field(default_factory=list)
```

### Fields

- `text_gen_truncate_length`: Maximum length of generated text during dry runs
- `nb_list_items`: Number of items to generate for list content during dry runs
- `nb_extract_pages`: Number of pages to simulate for OCR operations during dry runs
- `image_urls`: List of image URLs to use for dry run testing (must be non-empty)
- `allowed_to_fail_pipes`: List of pipe names that are allowed to fail during dry runs (optional)

## Example Configuration

```toml
[pipelex.dry_run_config]
text_gen_truncate_length = 100
nb_list_items = 3
nb_extract_pages = 2
image_urls = ["https://pipelex-pytest-assets.s3.eu-west-3.amazonaws.com/ai_lympics.jpg", "https://pipelex-pytest-assets.s3.eu-west-3.amazonaws.com/animal_lympics.jpg"]
allowed_to_fail_pipes = ["optional_pipe", "experimental_pipe"]
```

## Dry Run Behavior

During a dry run, no AI provider is ever called and no storage IO is performed: each inference leaf (LLM text/object generation, image generation, document extraction, web search, templating) produces a synthetic mock instead. Jinja2 templates are still parse-checked, so a broken template fails loudly even in a dry run.

### Text Generation

The `text_gen_truncate_length` controls:

- Maximum length of simulated text output
- Helps prevent excessive resource usage during testing
- Makes dry run output more manageable

## Use Cases

1. **Testing Pipeline Logic**

     - Validate pipeline structure
     - Check template syntax
     - Verify variable references

2. **Resource Estimation**

     - Estimate processing time
     - Calculate potential costs
     - Plan resource allocation

3. **Debugging**

     - Trace execution paths
     - Identify potential issues
     - Test error handling

## Best Practices

- Use dry runs for testing before production
- Set appropriate truncation lengths
- Review dry run logs for potential issues

## Related Documentation

- [Validation & Dry Run](../../features/validation-dry-run.md) - Overview of validation and dry run capabilities
- [CLI validate](../../tools/cli/validate.md) - Using the CLI to validate methods


## Logging

# Logging Configuration

Configuration section: `[pipelex.log_config]`

## Overview

The logging configuration controls how Pipelex handles log messages, their formatting, and their display. The configuration is split into general logging settings and Rich console output settings.

## General Settings

### Log Levels

```toml
default_log_level = "INFO"
```

- Sets the default logging level for all loggers
- Valid values: `"VERBOSE"`, `"DEBUG"`, `"DEV"`, `"INFO"`, `"WARNING"`, `"ERROR"`, `"CRITICAL"`, `"OFF"`

### Package-Specific Log Levels

```toml
[pipelex.log_config.package_log_levels]
anthropic = "INFO"
asyncio = "INFO"
botocore = "INFO"
openai = "INFO"
pipelex = "INFO"
```

- Override log levels for specific packages
- Use `-` instead of `.` in package names (e.g., `urllib3-connectionpool`)

### Console Output

```toml
is_console_logging_enabled = true
```

- Enable/disable console logging
- Default: `true`

### JSON Formatting

```toml
json_logs_indent = 4
presentation_line_width = 120
```

- `json_logs_indent`: Indentation for JSON log output
- `presentation_line_width`: Maximum line width for formatted output

### Caller Information

```toml
is_caller_info_enabled = false
caller_info_template = "file_line"
```

Available templates:

- `"file_line"`: "file.py:123"
- `"file_line_func"`: "file.py:123 function_name"
- `"func"`: "function_name"
- `"file_func"`: "file.py function_name"
- `"func_line"`: "function_name 123"
- `"func_module"`: "function_name module"
- `"func_module_line"`: "function_name module 123"

### Problem Silencing

```toml
silenced_problem_ids = ["azure_openai_no_stream_options"]
```

- List of problem IDs to silence
- Prevents specific warnings from being logged

### Poor Loggers

```toml
generic_poor_logger = "#poor-log"
poor_loggers = [
    "kajson.decoder.sandbox",
    "kajson.encoder.sandbox",
    "class_registry.sandbox"
]
```

- Loggers that use simplified formatting without fancy features
- Useful for sandbox environments or when simple logging is needed

## Rich Console Configuration

Configuration section: `[pipelex.log_config.rich_log_config]`

### Display Options

```toml
is_show_time = false
is_show_level = true
is_link_path_enabled = true
```

- `is_show_time`: Show timestamp in logs
- `is_show_level`: Show log level
- `is_link_path_enabled`: Make file paths clickable

### Syntax Highlighting

```toml
highlighter_name = "json"  # or "repr"
is_markup_enabled = true
```

- `highlighter_name`: Choose between JSON or repr highlighting
- `is_markup_enabled`: Enable Rich markup syntax in log messages

### Traceback Settings

```toml
is_rich_tracebacks = true
is_tracebacks_word_wrap = true
is_tracebacks_show_locals = false
tracebacks_suppress = []
```

- Control how Python tracebacks are displayed
- Enable/disable word wrapping and local variable display
- Suppress specific traceback patterns

### Keyword Highlighting

```toml
keywords_to_hilight = []
```

- List of keywords to highlight in log messages
- Useful for emphasizing important terms

## Example Configuration

```toml
[pipelex.log_config]
default_log_level = "INFO"
log_mode = "rich"
is_console_logging_enabled = true
json_logs_indent = 4
presentation_line_width = 120
is_caller_info_enabled = true
caller_info_template = "file_line_func"

silenced_problem_ids = []
generic_poor_logger = "#poor-log"
poor_loggers = []

[pipelex.log_config.package_log_levels]
pipelex = "INFO"
openai = "WARNING"
anthropic = "INFO"

[pipelex.log_config.rich_log_config]
is_show_time = false
is_show_level = true
is_link_path_enabled = true
highlighter_name = "json"
is_markup_enabled = true
is_rich_tracebacks = true
is_tracebacks_word_wrap = true
is_tracebacks_show_locals = false
tracebacks_suppress = []
keywords_to_hilight = ["error", "warning", "failed"]
```

## Best Practices

1. **Development Environment**:

    - Enable caller info for better debugging
    - Use verbose logging levels
    - Enable local variables in tracebacks

2. **Production Environment**:

    - Disable caller info for performance
    - Use INFO or higher log levels
    - Disable local variables in tracebacks

3. **Package Log Levels**:

    - Set noisy third-party packages to WARNING
    - Keep pipelex at INFO for important updates
    - Use VERBOSE only when debugging specific issues

## Related Documentation

- [Logging Tool](../../tools/logging.md) - Using Pipelex logging in your code


## Pipe Run

# Pipe Run Configuration

The `PipeRunConfig` class controls execution parameters for pipes in Pipelex.

## Configuration Options

```python
class PipeRunConfig(ConfigModel):
    pipe_stack_limit: int
```

### Fields

- `pipe_stack_limit`: Maximum depth of nested pipe executions allowed

## Example Configuration

```toml
[pipelex.pipe_run_config]
pipe_stack_limit = 20
```

## Stack Limit

The `pipe_stack_limit` prevents infinite recursion in pipe execution by:

- Limiting the depth of nested pipe calls
- Throwing an exception when the limit is exceeded
- Protecting against accidental circular dependencies

## Best Practices

- Set a reasonable stack limit based on your pipeline complexity
- Monitor stack usage in complex pipelines

## Related Documentation

- [Executing Pipelines](../../building-methods/pipes/executing-pipelines.md) - How to run pipes and pipelines
- [Pipeline Orchestration](../../features/pipeline-orchestration.md) - Orchestrating complex multi-step pipelines


## Reporting

# Reporting Configuration

Configuration section: `[pipelex.reporting_config]`

## Overview

The reporting configuration controls how Pipelex generates and manages cost reports for various operations (LLM usage, OCR processing, etc.). This configuration allows you to customize where and how cost reports are generated.

## Configuration Options

### Console Logging

```toml
is_log_costs_to_console = true
```

- Controls whether the end-of-run cost table is printed to the console
- When enabled (and `--costs` is on), prints a single cost-by-model table after the pipeline finishes — but only when the run had reportable usage; it is suppressed for dry runs (zero tokens and zero cost) and on a usage-assembly/aggregation failure (which never fails the run)
- Default: `true` — library embedders who don't want console output can set it to `false`

### Cost Report File Generation

```toml
is_generate_cost_report_file_enabled = false
```

- Controls whether cost report files are generated after the execution of the pipeline
- When enabled, creates detailed reports in CSV format in the directory `reports/` by default.
- Default: `false`

### Report File Location

```toml
cost_report_dir_path = "reports"
```

- Directory where cost report files will be saved
- Path is relative to your project root
- Directory will be created if it doesn't exist.
- Default: `"reports"`

### Report File Naming

```toml
cost_report_base_name = "cost_report"
cost_report_extension = "csv"
```

- Base name for report files
- File extension for report files (e.g., "csv" for CSV files)
- Final filename will include an incremental number (e.g., `cost_report_1.csv`, `cost_report_2.csv`)
- Default: Base name = `"cost_report"`, Extension = `"csv"`

### Cost Unit Scaling

```toml
cost_report_unit_scale = 1.0
```

- Scaling factor applied to cost values in reports
- Use 1.0 for actual costs
- Use different values to scale costs (e.g., 1000.0 for thousands)
- The currency is in USD.
- Default: `1.0`

## Example Configuration

```toml
[pipelex.reporting_config]
# Enable console logging for development
is_log_costs_to_console = true

# Generate report files
is_generate_cost_report_file_enabled = false

# Store reports in a custom directory
cost_report_dir_path = "analytics/cost_reports"

# Customize report file naming
cost_report_base_name = "llm_costs"
cost_report_extension = "csv"

# Show costs in thousands
cost_report_unit_scale = 1000.0
```

## Best Practices

!!! warning "Under Construction"
    This section is currently under development.

## Related Documentation

- [Cost Tracking](../../features/cost-tracking.md) - Track and monitor AI usage costs
- [Reporting Delegate Injection](../../advanced/reporting-delegate-injection.md) - Customize reporting behavior with dependency injection


## Telemetry

# Telemetry Configuration

## Overview

Telemetry configuration is stored in `.pipelex/telemetry.toml`. This file controls **your custom telemetry** destinations—where you want to send observability data for your own analytics and monitoring.

!!! info "Two Telemetry Streams"
    Pipelex has two independent telemetry streams:

    1. **Gateway Telemetry** (Pipelex-controlled): Automatic when using Pipelex Gateway, tied to your API key
    2. **Custom Telemetry** (User-controlled): Configured in this file, sent to your own backends

    For an overview, see [Telemetry Setup](../../setup/telemetry.md).

## Configuration File Location

```
.pipelex/telemetry.toml
```

This file is created when you run `pipelex init telemetry` or `pipelex init`.

### Global vs project config

Telemetry config is layered: a project `.pipelex/telemetry.toml` is merged **on top of** the user's global `~/.pipelex/telemetry.toml`, not as a replacement. The load order is:

1. `~/.pipelex/telemetry.toml`
2. `~/.pipelex/telemetry_override.toml`
3. `{project_root}/.pipelex/telemetry.toml` *(if it exists and the project dir is distinct from the global dir)*
4. `{project_root}/.pipelex/telemetry_override.toml` *(same condition)*

Files are deep-merged, with later layers winning per leaf key. This means Langfuse keys or OTLP endpoints declared once in `~/.pipelex/telemetry.toml` keep applying across every project — a project file can override specific keys without redeclaring the rest.

!!! info "Project telemetry.toml is empty by default"
    To make the layering above actually work, `pipelex init` (when targeting a project's `.pipelex/`) drops in a **fully commented-out template**. Every section is present as documentation, but no key is set. This is deliberate: a value defined in the project file would otherwise override your global setting. Uncomment a key in the project template only when you want this project to diverge from your global telemetry config. Global init (`~/.pipelex/`) gets the active template with real defaults.

!!! note "List merge behavior"
    The `[[otlp]]` array and the `redact_properties` list do not concatenate across layers — the later layer replaces the whole list. If you want to add an OTLP exporter on top of the global set, redeclare all exporters in the layer that wins.

## Full Example Configuration

```toml
# PostHog Configuration (Event tracking + AI span tracing)
[custom_posthog]
mode = "anonymous"                                   # "off" | "anonymous" | "identified"
# user_id = "your_user_id"                           # Required when mode = "identified"
endpoint = "${POSTHOG_ENDPOINT}"                     # Default: https://us.i.posthog.com
api_key = "${POSTHOG_API_KEY}"                       # Get from PostHog Project Settings
geoip = true                                         # Enable GeoIP lookup
debug = false                                        # Enable PostHog debug mode
redact_properties = [
    "prompt",
    "system_prompt", 
    "response",
    "file_path",
    "url",
]                                                    # Event properties to redact

# AI span tracing to YOUR PostHog
[custom_posthog.tracing]
enabled = true                                       # Send AI spans to your PostHog

# Privacy controls for data sent to YOUR PostHog only
[custom_posthog.tracing.capture]
content = false                                      # Capture prompt/completion content
content_max_length = 1000                            # Max length for captured content
pipe_codes = true                                    # Include pipe codes in span names
output_class_names = true                            # Include output class names in spans

# Portkey SDK Configuration
[custom_portkey]
force_debug_enabled = false                          # Force-enable Portkey SDK debug mode
force_tracing_enabled = false                        # Force-enable Portkey SDK tracing

# Langfuse Integration (receives FULL span data, no redaction)
[langfuse]
enabled = true
endpoint = "https://cloud.langfuse.com"              # Override for self-hosted Langfuse
public_key = "${LANGFUSE_PUBLIC_KEY}"
secret_key = "${LANGFUSE_SECRET_KEY}"

# Additional OTLP Exporters (array, receives FULL span data)
[[otlp]]
name = "my-collector"                                # Identifier for logging
endpoint = "https://otel.example.com/v1/traces"      # OTLP endpoint URL
headers = { Authorization = "Bearer ${OTLP_AUTH_TOKEN}" }
```

---

## PostHog Configuration

The `[custom_posthog]` section configures event tracking and optional AI span tracing to your own PostHog instance.

### `[custom_posthog]` Settings

| Setting | Type | Default | Description |
|---------|------|---------|-------------|
| `mode` | string | `"off"` | Telemetry mode: `"off"`, `"anonymous"`, or `"identified"` |
| `user_id` | string | (none) | Required when `mode = "identified"` |
| `endpoint` | string | `"https://us.i.posthog.com"` | PostHog endpoint URL |
| `api_key` | string | (required) | Your PostHog project API key |
| `geoip` | boolean | `true` | Enable GeoIP lookup for location data |
| `debug` | boolean | `false` | Log events locally without sending |
| `redact_properties` | array | `[]` | Event properties to redact before sending |

#### Mode Options

- **`"off"`**: No events sent to your PostHog
- **`"anonymous"`**: Events sent without user identification
- **`"identified"`**: Events sent with your `user_id` for cross-session tracking

### `[custom_posthog.tracing]` Settings

Controls AI span tracing to your PostHog instance.

| Setting | Type | Default | Description |
|---------|------|---------|-------------|
| `enabled` | boolean | `false` | Send AI spans to your PostHog |

### `[custom_posthog.tracing.capture]` Settings

Privacy controls for what data is included in spans sent to **your** PostHog.

| Setting | Type | Default | Description |
|---------|------|---------|-------------|
| `content` | boolean | `false` | Capture prompt/completion content |
| `content_max_length` | integer | (unlimited) | Max length for captured content |
| `pipe_codes` | boolean | `false` | Include pipe codes in span names/attributes |
| `output_class_names` | boolean | `false` | Include output class names in spans |

!!! warning "Privacy Note"
    These capture settings only affect data sent to **your** PostHog. Langfuse and OTLP exporters always receive full span data without redaction.

---

## Portkey SDK Configuration

The `[custom_portkey]` section configures the Portkey SDK behavior when using a custom Portkey backend (not the Pipelex Gateway).

### `[custom_portkey]` Settings

| Setting | Type | Default | Description |
|---------|------|---------|-------------|
| `force_debug_enabled` | boolean | `false` | Force-enable Portkey SDK debug mode regardless of backend setting |
| `force_tracing_enabled` | boolean | `false` | Force-enable Portkey SDK tracing regardless of backend setting |

!!! info "When to Use"
    These settings are useful when you want to enable Portkey debugging or tracing globally without modifying individual backend configurations.

---

## Langfuse Configuration

The `[langfuse]` section enables integration with [Langfuse](https://langfuse.com/) for LLM observability.

| Setting | Type | Default | Description |
|---------|------|---------|-------------|
| `enabled` | boolean | `false` | Enable Langfuse integration |
| `endpoint` | string | `"https://cloud.langfuse.com"` | Langfuse endpoint (override for self-hosted) |
| `public_key` | string | (required if enabled) | Langfuse public key |
| `secret_key` | string | (required if enabled) | Langfuse secret key |

!!! info "Full Data"
    Langfuse receives **full span data** including prompts, completions, and all metadata. There is no redaction applied.

---

## OTLP Exporters Configuration

The `[[otlp]]` section is an **array** that allows you to configure multiple OpenTelemetry Protocol exporters.

| Setting | Type | Default | Description |
|---------|------|---------|-------------|
| `name` | string | (required) | Identifier for logging |
| `endpoint` | string | (required) | OTLP endpoint URL |
| `headers` | object | `{}` | HTTP headers for authentication |

### Example: Multiple OTLP Exporters

```toml
[[otlp]]
name = "datadog"
endpoint = "https://trace.agent.datadoghq.com/v1/traces"
headers = { "DD-API-KEY" = "${DD_API_KEY}" }

[[otlp]]
name = "honeycomb"
endpoint = "https://api.honeycomb.io/v1/traces"
headers = { "x-honeycomb-team" = "${HONEYCOMB_API_KEY}" }
```

!!! info "Full Data"
    OTLP exporters receive **full span data** including prompts, completions, and all metadata. There is no redaction applied.

---

## Environment Variable Substitution

All string values in `telemetry.toml` support environment variable substitution using the `${VAR_NAME}` syntax:

```toml
[custom_posthog]
endpoint = "${POSTHOG_ENDPOINT}"
api_key = "${POSTHOG_API_KEY}"

[langfuse]
public_key = "${LANGFUSE_PUBLIC_KEY}"
secret_key = "${LANGFUSE_SECRET_KEY}"

[[otlp]]
headers = { Authorization = "Bearer ${OTLP_AUTH_TOKEN}" }
```

This allows you to keep sensitive credentials in environment variables or `.env` files rather than committing them to version control.

### Variable Syntax Options

| Syntax | Description |
|--------|-------------|
| `${VAR_NAME}` | Use secrets provider (default: environment variable) |
| `${env:VAR_NAME}` | Force environment variable |
| `${secret:VAR_NAME}` | Force secrets provider |
| `${env:VAR\|secret:FALLBACK}` | Try env first, then secret as fallback |

---

## DO_NOT_TRACK Override

The `DO_NOT_TRACK` environment variable disables **all** telemetry globally:

```bash
# Linux/macOS
export DO_NOT_TRACK=1

# Windows PowerShell
$env:DO_NOT_TRACK = "1"

# Windows CMD
set DO_NOT_TRACK=1
```

When set, this disables:

- All custom telemetry (PostHog, Langfuse, OTLP)
- Gateway telemetry (note: Gateway won't work without telemetry)

---

## Resetting Configuration

To reset your telemetry configuration to defaults:

```bash
rm .pipelex/telemetry.toml
pipelex init telemetry
```

---

## Privacy and Compliance

- **Custom telemetry** gives you full control over what data is captured and where it's sent
- **PostHog tracing** includes granular privacy controls via the `capture` settings
- **Langfuse and OTLP** receive full data—configure these only if you're comfortable with that
- The `DO_NOT_TRACK` environment variable provides a global kill switch

For more information about our data practices, see our [Privacy Policy](https://go.pipelex.com/privacy-policy).

## Related Documentation

- [Telemetry Feature](../../features/telemetry.md) - Overview of telemetry capabilities
- [Telemetry Setup](../../setup/telemetry.md) - Step-by-step telemetry configuration guide


## AWS

# AWS Configuration

Configuration section: `[pipelex.aws_config]`

## Overview

The AWS configuration controls how Pipelex authenticates with AWS services. It supports two authentication methods: environment variables or a secret provider.

## Authentication Methods

```toml
[pipelex.aws_config]
api_key_method = "env"  # or "secret_provider"
bedrock_access_variant = "bedrock_token" # or "aws_access"
```

### Environment Variables Method (`"env"`)
When using `api_key_method = "env"`, Pipelex expects the following environment variables:

- `AWS_ACCESS_KEY_ID`: Your AWS access key ID
- `AWS_SECRET_ACCESS_KEY`: Your AWS secret access key
- `AWS_REGION`: Your AWS region

Example `.env` file:
```env
AWS_ACCESS_KEY_ID=your_access_key_id
AWS_SECRET_ACCESS_KEY=your_secret_access_key
AWS_REGION=us-east-1
```

### Secret Provider Method (`"secret_provider"`)
When using `api_key_method = "secret_provider"`, Pipelex will:

1. Connect to your configured secret provider

2. Look for the same keys as environment variables:
<ul>
   <li><code>AWS_ACCESS_KEY_ID</code></li>
   <li><code>AWS_SECRET_ACCESS_KEY</code></li>
   <li><code>AWS_REGION</code></li>
</ul>

!!! warning "Secret Provider Requirements"
    To use the secret provider method, you must:
    
1. Configure a secret provider in your project using the `SecretsProviderAbstract`: See more in the [Secrets](../../advanced/secrets-provider-injection.md) documentation.
2. Store your AWS credentials in your secret provider
3. Ensure your secret provider is properly authenticated

## Dependency Injection

Pipelex uses dependency injection to manage AWS clients and credentials. You can:

- Inject custom AWS client implementations
- Override default credential providers
- Mock AWS services for testing

For detailed information about dependency injection, including examples and best practices, see the [Dependency Injection](../../advanced/index.md) documentation.

## Best Practices

!!! warning "Under Construction"
    This section is currently under development.

## Related Documentation

- [Cloud Storage](../../features/cloud-storage.md) - Using cloud storage with Pipelex
- [Storage Provider Injection](../../advanced/storage-provider-injection.md) - Customize storage providers with dependency injection


## Cogt

# Cognitive Tools (Cogt) Configuration

The Cogt configuration manages all cognitive tools in Pipelex, including LLM (Language Models), Image Generation, and OCR (Optical Character Recognition) capabilities.

## Overview

```toml
[cogt]
# Tier 1 transport retry attempts
transport_max_retries = 2

# Main Cogt configuration sections
[cogt.llm_config]
[cogt.img_gen_config]
[cogt.extract_config]
```

## Transport Retry

`transport_max_retries` is a top-level `[cogt]` setting that controls how many times an inference SDK client retries a transient transport failure before giving up.

```toml
[cogt]
transport_max_retries = 2
```

- `transport_max_retries` (int, `0`–`10`, default `2`): the number of retries attempted **on top of** the initial request when a transport-level failure occurs — a connection error, or an HTTP `408` / `409` / `429` / `5xx` response. A value of `2` therefore allows up to 3 attempts total. Retries honor a `Retry-After` response header when present.

This is "Tier 1" of the retry model. It is wired uniformly into every inference SDK client factory — Anthropic, OpenAI / Azure OpenAI, the Pipelex Gateway clients, Mistral, and Google — as well as the raw-`httpx` Azure image-generation path, so the retry posture is a deliberate, uniform policy rather than a per-provider SDK default.

It is distinct from `llm_config.schema_reask_max_attempts`, which is `instructor`'s schema re-ask count for structured-output validation failures — a different concern.

## LLM Configuration

Configuration for all Language Model interactions:

```toml
[cogt.llm_config]
default_max_images = 100  # Maximum number of images in prompts
is_structure_prompt_enabled = true
schema_reask_max_attempts = 3  # instructor schema re-ask attempts, between 1 and 10
```

### LLM Job Parameters

When configuring LLM jobs, you can set:

- `temperature` (float, 0-1): Controls randomness in outputs
- `max_tokens` (optional int): Maximum tokens in response
- `seed` (optional int): For reproducible outputs

## Image Generation Configuration

Configuration for image generation capabilities:

```toml
[cogt.img_gen_config.img_gen_job_config]
is_sync_mode = false

# Default parameters for image generation
[cogt.img_gen_config.img_gen_param_defaults]
aspect_ratio = "square"  # Options: square, landscape_4_3, landscape_3_2, landscape_16_9, landscape_21_9, landscape_4_1, landscape_8_1,
                         # portrait_3_4, portrait_2_3, portrait_9_16, portrait_9_21, portrait_1_4, portrait_1_8
background = "auto"     # Options: transparent, opaque, auto
# size is optional and has no default key: when unset, no size intent is sent and the provider's
# default size applies (Gemini jobs omit image_size; OpenAI models use their fixed 1K-class preset
# for the aspect ratio). Set it to a size tier ("0.5k", "1k", "2k", "4k") or an exact size like "2048x1152"
# to apply a default size to every PipeImgGen that doesn't set its own. A tier default composes
# with any aspect_ratio; an exact-size default implies its own ratio, so it is not applied to
# pipes that explicitly set aspect_ratio (the pipe's ratio wins).
quality = "low"        # Options: low, medium, high
nb_steps = 28           # Number of diffusion steps (28 is good for Flux, [1,2,4,8] for SDXL Lightning)
guidance_scale = 2.5    # Controls adherence to prompt
is_moderated = true    # Enable content moderation
safety_tolerance = 5    # Safety level (1-6)
is_raw = false         # Raw output mode
seed = "auto"          # "auto" or specific integer
```

Note: `output_format` is a per-job parameter (set on the pipe), not a valid key under `img_gen_param_defaults`.

### ImageGen Job Parameters

Image generation jobs support these parameters:

- **Dimensions**:

    - `aspect_ratio`: Predefined ratios for image dimensions
    - `size` (optional): Portable size intent — a size tier (`"0.5k"`, `"1k"`, `"2k"`, `"4k"`) or an exact pixel size like `"2048x1152"`. Unset means the provider's default size. See [PipeImgGen — Image size and portability](../../building-methods/pipes/pipe-operators/PipeImgGen.md#image-size-and-portability)
    - `background`: Background handling mode

- **Quality Control**:

    - `quality`: Output quality level
    - `nb_steps`: Number of generation steps
    - `guidance_scale`: How closely to follow the prompt

- **Safety**:

    - `is_moderated`: Enable content moderation
    - `safety_tolerance`: Safety check strictness (1-6)

- **Output**:

    - `is_raw`: Raw output mode
    - `output_format`: Image format (PNG/JPG/WEBP)
    - `seed`: For reproducible generation

## OCR Configuration

Configuration for Optical Character Recognition:

```toml
[cogt.extract_config]
default_page_views_dpi = 72
```

## Unified Backend Integration

All cognitive tools (LLMs, OCR, and Image Generation) now use the same unified inference backend system:

### Benefits of Unified System

- **Consistent Configuration**: Same configuration patterns across all AI capabilities
- **Unified Routing**: All models routed through the same routing profiles
- **Shared Presets**: Presets for LLMs, OCR, and image generation in the same deck
- **Single API Management**: Manage all AI provider credentials in one place

### Backend Integration Details

- **LLM Models**: `model_type = "llm"` for text generation and structured outputs
- **OCR Models**: `model_type = "text_extractor"` for document processing
- **Image Generation Models**: `model_type = "img_gen"` for image creation

All model types support the same routing, aliasing, and preset systems.

## Validation Rules

### LLM Configuration
- Temperature must be between 0 and 1
- Max tokens must be positive
- Schema re-ask attempts must be between 1 and 10
- Seeds must be non-negative

### ImageGen Configuration
- Guidance scale must be positive
- Safety tolerance must be between 1 and 6
- Number of steps must be positive
- Strict validation for enums (aspect ratio, background, quality, output format)

## Best Practices

1. **LLM Settings**:

    - Start with lower temperatures (0.1-0.3) for consistent outputs
    - Use streaming for better user experience
    - Set appropriate retry limits based on your use case

2. **ImageGen Settings**:

    - Enable moderation for production use
    - Use appropriate aspect ratios for your use case
    - Balance quality and performance with step count

3. **General**:

     - Use platform preferences to ensure consistent model selection
     - Configure OCR handles based on your accuracy needs

## Example Complete Configuration

```toml
[cogt]
transport_max_retries = 2

[cogt.llm_config]
default_max_images = 100
is_structure_prompt_enabled = true
schema_reask_max_attempts = 3

[cogt.img_gen_config.img_gen_job_config]
is_sync_mode = false

[cogt.img_gen_config.img_gen_param_defaults]
aspect_ratio = "square"
background = "auto"
quality = "low"
nb_steps = 28
guidance_scale = 2.5
is_moderated = true
safety_tolerance = 5
is_raw = false
seed = "auto"

[cogt.extract_config]
default_page_views_dpi = 72
```

## Related Documentation

- [PipeExtract Operator](../../building-methods/pipes/pipe-operators/PipeExtract.md) - Extract text and images from documents
- [PipeImgGen Operator](../../building-methods/pipes/pipe-operators/PipeImgGen.md) - Generate images with AI models
- [Document Extraction](../../features/document-extraction.md) - Overview of document extraction capabilities
- [Image Generation](../../features/image-generation.md) - Overview of image generation capabilities


## LLM Providers & Models

# Inference Backend Configuration

The Inference Backend Configuration System manages how Pipelex handles AI model providers, model routing, and inference settings across LLMs, OCR, and image generation. This unified system provides a flexible and scalable way to configure multiple inference backends and route different types of AI models to the appropriate providers.

## Configuration Approaches

Pipelex supports three flexible approaches for accessing AI models:

### Option A: Pipelex Gateway (Optional & Free)

Get a single API key that works with all major providers (OpenAI, Anthropic, Google, Mistral, FAL, and more). This is the **recommended approach for getting started quickly**.

- ✅ Single API key for all providers
- ✅ Simplified configuration
- ✅ Automatic model routing
- ✅ Free on Discord (limited time offer)

!!! note "Terms of Service"
    Using Pipelex Gateway requires accepting our terms of service. When you run `pipelex init`, you'll be prompted to review and accept the terms. Gateway usage enables identified telemetry (tied to your hashed API key) for service monitoring. See our [Privacy Policy](https://go.pipelex.com/privacy-policy) for details.

### Option B: Bring Your Own Keys

Use your own API keys from individual providers for full control and direct billing. Ideal for production deployments with existing provider relationships.

- ✅ Direct provider relationships
- ✅ Full control over billing
- ✅ No intermediary
- ✅ Support for all provider-specific features

See [Inference Backends](#inference-backends) section below for configuration.

### Option C: Mix & Match (Custom Routing)

Configure custom routing profiles to use your own keys for some models and Pipelex Gateway for others. This gives you full flexibility to optimize for cost, performance, or rate limits.

- ✅ Hybrid approach
- ✅ Cost optimization
- ✅ Performance tuning
- ✅ Gradual migration between approaches

See [Routing Profiles](#routing-profiles) section below for setup.

## Overview

The inference backend system is built around four key concepts:

1. **Inference Backends**: Providers of AI services (OpenAI, Anthropic, Google Vertex AI, FAL, etc.) for LLMs, OCR, and image generation
2. **Model Specs**: Detailed information about specific models available through backends (text generation, text extraction, image generation)
3. **Routing Profiles**: Rules for selecting which backend should handle specific models across all AI capabilities
4. **Model Deck**: Unified collection of configured models, aliases, and presets for LLMs, OCR, and image generation

## Directory Structure

All inference backend configurations are stored in the `.pipelex/inference/` directory:

```
.pipelex/
└── inference/
    ├── backends.toml           # Backend provider configurations
    ├── routing_profiles.toml   # Model routing rules
    ├── backends/               # Individual backend model specifications
    │   ├── openai.toml         # OpenAI models (LLMs, image generation)
    │   ├── anthropic.toml      # Anthropic models (LLMs)
    │   ├── bedrock.toml        # Amazon Bedrock models (LLMs)
    │   ├── mistral.toml        # Mistral models (LLMs, OCR)
    │   ├── vertexai.toml       # Google Vertex AI models (LLMs)
    │   ├── fal.toml            # FAL models (image generation)
    │   ├── linkup.toml          # Linkup models (web search)
    │   ├── internal.toml       # Internal/local models (OCR)
    │   └── ...
    └── deck/                   # Model deck configurations
        ├── 1_llm_deck.toml           # LLM aliases & presets
        ├── 2_img_gen_deck.toml       # Image generation config
        ├── 3_extract_deck.toml       # Document extraction config
        ├── 4_search_deck.toml        # Web search config
        ├── x_custom_llm_deck.toml    # Custom LLM waterfalls/overrides
        └── x_custom_extract_deck.toml # Custom extract waterfalls
```

Deck files are loaded in order by their numeric prefix (`1_`, `2_`, `3_`), with custom/override files (`x_` prefix) loaded last.

!!! tip "Numbered files are pipelex-managed; overrides go in `x_custom_*.toml`"
    The numbered deck files (`1_llm_deck.toml`...`4_search_deck.toml`) are refreshed by `pipelex update` when a new release ships an updated deck. Local edits to those files are preserved with a timestamped `.bak` backup but will not survive future updates.

    To customize aliases, presets, or default choices without conflict, edit (or create) any file in this directory whose name starts with `x_custom_` — Pipelex never tracks or overwrites those. See [`pipelex update`](../../tools/cli/update.md) for the full workflow.

## Pipelex Gateway (Optional & Free)

Pipelex Gateway is a unified inference backend that provides access to all major AI providers through a single API key. This is the **recommended approach for getting started quickly** with Pipelex and **unlocking its full power**—with many models already available and new ones being added constantly. Using Pipelex Gateway is **completely optional**.

### Benefits

- **Single API Key**: Access OpenAI, Anthropic, Google, Mistral, FAL, and more with one key
- **Free to Get Started**: Available free on [app.pipelex.com](https://app.pipelex.com/) (no credit card required, limited time offer)
- **Simplified Configuration**: No need to manage multiple provider credentials
- **Automatic Routing**: All AI models (LLMs, OCR, image generation) are automatically routed to their respective providers
- **Unified Interface**: Same configuration system for text generation, OCR, and image generation

### Terms of Service & Telemetry

Using Pipelex Gateway requires accepting our terms of service. When you enable Gateway and run `pipelex init`, you'll be prompted with a terms panel explaining:

- **What we collect**: Model names, token counts, latency, error rates (technical data only)
- **What we do NOT collect**: Your prompts, completions, pipe codes, or business data
- **Why**: To monitor service quality, enforce fair usage, and provide better support
- **Your choice**: If you decline, Gateway is disabled and you can use direct provider backends instead

Your API key is hashed for security. Gateway telemetry operates independently from your `telemetry.toml` settings. See our [Privacy Policy](https://go.pipelex.com/privacy-policy) for details.

### Setup

1. **Get your API key:**

    - Visit [https://go.pipelex.com/discord](https://go.pipelex.com/discord) to join our Discord
    - Request your free API key in the appropriate channel
    - No credit card required (limited time offer)

2. **Configure environment variables:**

    ```bash
    # Copy the example environment file
    cp .env.example .env
    
    # Edit .env and add your Pipelex Gateway API key
    PIPELEX_GATEWAY_API_KEY="your-api-key"
    ```

3. **Accept terms and verify configuration:**

    ```bash
    pipelex init
    ```
    
    When prompted, review and accept the Gateway terms of service.

4. **Verify backend configuration:**
   
   The `pipelex_gateway` backend should be enabled in `.pipelex/inference/backends.toml`:
   
   ```toml
   [pipelex_gateway]
   display_name = "⭐ Pipelex Gateway"
   enabled = true
   api_key = "${PIPELEX_GATEWAY_API_KEY}"
   ```
   
   The environment variable `${PIPELEX_GATEWAY_API_KEY}` will be automatically loaded from your environment.

4. **Verify routing configuration:**

   The default routing profile in `.pipelex/inference/routing_profiles.toml` should be set to `all_pipelex_gateway`:

   ```toml
   active = "all_pipelex_gateway"

   [profiles.all_pipelex_gateway]
   description = "Use Pipelex Gateway for all its supported models"
   default = "pipelex_gateway"
   ```

### Usage

Once configured, all models are available through the unified backend. Use standard model names in your pipelines:

```toml
[pipe.example]
type = "PipeLLM"
model = { model = "claude-4.5-sonnet", temperature = 0.7 }
# Model automatically routed through Pipelex Gateway
```

### Gateway Model Overrides

!!! warning "Advanced Feature - Use at Your Own Risk"
    The Pipelex Gateway model configuration is fetched remotely from Pipelex servers. Any local override may cause unexpected behavior or failures, as the remote configuration may change at any time.

If you need to customize how a specific model behaves through the Gateway, you can add per-model overrides in `.pipelex/inference/backends/pipelex_gateway.toml`. However, only two keys are supported:

- `sdk`: The SDK to use for the model (e.g., `gateway_completions`)
- `structure_method`: The method for structured output (e.g., `instructor/openai_tools`)

All other keys will be ignored.

```toml
# .pipelex/inference/backends/pipelex_gateway.toml

# Per-model overrides example:
[gpt-4o]
sdk = "gateway_completions"
structure_method = "instructor/openai_tools"
```

!!! tip "Prefer Direct Backends for Custom Configurations"
    If you need custom configurations beyond `sdk` and `structure_method`, consider using your own API keys with direct provider backends (openai, anthropic, etc.) instead of Gateway overrides.

### Model Availability Note

While Pipelex Gateway provides access to most AI models through a unified API, certain specialized models require their native backend to be enabled directly:

- **FAL image generation models** (e.g., Flux models) - Enable the FAL backend
- **OpenAI image generation** (`gpt-image-1`) - Enable the OpenAI backend (should also work via Azure OpenAI, but we haven't been able to test this - if you've successfully used it on Azure, please let us know on [Discord](https://go.pipelex.com/discord) so we can validate this configuration)
- **Mistral OCR models** - Enable the Mistral backend

These models are not proxied through Pipelex Gateway and require direct configuration of their respective backends with appropriate API keys.

## Inference Backends

Backends represent AI service providers that can offer LLMs, OCR models, or image generation models. Each backend is configured with its endpoint and authentication details.

### Backend Configuration

#### Step 1: Configure Environment Variables

First, set up your API keys in the `.env` file:

```bash
# Copy the example file
cp .env.example .env

# Edit .env and add your provider API keys
```

> **Note:** Pipelex automatically loads environment variables from `.env` files using python-dotenv. No need to manually source or export them.

The `.env.example` file contains all available providers with helpful comments:

```bash
# [OPTIONAL] Free Pipelex Gateway API key - Get yours on Discord: https://go.pipelex.com/discord
PIPELEX_GATEWAY_API_KEY=

OPENAI_API_KEY=

# Amazon Bedrock - For accessing models via Amazon Bedrock
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_REGION=

ANTHROPIC_API_KEY=
MISTRAL_API_KEY=

# Google AI Studio - Use GOOGLE_API_KEY for direct API access (simpler, rate-limited)
GOOGLE_API_KEY=

# Google Cloud Platform (GCP) - Use these for production Vertex AI access
# Choose GOOGLE_API_KEY OR GCP credentials, not both
GCP_PROJECT_ID=
GCP_LOCATION=
GCP_CREDENTIALS_FILE_PATH=gcp_credentials.json

FAL_API_KEY=

LINKUP_API_KEY=
# ... (see .env.example for full list)
```

#### Step 2: Enable/Disable Backends

Configure which backends to use in `.pipelex/inference/backends.toml`:

```toml
[openai]
enabled = true  # Set to false to disable
api_key = "${OPENAI_API_KEY}"

[anthropic]
enabled = true
api_key = "${ANTHROPIC_API_KEY}"

[mistral]
enabled = true
api_key = "${MISTRAL_API_KEY}"

[fal]
enabled = true
api_key = "${FAL_API_KEY}"

[linkup]
enabled = true
api_key = "${LINKUP_API_KEY}"

[internal]
enabled = true
# No API key needed for internal/local processing
```

The `${VARIABLE_NAME}` syntax automatically loads values from your `.env` file. Set `enabled = true` to activate a backend, or `false` to disable it.

### Model Specifications

Each backend has its own model specification file in `.pipelex/inference/backends/`:

```toml
# openai.toml
default_sdk = "openai"
default_prompting_target = "openai"

[gpt-4o-mini]
model_id = "gpt-4o-mini"
inputs = ["text", "images"]
outputs = ["text", "structured"]
costs = { input = 0.15, output = 0.6 }

[gpt-4-turbo]
model_id = "gpt-4-turbo"
inputs = ["text"]
outputs = ["text", "structured"]
costs = { input = 10.0, output = 30.0 }

[gpt-image-1]
model_id = "gpt-image-1"
inputs = ["text"]
outputs = ["image"]
costs = { input = 0.04, output = 0.0 }
```

## Routing Profiles

Routing profiles determine which backend handles specific models. This is where you configure the **Mix & Match approach** (Option C) to optimize your setup. Configure them in `.pipelex/inference/routing_profiles.toml`:

### Profile Examples

**All Pipelex Gateway (Option A):**

Setup:
```bash
# In .env
PIPELEX_GATEWAY_API_KEY="your-pipelex-key"
```

In `.pipelex/inference/routing_profiles.toml`:
```toml
# Which profile to use
active = "all_pipelex_gateway"

[profiles.all_pipelex_gateway]
description = "Use Pipelex Gateway for all its supported models"
default = "pipelex_gateway"
```

**Native Providers Only (Option B):**

Setup:
```bash
# In .env - add all provider keys you need
OPENAI_API_KEY="your-openai-key"
ANTHROPIC_API_KEY="your-anthropic-key"
GOOGLE_API_KEY="your-google-key"
FAL_API_KEY="your-fal-key"
```

In `.pipelex/inference/routing_profiles.toml`:
```toml
active = "custom_routing"

[profiles.custom_routing]
description = "Route models to their native providers"
default = "openai"

[profiles.custom_routing.routes]
"claude-*" = "anthropic"
"gemini-*" = "google"
"mistral-*" = "mistral"
"gpt-*" = "openai"
"gpt-image-*" = "openai"
"flux-*" = "fal"
```

**Mix & Match (Option C):**

Setup:
```bash
# In .env - combine Pipelex with specific provider keys
PIPELEX_GATEWAY_API_KEY="your-pipelex-key"
OPENAI_API_KEY="your-openai-key"  # For GPT models
FAL_API_KEY="your-fal-key"        # For image generation
```

In `.pipelex/inference/routing_profiles.toml`:
```toml
active = "hybrid"

[profiles.hybrid]
description = "Use Pipelex Gateway for most models, native providers for specific ones"
default = "pipelex_gateway"

[profiles.hybrid.routes]
# Use your own OpenAI key for GPT models (better rate limits)
"gpt-*" = "openai"
# Use your own FAL key for image generation (direct billing)
"flux-*" = "fal"
# All other models use Pipelex Gateway (claude, gemini, mistral, etc.)
```

### Routing System Features

The routing system supports:

- **Exact matches**: `"gpt-4o-mini" = "openai"`
- **Wildcard patterns**: 
  - Prefix: `"gpt-*" = "openai"`
  - Suffix: `"*-turbo" = "openai"`
  - Contains: `"*-vision-*" = "openai"`
- **Default fallback**: `default = "pipelex_gateway"`

### Use Cases for Mix & Match

Common scenarios for hybrid routing:

1. **Cost Optimization**: Use Pipelex Gateway for expensive models, your own keys for cheaper ones
2. **Rate Limits**: Use your own keys for high-volume models to avoid shared rate limits
3. **Gradual Migration**: Start with Pipelex Gateway, gradually move to your own keys as usage grows
4. **Provider Features**: Use native providers for models requiring specific features not proxied through Pipelex Gateway

### Internal Backend (Always Available)

The **internal backend** is a special backend containing software-only models that run locally without requiring AI services. These include models for PDF text extraction, local document parsing, and other processing tasks that don't need external API calls.

Unlike other backends, internal backend models are **always available** regardless of which routing profile you select. This means you can use these models even when your routing profile is focused on a specific AI provider (e.g., `all_pipelex_gateway` or `all_openai`).

This behavior is automatic and requires no additional configuration. To see which models are available from the internal backend, check `.pipelex/inference/backends/internal.toml`.

## Model Deck

The Model Deck is the unified configuration hub for all AI model-related settings, including LLMs, OCR models, and image generation models.

### Aliases

Define user-friendly names that map to model names. Aliases are defined in the deck files (e.g., `.pipelex/inference/deck/1_llm_deck.toml`):

```toml
[llm.aliases]
# Simple aliases map to a single model
best-claude = "claude-4.8-opus"
best-gpt = "gpt-5.5"
best-gemini = "gemini-pro-latest"

# Default aliases (used in presets)
default-general = "claude-4.6-sonnet"
default-premium = "claude-4.8-opus"
default-large-context-text = "gemini-flash-latest"
default-small = "gpt-4o-mini"
```

When using aliases in `.mthds` files or other configurations, prefix them with `@`:

```toml
model = "@best-claude"           # References the best-claude alias
model = "@default-general"       # References the default-general alias
```

### LLM Presets

Presets combine model selection with optimized parameters for specific tasks. Defined in `.pipelex/inference/deck/1_llm_deck.toml`:

```toml
[llm.presets]
# Writing presets
writing-factual = { model = "@default-premium", temperature = 0.1 }
writing-creative = { model = "@default-premium", temperature = 0.9 }

# Retrieval
retrieval = { model = "@default-large-context-text", temperature = 0.1 }

# Engineering
engineering-structured = { model = "@default-premium-structured", temperature = 0.2 }
engineering-code = { model = "@default-premium", temperature = 0.1 }

# Vision
vision = { model = "@default-premium-vision", temperature = 0.5 }
vision-cheap = { model = "@default-small-vision", temperature = 0.5 }
vision-diagram = { model = "@default-premium-vision", temperature = 0.3 }
```

When using presets in `.mthds` files, prefix them with `$`:

```toml
model = "$engineering-structured"   # Uses preset for structured extraction
model = "$vision"                   # Uses preset for image-to-text
model = "$writing-creative"         # Uses preset for creative writing
```

### Extract Presets

Extract presets combine document extraction model selection with optimized parameters. Defined in `.pipelex/inference/deck/3_extract_deck.toml`:

```toml
[extract.presets]
# Testing preset
extract-testing = { model = "@default-extract-document", max_nb_images = 5, image_min_size = 50 }
```

You can also use aliases directly in `.mthds` files for document extraction:

```toml
model = "@default-extract-document"   # Uses default document extraction alias
model = "@default-text-from-pdf"      # Uses alias for basic PDF text extraction
```

### Image Generation Presets

Image generation presets combine model selection with generation parameters. Defined in `.pipelex/inference/deck/2_img_gen_deck.toml`:

```toml
[img_gen.presets]
# General purpose image generation
gen-image = { model = "@default-general", quality = "medium" }
gen-image-fast = { model = "@default-small", quality = "low" }
gen-image-high-quality = { model = "@default-premium", quality = "high" }
```

When using image generation presets in `.mthds` files, prefix them with `$`:

```toml
model = "$gen-image"              # Uses default image generation preset
model = "$gen-image-fast"         # Uses fast image generation preset
model = "$gen-image-high-quality" # Uses high quality image generation preset
```

### Search Presets

Search presets combine a search model with result options. Defined in `.pipelex/inference/deck/4_search_deck.toml`:

```toml
[search.presets]
standard = { model = "linkup-standard", include_images = false, include_inline_citations = true }
deep = { model = "linkup-deep", include_images = false, include_inline_citations = true }
```

When using search presets in `.mthds` files, prefix them with `$`:

```toml
model = "$standard"    # Standard web search
model = "$deep"        # More thorough web search
```

Search presets support the following options:

- `model`: The search model to use (e.g., `linkup-standard`, `linkup-deep`)
- `include_images`: Whether to include images in search results
- `include_inline_citations`: Whether to include inline citations in the answer

### Default Choices

Set default models for different types of AI operations:

```toml
[llm.choice_defaults]
for_text = "@default-general"
for_object = "@default-general"

[extract]
choice_default = "@default-extract-document"

[img_gen]
choice_default = "$gen-image"

[search]
choice_default = "@default-search"
```

Note the sigil prefixes: `@` for aliases and `$` for presets.

## Customization

### Local Overrides

Use custom deck files (prefixed with `x_`) for project-specific customizations:

**For LLMs** (`.pipelex/inference/deck/x_custom_llm_deck.toml`):

```toml
# Override default choices
[llm.choice_overrides]
for_text = "@my-custom-alias"
for_object = "@my-custom-alias"

# Add custom waterfalls - lists of models tried in order
[llm.waterfalls]
premium-llm = ["claude-4.5-opus", "gemini-3.1-pro", "gpt-5.2"]
small-llm = ["gemini-2.5-flash-lite", "gpt-4o-mini", "claude-3-haiku"]
```

**For Extract** (`.pipelex/inference/deck/x_custom_extract_deck.toml`):

```toml
[extract.waterfalls]
document_extractor = ["azure-document-intelligence", "mistral-document-ai-2505"]
```

When using waterfalls in `.mthds` files, prefix them with `~`:

```toml
model = "~premium-llm"    # Will try claude-4.5-opus, then gemini-3.1-pro, then gpt-5.2
model = "~small-llm"      # Will try gemini-2.5-flash-lite, then gpt-4o-mini, etc.
```

### Adding New Backends

To add a new backend:

1. Add backend configuration to `backends.toml`
2. Create model specification file in `backends/` directory
3. Update routing profile if needed

## Loading Process

The system loads configurations in this order:

1. **Load Backends**: Read `backends.toml` to get enabled backends
2. **Load Model Specs**: For each backend, load model specifications (LLMs, OCR models, image generation models)
3. **Load Routing Profiles**: Read routing rules and identify active profile
4. **Build Model Deck**: 
   - Apply routing rules to determine backend for each model across all AI capabilities
   - Load aliases and presets from deck files for LLMs, OCR, and image generation
   - Apply overrides
5. **Finalization**: Validate complete configuration

## Error Handling

Common error types:

- `ModelDeckNotFoundError`: Missing LLM deck configuration files
- `ModelNotFoundError`: Referenced model not found in the deck
- `LLMHandleNotFoundError`: Referenced model or alias not found
- `ModelChoiceNotFoundError`: Referenced preset/choice not found

## Best Practices

1. **Choosing Your Configuration Approach**:
   - **Starting out?** Use Pipelex Gateway (Option A) to get running quickly
   - **Production deployment?** Consider bringing your own keys (Option B) for direct billing control
   - **Optimizing costs/performance?** Use Mix & Match (Option C) for maximum flexibility
   - You can switch between approaches at any time by changing your routing profile

2. **Backend Management**:
   - Keep API keys in environment variables (never commit them)
   - Enable only the backends you need to reduce configuration complexity
   - Document custom backend configurations for your team

3. **Model Routing**:
   - Use specific routing profiles for different environments (dev, staging, prod)
   - Test routing rules before production deployment
   - Consider cost implications when routing models (some providers are cheaper for certain models)
   - Monitor usage patterns to optimize your routing strategy

4. **Presets and Aliases**:
   - Create task-specific presets for consistency across your pipelines
   - Use kebab-case naming (e.g., `engineering-structured`, `vision-diagram`)
   - Use proper sigil prefixes: `$` for presets, `@` for aliases, `~` for waterfalls
   - Document custom presets and their use cases in your team documentation

5. **Customization**:
   - Use `x_custom_*.toml` deck files for project-specific settings
   - Keep base configurations unchanged to make upgrades easier
   - Version control your custom configurations
   - Share routing profiles and presets across your team

## Related Documentation

- [LLM Integration](../../features/llm-integration.md) - Overview of LLM provider support and capabilities
- [PipeLLM Operator](../../building-methods/pipes/pipe-operators/PipeLLM.md) - The pipe operator that uses inference backends
- [Configure AI Providers](../../get-started/configure-ai-providers.md) - Setup guide for connecting AI providers


## Library

# Library Directories and Bundle Loading

When running pipelines, Pipelex needs to find your `.mthds` bundle files. There are two approaches:

1. **Point to the bundle file directly** - The simplest option. Just pass the path to your `.mthds` file. No configuration needed.

2. **Configure library directories** - For larger projects. Pipelex scans directories to discover all bundles, letting you reference pipes by code.

Most users should start with the first approach.

## The Simplest Way: Use the Bundle Path Directly

If you just want to run a pipe from a single `.mthds` file, **you don't need any library configuration**. Simply point to your bundle file:

```bash
# CLI: run the bundle's main_pipe
pipelex run bundle path/to/my_bundle.mthds

# CLI: run a specific pipe from the bundle
pipelex run bundle path/to/my_bundle.mthds --pipe my_pipe
```

```python
from pipelex.pipeline.runner import PipelexMTHDSProtocol

# Python: run the bundle's main_pipe
runner = PipelexMTHDSProtocol(
    bundle_uris=["path/to/my_bundle.mthds"],
)
response = await runner.execute(
    inputs={...},
)
pipe_output = response.pipe_output

# Python: run a specific pipe from the bundle
runner = PipelexMTHDSProtocol(
    bundle_uris=["path/to/my_bundle.mthds"],
)
response = await runner.execute(
    pipe_code="my_pipe",
    inputs={...},
)
pipe_output = response.pipe_output
```

This is the recommended approach for newcomers and simple projects. Pipelex reads the file directly - no discovery needed.

!!! tip "When to use library directories"
    The library directory configuration below is useful when you have **multiple bundles across different directories** and want to reference pipes by code without specifying the bundle path each time. For most use cases, pointing to the `.mthds` file directly is simpler.

---

## How Pipeline Discovery Works

When you initialize Pipelex with `Pipelex.make()`, the system:

1. **Scans your project directory** for all `.mthds` files
2. **Discovers Python structure classes** that inherit from `StructuredContent`
3. **Loads pipeline definitions** including domains, concepts, and pipes
4. **Registers custom functions** decorated with `@pipe_func()`

All of this happens automatically - no configuration needed.

## Configuring Library Directories

When executing pipelines, Pipelex needs to know where to find your `.mthds` files and Python structure classes. You can configure this using a **3-tier priority system** that gives you flexibility from global defaults to per-execution overrides.

### The 3-Tier Priority System

Pipelex resolves library directories using this priority order (highest to lowest):

| Priority | Source | When to Use |
|----------|--------|-------------|
| **1 (Highest)** | Per-call `library_dirs` parameter | Override for specific pipeline executions |
| **2** | Instance default via `Pipelex.make(library_dirs=...)` | Application-wide default |
| **3 (Fallback)** | `PIPELEXPATH` environment variable | System-wide or shell session default |

!!! info "Empty List is Valid"
    Passing an empty list `[]` to `library_dirs` is a valid explicit value that **disables** directory-based library loading. This is useful when using `mthds_contents` directly without needing files from the filesystem.

### Using the PIPELEXPATH Environment Variable

The `PIPELEXPATH` environment variable provides a system-wide default for library directories. It uses PATH-style syntax:

- **Unix/macOS**: Colon-separated paths (`:`)
- **Windows**: Semicolon-separated paths (`;`)

**Bash / Zsh (macOS/Linux):**

```bash
# Single directory
export PIPELEXPATH="/path/to/my/pipelines"

# Multiple directories (colon-separated)
export PIPELEXPATH="/path/to/shared_pipes:/path/to/project_pipes"

# Add to your ~/.bashrc or ~/.zshrc for persistence
echo 'export PIPELEXPATH="/path/to/my/pipelines"' >> ~/.bashrc
```

**PowerShell (Windows):**

```powershell
# Single directory
$env:PIPELEXPATH = "C:\path\to\my\pipelines"

# Multiple directories (semicolon-separated)
$env:PIPELEXPATH = "C:\shared_pipes;C:\project_pipes"
```

**In a `.env` file:**

```bash
# .env file in your project root
PIPELEXPATH=/path/to/shared_pipes:/path/to/project_pipes
```

### Using the CLI `--library-dir` Option

When running pipelines from the command line, use the `-L` or `--library-dir` option. This option can be specified multiple times to include multiple directories.

```bash
# Single directory
pipelex run pipe my_pipe -L /path/to/pipelines

# Multiple directories
pipelex run pipe my_pipe -L /path/to/shared_pipes -L /path/to/project_pipes

# Combined with other options
pipelex run bundle my_bundle.mthds --inputs data.json -L /path/to/pipelines

# Available on multiple commands
pipelex validate --all -L /path/to/pipelines/dir
pipelex show pipe --all -L /path/to/pipelines/dir
pipelex which my_pipe -L /path/to/pipelines/dir
```

!!! tip "CLI Option Overrides PIPELEXPATH"
    When you use `-L` on the command line, it takes precedence over any `PIPELEXPATH` environment variable that may be set.

### Setting Instance Defaults with `Pipelex.make()`

For Python applications, you can set a default library directory when initializing Pipelex. This default will be used for all subsequent `PipelexMTHDSProtocol.execute()` calls unless overridden.

```python
from pipelex.pipelex import Pipelex
from pipelex.pipeline.runner import PipelexMTHDSProtocol

# Set instance-level defaults at initialization
Pipelex.make(
    library_dirs=["/path/to/shared_pipes", "/path/to/project_pipes"]
)

# All PipelexMTHDSProtocol.execute() calls will use these directories by default
runner = PipelexMTHDSProtocol()
response = await runner.execute(
    pipe_code="my_pipe",
    inputs={"input": "value"},
)
pipe_output = response.pipe_output
```

### Per-Runner Override with `PipelexMTHDSProtocol`

For maximum flexibility, you can override library directories on each `PipelexMTHDSProtocol` instance:

```python
from pipelex.pipelex import Pipelex
from pipelex.pipeline.runner import PipelexMTHDSProtocol

# Initialize with default directories
Pipelex.make(
    library_dirs=["/path/to/default_pipes"]
)

# Use the default directories
runner1 = PipelexMTHDSProtocol()
response1 = await runner1.execute(
    pipe_code="default_pipe",
    inputs={"input": "value"},
)
pipe_output1 = response1.pipe_output

# Override for a specific execution
runner2 = PipelexMTHDSProtocol(
    library_dirs=["/path/to/special_pipes"],  # Overrides instance default
)
response2 = await runner2.execute(
    pipe_code="special_pipe",
    inputs={"input": "value"},
)
pipe_output2 = response2.pipe_output

# Disable directory loading (use only mthds_contents)
runner3 = PipelexMTHDSProtocol(
    library_dirs=[],  # Empty list disables directory-based loading
)
response3 = await runner3.execute(
    mthds_contents=[my_mthds_string],
    inputs={"input": "value"},
)
pipe_output3 = response3.pipe_output
```

### Priority Resolution Examples

**Example 1: Environment variable as fallback**

```bash
# Shell: Set PIPELEXPATH
export PIPELEXPATH="/shared/pipes"
```

```python
from pipelex.pipeline.runner import PipelexMTHDSProtocol

# Python: No library_dirs specified anywhere
Pipelex.make()  # No library_dirs

# Uses PIPELEXPATH: /shared/pipes
runner = PipelexMTHDSProtocol()
response = await runner.execute(pipe_code="my_pipe", inputs={...})
pipe_output = response.pipe_output
```

**Example 2: Instance default overrides PIPELEXPATH**

```bash
# Shell: PIPELEXPATH is set
export PIPELEXPATH="/shared/pipes"
```

```python
from pipelex.pipeline.runner import PipelexMTHDSProtocol

# Python: Instance default set
Pipelex.make(library_dirs=["/project/pipes"])

# Uses instance default: /project/pipes (PIPELEXPATH ignored)
runner = PipelexMTHDSProtocol()
response = await runner.execute(pipe_code="my_pipe", inputs={...})
pipe_output = response.pipe_output
```

**Example 3: Per-call override takes highest priority**

```python
from pipelex.pipeline.runner import PipelexMTHDSProtocol

Pipelex.make(library_dirs=["/default/pipes"])

# Uses per-runner value: /special/pipes
runner = PipelexMTHDSProtocol(
    library_dirs=["/special/pipes"],  # Highest priority
)
response = await runner.execute(
    pipe_code="my_pipe",
    inputs={...},
)
pipe_output = response.pipe_output
```

### Best Practices

1. **Use PIPELEXPATH for shared libraries**: Set it in your shell profile for directories that should always be available.

2. **Use `Pipelex.make(library_dirs=...)` for application defaults**: When building an application, set sensible defaults at initialization.

3. **Use per-call `library_dirs` for exceptions**: Override only when a specific execution needs different directories.

4. **Use empty list `[]` for isolated execution**: When you want to execute only from `mthds_contents` without loading any file-based definitions.

5. **Include structure class directories**: Remember that `library_dirs` must contain both `.mthds` files AND Python files defining `StructuredContent` classes.

## Excluded Directories

To improve performance and avoid loading unnecessary files, Pipelex automatically excludes common directories from discovery:

- `.venv`, `venv`, `env`, `virtualenv`, `.virtualenv` - Virtual environments
- `.git` - Git repository data
- `__pycache__` - Python bytecode cache
- `.pytest_cache` - Pytest cache
- `.mypy_cache` - Mypy type checker cache
- `.ruff_cache` - Ruff linter cache
- `node_modules` - Node.js dependencies
- `.env` - Environment files
- `results` - Common output directory

Files in these directories will not be scanned, even if they contain `.mthds` files or structure classes.

## Project Organization

**Golden rule:** Put `.mthds` files where they make sense in YOUR project. Pipelex finds them automatically.

### Common Patterns

**1. Topic-Based (Recommended for structured projects)**

Keep pipelines with related code:

```
your_project/
├── my_project/
│   ├── finance/
│   │   ├── models.py
│   │   ├── services.py
│   │   ├── invoices.mthds          # With finance code
│   │   └── invoices_struct.py
│   └── legal/
│       ├── models.py
│       ├── contracts.mthds         # With legal code
│       └── contracts_struct.py
├── .pipelex/
└── requirements.txt
```

**Benefits:**

- Related things stay together
- Easy to find pipeline for a specific module
- Natural code organization

**2. Centralized (For simpler discovery)**

Group all pipelines in one place:

```
your_project/
├── my_project/
│   ├── pipelines/          # All pipelines here
│   │   ├── finance.mthds
│   │   ├── finance_struct.py
│   │   ├── legal.mthds
│   │   └── legal_struct.py
│   └── core/
└── .pipelex/
```

**Benefits:**

- All pipelines in one location
- Simple structure for small projects

## Alternative Structures

Pipelex supports flexible organization. Here are other valid approaches:

### Feature-Based Organization

```
your_project/
├── my_project/
│   ├── features/
│   │   ├── document_processing/
│   │   │   ├── extract.mthds
│   │   │   └── extract_struct.py
│   │   └── image_generation/
│   │       ├── generate.mthds
│   │       └── generate_struct.py
│   └── main.py
└── .pipelex/
```

### Domain-Driven Organization

```
your_project/
├── my_project/
│   ├── finance/
│   │   ├── pipelines/
│   │   │   └── invoices.mthds
│   │   └── invoice_struct.py
│   ├── legal/
│   │   ├── pipelines/
│   │   │   └── contracts.mthds
│   │   └── contract_struct.py
│   └── main.py
└── .pipelex/
```

### Flat Organization (Small Projects)

```
your_project/
├── my_project/
│   ├── invoice_processing.mthds
│   ├── invoice_struct.py
│   └── main.py
└── .pipelex/
```

## Loading Process

Pipelex loads your pipelines in a specific order to ensure dependencies are resolved correctly:

### 1. Domain Loading

- Loads domain definitions from all `.mthds` files
- Each domain must be defined exactly once
- Supports system prompts and structure templates per domain

### 2. Concept Loading

- Loads native concepts (Text, Image, Document, etc.)
- Loads custom concepts from `.mthds` files
- Validates concept definitions and relationships
- Links concepts to Python structure classes by name

### 3. Structure Class Registration

- Discovers all classes inheriting from `StructuredContent`
- Registers them in the class registry
- Makes them available for structured output generation

### 4. Pipe Loading

- Loads pipe definitions from `.mthds` files
- Validates pipe configurations
- Links pipes with their respective domains
- Resolves input/output concept references

### 5. Function Registration

- Discovers functions decorated with `@pipe_func()`
- Registers them in the function registry
- Makes them available for `PipeFunc` operators

## Custom Function Registration

For custom functions used in `PipeFunc` operators, add the `@pipe_func()` decorator:

```python
from pipelex.system.registries.func_registry import pipe_func
from pipelex.core.memory.working_memory import WorkingMemory
from pipelex.core.stuffs.text_content import TextContent

@pipe_func()
async def my_custom_function(working_memory: WorkingMemory) -> TextContent:
    """
    This function is automatically discovered and registered.
    """
    input_data = working_memory.get_stuff("input_name")
    # Process data
    return TextContent(text=f"Processed: {input_data.content.text}")

# Optional: specify a custom name
@pipe_func(name="custom_processor")
async def another_function(working_memory: WorkingMemory) -> TextContent:
    # Implementation
    pass
```

## Validation

After making changes to your pipelines, validate them:

```bash
# Validate all pipelines
pipelex validate --all

# Validate a specific pipe
pipelex validate YOUR_PIPE_CODE

# Show all available pipes
pipelex show pipe --all

# Show details of a specific pipe
pipelex show pipe YOUR_PIPE_CODE
```

## Best Practices

### 1. Organization

- Keep related concepts and pipes in the same `.mthds` file
- Use meaningful domain codes that reflect functionality
- Match Python file names with MTHDS file names (`finance.mthds` → `finance.py`)
- Group complex pipelines using subdirectories

### 2. Structure Classes

- Only create Python classes when you need structured output
- Name classes to match concept names exactly
- Use `_struct.py` suffix for files containing structure classes (e.g., `finance_struct.py`)
- Inherit from `StructuredContent` or its subclasses
- Place structure class files near their corresponding `.mthds` files
- **Keep modules clean**: Avoid module-level code that executes on import (Pipelex imports modules during auto-discovery)

### 3. Custom Functions

- Always use the `@pipe_func()` decorator
- Use descriptive function names
- Document function parameters and return types
- Keep functions focused and testable
- **Keep modules clean**: Avoid module-level code that executes on import (Pipelex imports modules during auto-discovery)

### 4. Validation

- Run `pipelex validate --all` after making changes
- Check for domain code consistency
- Verify concept relationships
- Test pipes individually before composing them

## Troubleshooting

### Pipelines Not Found

**Problem:** Pipelex doesn't find your `.mthds` files.

**Solutions:**

1. Ensure files have the `.mthds` extension
2. Check that files are not in excluded directories
3. Verify file permissions allow reading
4. Run `pipelex show pipe --all` to see what was discovered

### Structure Classes Not Registered

**Problem:** Your Python classes aren't recognized.

**Solutions:**

1. Ensure classes inherit from `StructuredContent`
2. Check class names match concept names exactly
3. Verify files are not in excluded directories
4. Make sure class definitions are valid Python

### Custom Functions Not Found

**Problem:** `PipeFunc` can't find your function.

**Solutions:**

1. Add the `@pipe_func()` decorator
2. Ensure function signature matches requirements
3. Check function is `async` and accepts `working_memory`
4. Verify function is in a discoverable location

### Validation Errors

**Problem:** `pipelex validate --all` shows errors.

**Solutions:**

1. Read error messages carefully - they indicate the problem
2. Check concept references are spelled correctly
3. Verify pipe configurations match expected format
4. Ensure all required fields are present

## Related Documentation

- [Libraries](../../building-methods/libraries.md) - Understanding method libraries
- [Packages](../../building-methods/packages.md) - Organizing methods into packages


## plxt (Formatter & Linter)

# plxt (Formatter & Linter)

## Overview

`plxt` is a fast formatting and linting tool for TOML, MTHDS, and PLX files in Pipelex projects. It ensures consistent style across all configuration and pipeline definition files, powered by the [taplo](https://taplo.tamasfe.dev/) engine.

## Installation

`plxt` is included as a Pipelex development dependency. It is automatically installed into your virtual environment when you run:

```bash
make install
```

You can verify the installation with:

```bash
.venv/bin/plxt --help
```

## Configuration

`plxt` reads its configuration from `.pipelex/plxt.toml` at the root of your project. This file controls file discovery, formatting rules, and per-file-type overrides.

### File Discovery

The `include` and `exclude` top-level keys control which files `plxt` processes:

```toml
include = ["**/*.toml", "**/*.mthds", "**/*.plx"]

exclude = [
  ".venv/**",
  ".mypy_cache/**",
  ".ruff_cache/**",
  ".pytest_cache/**",
  "__pycache__/**",
  "target/**",
  "node_modules/**",
  ".git/**",
  "*.lock",
]
```

### Supported File Types

| Extension | Description |
|-----------|-------------|
| `.mthds` | Method definitions |
| `.plx` | Method definitions (legacy extension) |
| `.toml` | Standard TOML configuration files |

### Key Formatting Options

The `[formatting]` section in `plxt.toml` controls the global formatting behavior. Each option can be overridden per file type using `[[rule]]` sections.

| Option | Default | Description |
|--------|---------|-------------|
| `align_entries` | `false` | Align consecutive `key = value` entries so `=` signs line up |
| `align_comments` | `true` | Align end-of-line comments on consecutive lines |
| `array_trailing_comma` | `true` | Add a trailing comma after the last element in multiline arrays |
| `array_auto_expand` | `true` | Expand arrays to multiple lines when exceeding `column_width` |
| `column_width` | `80` | Target maximum line width used for auto-expand/collapse |
| `compact_arrays` | `true` | Omit spaces inside single-line array brackets |
| `trailing_newline` | `true` | Ensure files end with a newline character |
| `reorder_keys` | `false` | Sort top-level keys alphabetically |

For the full list of options, see the comments in `.pipelex/plxt.toml`.

### Per-File-Type Rules

You can define `[[rule]]` sections to apply different formatting settings to different file types. For example, the default configuration includes separate rules for `.toml` files and for `.mthds`/`.plx` files:

```toml
[[rule]]
include = ["**/*.toml"]
[rule.formatting]
# TOML-specific overrides here

[[rule]]
include = ["**/*.mthds", "**/*.plx"]
[rule.formatting]
align_entries = true
# ... more MTHDS/PLX-specific overrides
```

## Usage

### Command Line

Format all discovered files in place:

```bash
.venv/bin/plxt fmt
```

Check formatting without modifying files (useful for CI):

```bash
.venv/bin/plxt fmt --check
```

Lint all discovered files:

```bash
.venv/bin/plxt lint
```

### Make Targets

The following Make targets are available for convenience:

| Target | Description |
|--------|-------------|
| `make plxt-format` | Format all MTHDS/TOML/PLX files |
| `make plxt-lint` | Lint all MTHDS/TOML/PLX files |
| `make merge-check-plxt-format` | Check formatting without modifying files |
| `make merge-check-plxt-lint` | Run lint check |

`plxt` is also included in the composite check targets:

- `make c` (check) runs `plxt-format` and `plxt-lint` alongside ruff, pyright, and mypy
- `make agent-check` includes `plxt-format` and `plxt-lint` in the full quality pipeline


## Logging

# Pipelex Logging System

## Overview

Pipelex provides a sophisticated logging system that extends Python's standard logging with additional features like emoji support, rich formatting, and custom log levels.

## Log Levels

In addition to standard Python log levels, Pipelex introduces custom levels:

| Level | Value | Description |
|-------|-------|-------------|
| VERBOSE | 5 | Most detailed logging, below DEBUG |
| DEBUG | 10 | Standard debug information |
| DEV | 15 | Development-specific logging (between DEBUG and INFO) |
| INFO | 20 | General informational messages |
| WARNING | 30 | Warning messages |
| ERROR | 40 | Error messages |
| CRITICAL | 50 | Critical errors |
| OFF | 999 | Disable logging |

## Logging Features

### 1. Rich Formatting

- Beautiful console output with color coding
- Syntax highlighting for JSON and other data structures
- Support for clickable file paths
- Word wrapping for better readability

### 2. Emoji Support

Built-in emoji indicators for different components:

- 🧠 Pipelex core messages
- ⚪️ OpenAI-related logs
- 🌀 Google-related logs
- ⚡️ Network connections
- *️⃣ JSON processing
- 🧿 Poor-log channel (`#poor-log`, the simplified fallback logger)

### 3. Caller Information

Optional inclusion of caller information in logs:

- File name and line number
- Function name
- Module name
- Customizable format templates

### 4. Structured Data Logging

Intelligent handling of different data types:

- Pretty-printing for dictionaries and lists
- JSON formatting with customizable indentation
- Special handling for None values
- Exception traceback integration

### 5. Log Dispatch System

Smart routing of log messages:

- Automatic detection of log origin
- Separate handlers for different logging needs
- Configurable rich or poor log modes
- Support for poor loggers (simplified logging)

## Using the Logger

```python
from pipelex import log

# Basic logging
log.info("Simple message")

# Logging with title
log.info("Detailed message", title="Process Status")

# Logging with inline title
log.info("Quick update", inline="Status")

# Logging structured data
data = {"key": "value", "nested": {"data": True}}
log.verbose(data, title="Configuration")

# Warning with problem ID
log.warning("API rate limit approaching", problem_id="rate_limit_warning")

# Error with exception traceback
log.error("Failed to process", include_exception=True)

# Development logging
log.dev("Testing new feature")

# Verbose logging
log.verbose("Detailed debug information")
```

## Best Practices

1. **Log Level Selection**:

    - Use VERBOSE for detailed debugging
    - Use DEBUG for general debugging
    - Use DEV for development-specific logging
    - Use INFO for general progress
    - Use WARNING for potential issues
    - Use ERROR for actual errors
    - Use CRITICAL for system-critical issues

2. **Structured Data**:

    - Log complex data structures directly
    - Use titles for context
    - Include problem IDs for trackable issues

3. **Exception Handling**:

    - Use `include_exception=True` for error context
    - Include relevant data in error logs
    - Use appropriate log levels for exceptions

## Related Documentation

- [Logging Configuration](../configuration/config-practical/logging-config.md) - Configure log behavior in `pipelex.toml`
- [CLI](./cli/index.md) - Commands that surface runtime logs during development


## Observer Data Extraction

# Observer Data Extraction

The Observer system in Pipelex automatically collects execution data from your pipes for later analysis. This data extraction feature is currently a work in progress but provides valuable insights into pipeline performance, usage patterns, and debugging information.

## Data Collection Points

The observer extracts data at three critical moments during pipe execution:

### 1. Before Pipe Execution
- **When**: Just before a pipe starts running
- **Data Collected**: Initial working memory state, pipe configuration, input parameters... [WIP]
- **Purpose**: Capture the starting conditions and context for analysis

### 2. After Successful Execution
- **When**: When a pipe completes successfully
- **Data Collected**: Final working memory, outputs, execution metadata, performance metrics... [WIP]
- **Purpose**: Track successful operations and measure performance

### 3. After Failed Execution
- **When**: When a pipe encounters an error or exception
- **Data Collected**: Error context, partial results, failure metadata... [WIP]
- **Purpose**: Debug issues and analyze failure patterns

## Where Data is Stored

The data extraction location depends on your observer implementation:

### LocalObserver (opt-in)

Pipelex ships a `LocalObserver` that writes each event to JSONL files on the local filesystem, but it is not registered by default: out of the box, setup wires a no-op observer and a telemetry observer, so no JSONL files are written. To enable JSONL extraction, construct a `LocalObserver` and pass it through the `observers` parameter of `Pipelex.make()` (a mapping of name to observer — note that providing it replaces the default observers):

```python
from pipelex.observer.local_observer import LocalObserver
from pipelex.pipelex import Pipelex

Pipelex.make(observers={"local": LocalObserver(storage_dir="results/observer")})
```

Pass an explicit `storage_dir`: the observer is constructed before `Pipelex.make()` boots the runtime, so it cannot read the configured default at that point.

Once registered:

- **Location**: Local filesystem in JSONL format
- **Default Directory**: if you omit `storage_dir`, the directory comes from `observer_config.observer_dir` in your Pipelex settings — but that fallback only works once Pipelex is booted, not in the pre-boot construction shown above
- **Files Created**:

- `before_run.jsonl` - Pre-execution snapshots
- `after_successful_run.jsonl` - Success events and results
- `after_failing_run.jsonl` - Failure events and error context

### Custom Observers
You can implement custom data extraction to:

- **Databases**: PostgreSQL, MongoDB, InfluxDB for structured analysis
- **Cloud Storage**: S3, GCS for large-scale data retention
- **Streaming Systems**: Kafka, Redis for real-time processing
- **Monitoring Tools**: DataDog, New Relic for operational insights

## Current Status: Work in Progress

See [Observer Provider Injection](../advanced/observer-provider-injection.md) for implementation details.

## Telemetry

# Telemetry

Pipelex supports two independent telemetry streams that serve different purposes. Understanding how they work helps you make informed decisions about data collection.

## Two Telemetry Streams

### 1. Gateway Telemetry (Pipelex-Controlled)

When you use **Pipelex Gateway** as your inference backend, identified telemetry is **automatically enabled**. This telemetry is tied to your Gateway API key (hashed for security) and operates independently from your `telemetry.toml` settings.

**What we collect:**

- Model names used (e.g., `gpt-5.2`, `claude-4.5-sonnet`) and parameters
- Pipe types (e.g., `PipeLLM`, `PipeSequence`, etc.)
- Token counts (input/output)
- Latency metrics
- Error rates (without error details)

**What we do NOT collect:**

- Your prompts or completions
- Your pipe codes or output class names
- File contents or business data

This telemetry allows us to:

- Monitor and improve service quality
- Enforce fair usage limits and prevent abuse
- Provide you with usage insights and better support

!!! info "Gateway Telemetry is Optional"
    Using Pipelex Gateway is entirely optional. If you prefer not to send telemetry to Pipelex servers, simply use your own API keys with direct provider backends (OpenAI, Anthropic, Azure, Bedrock, etc.).

### 2. Custom Telemetry (User-Controlled)

Custom telemetry is configured in `.pipelex/telemetry.toml` and allows you to send observability data to **your own** analytics and monitoring systems:

- **PostHog**: Event tracking and AI span tracing with privacy controls
- **Langfuse**: Full LLM observability (receives full span data)
- **OTLP**: Send spans to any OpenTelemetry-compatible backend (receives full span data)

Custom telemetry is completely independent from Gateway telemetry—you can use both, either, or neither.

## Quick Setup

When you run `pipelex init`, a default `telemetry.toml` configuration file is created:

```bash
pipelex init telemetry
```

The behavior depends on the target:

- **Global init** (default — writes to `~/.pipelex/telemetry.toml`): an active template is created with all options disabled. Edit it to enable your preferred telemetry destinations machine-wide.
- **Project init** (`pipelex init telemetry --local` — writes to `{project_root}/.pipelex/telemetry.toml`): a fully **commented-out** template is created. The project file is layered on top of your global config, so leaving it commented means the project inherits your global telemetry settings as-is. Uncomment a key only when this project should diverge.

!!! note "Global vs project config"
    Pipelex looks for telemetry config in **both** `~/.pipelex/` (machine-wide) and `{project_root}/.pipelex/` (per-project). Both are deep-merged, with the project file winning on key collisions — your global Langfuse keys keep applying across every project unless a project explicitly overrides them. See [Telemetry Configuration → Global vs project config](../configuration/config-practical/telemetry-config.md#global-vs-project-config) for the full load order.

### Example: Enable PostHog Tracing

```toml
[custom_posthog]
mode = "anonymous"  # or "identified" with user_id
endpoint = "${POSTHOG_ENDPOINT}"
api_key = "${POSTHOG_API_KEY}"

[custom_posthog.tracing]
enabled = true

[custom_posthog.tracing.capture]
content = false        # Don't capture prompts/completions
pipe_codes = true      # Include pipe codes in span names
```

### Example: Enable Langfuse

```toml
[langfuse]
enabled = true
public_key = "${LANGFUSE_PUBLIC_KEY}"
secret_key = "${LANGFUSE_SECRET_KEY}"
```

## DO_NOT_TRACK Global Override

The `DO_NOT_TRACK` environment variable provides a universal way to disable **all** telemetry:

```bash
export DO_NOT_TRACK=1
```

When set, this disables:

- Gateway telemetry (but note: Gateway won't work without telemetry)
- All custom telemetry destinations

!!! warning "Gateway Requires Telemetry"
    If you set `DO_NOT_TRACK=1` while using Pipelex Gateway, the Gateway will not function. Use direct provider backends instead if you need to disable all telemetry.

## Privacy

We take your privacy seriously:

- **Gateway telemetry** never collects prompts, completions, or business data
- **Custom telemetry** gives you full control over what data is captured
- PostHog tracing includes privacy controls to redact sensitive content
- All telemetry can be completely disabled

For detailed configuration options, see [Telemetry Configuration](../configuration/config-practical/telemetry-config.md).

For more information about our data practices, see our [Privacy Policy](https://go.pipelex.com/privacy-policy).


## Gateway Available Models

--8<-- ".pipelex/inference/backends/pipelex_gateway_models.md"



# Advanced

## Overview

# Dependency Injection

## Overview

Pipelex uses dependency injection to manage service dependencies and make components more modular and testable. The system allows you to customize and extend Pipelex's functionality by injecting your own implementations of various services.

## Injecting your implementation

Inject at boot, by passing your implementation to `Pipelex.make()`:

```python
from pipelex.pipelex import Pipelex

pipelex = Pipelex.make(
    reporting_delegate=MyReportingDelegate(),
    secrets_provider=MySecretsProvider(),
    content_generator=MyContentGenerator(),
    pipe_router=MyPipeRouter()
)
```

This is the supported path. `Pipelex.make()` builds the hubs, applies your overrides in the right order, and wires the results into the components that consume them.

## Reading a dependency: the two hubs

Injection happens at boot; *lookup* happens through a hub. Pipelex has two, split by lifecycle:

- `pipelex.runtime_hub` — process-scoped infrastructure: config, console, secrets, storage, telemetry, the model deck, inference workers, the content generator, the plugin registries.
- `pipelex.interpreter_hub` — library-scoped method machinery: the library manager and the concept/domain/pipe libraries, the current-library binding, the pipe router, the pipeline manager.

Read a dependency through the module-level accessor for the half that owns it:

```python
from pipelex.interpreter_hub import get_required_pipe
from pipelex.runtime_hub import get_model_deck, get_secrets_provider

secrets = get_secrets_provider()
deck = get_model_deck()
pipe = get_required_pipe(pipe_code="my_domain.my_pipe")
```

!!! warning "Don't construct a hub yourself"
    Building a `RuntimeHub()` or `InterpreterHub()` by hand and calling setters on it has no effect on a running Pipelex — the instance you build is not the one the process resolves. Inject through `Pipelex.make()` instead.

Contributors adding a new dependency should read [Hub Layering](../contribute/hub-layering.md), which specifies which half a symbol belongs on and why the boundary is enforced.

## Protocol Compliance

All custom implementations MUST:

1. Subclass the protocol rather than rely on structural conformance — a protocol carrying concrete default bodies (`PipeRouterProtocol.run_batch_branch`) hands them down by inheritance only
2. Implement ALL methods defined in their respective protocols
3. Match the exact method signatures (parameter names and types)
4. Follow the protocol's documented behavior
5. Handle errors appropriately
6. Clean up resources when needed

## Available Injectable Components

Pipelex supports injection of the following components:

**Reporting Delegate** (`ReportingManager`)

- Protocol: `ReportingProtocol`
- Default: `ReportingManager`
- No-op option: inject `ReportingNoOp` to disable reporting explicitly
- [Details](reporting-delegate-injection.md)

**Secrets Provider** (`EnvSecretsProvider`)

- Protocol: `SecretsProviderAbstract`
- Default: `EnvSecretsProvider`
- [Details](secrets-provider-injection.md)

**Content Generator** (`ContentGenerator`)

- Protocol: `ContentGeneratorProtocol`
- Default: `ContentGenerator`
- [Details](content-generator-injection.md)

**Pipe Router** (`PipeRouter`)

- Protocol: `PipeRouterProtocol`
- Default: `PipeRouter`
- [Details](pipe-router-injection.md)


## Secrets Provider

# Secrets Provider Injection

!!! warning "Under Construction"
    This section is currently under development.


## Reporting Delegate

# Reporting Delegate Injection

!!! warning "Under Construction"
    This section is currently under development.


## Storage Provider

# Storage Provider Injection

!!! warning "Under Construction"
    This section is currently under development.


## Observer

# Observer Provider Injection

The Observer system in Pipelex allows you to monitor and collect data about pipe executions. This guide explains how to implement your own observer by following the dependency injection pattern and the ObserverProtocol.

## Overview

The Observer system is designed to collect execution data at three key moments:

- **Before running a pipe**: Capture initial state and inputs
- **After successful execution**: Record outputs and success metrics
- **After failed execution**: Log errors and failure context

This data collection is designed for later analysis and is still a work in progress.

## Observer Protocol

All observers must implement the `ObserverProtocol`:

```python
from typing import Any, Protocol

PayloadType = dict[str, Any]

class ObserverProtocol(Protocol):
    async def observe_before_run(self, payload: PayloadType) -> None:
        """Process and store the payload before the run"""
        ...

    async def observe_after_successful_run(self, payload: PayloadType) -> None:
        """Process and store the payload after the run is successful"""
        ...

    async def observe_after_failing_run(self, payload: PayloadType) -> None:
        """Process and store the payload after the run fails"""
        ...
```

## Payload Structure

The payload contains different information depending on the execution phase:

### Before Run
```python
payload = {
    "pipeline_run_id": str,  # Identifier of the pipeline run
    "pipe_job": PipeJob,  # Contains pipe metadata and working memory
}
```

### Successful Run
```python
payload = {
    "pipeline_run_id": str,  # Identifier of the pipeline run
    "pipe_job": PipeJob,     # Initial job information
    "pipe_output": PipeOutput, # Results and final working memory
}
```

### Failing Run
```python
payload = {
    "pipeline_run_id": str,  # Identifier of the pipeline run
    "pipe_job": PipeJob,  # Initial job information
    "error": Exception,   # The exception that caused the failure
}
```

## Implementing a Custom Observer

### Step 1: Create Your Observer Class

```python
import os
from typing import Optional
from pipelex.observer.observer_protocol import ObserverProtocol, PayloadType

class MyCustomObserver(ObserverProtocol):
    def __init__(self, config_param: str = "default_value"):
        self.config_param = config_param
        # Initialize your storage/logging mechanism here

    async def observe_before_run(self, payload: PayloadType) -> None:
        pipe_job = payload["pipe_job"]
        # Process pipe_job data before execution
        # Example: Log pipe code, inputs, parameters
        print(f"Starting pipe: {pipe_job.pipe.code}")

    async def observe_after_successful_run(self, payload: PayloadType) -> None:
        pipe_job = payload["pipe_job"]
        pipe_output = payload["pipe_output"]
        # Process successful execution data
        # Example: Log execution time, output size, tokens used
        print(f"Completed pipe: {pipe_job.pipe.code}")

    async def observe_after_failing_run(self, payload: PayloadType) -> None:
        pipe_job = payload["pipe_job"]
        # Process failure data
        # Example: Log error context, partial results
        print(f"Failed pipe: {pipe_job.pipe.code}")
```

### Step 2: Register Your Observer at Boot

Observers are injected at boot, through `Pipelex.make()`. Pass a dict keyed by a name of your choosing — Pipelex wraps the whole dict in a `MultiObserver` and hands it to the pipe router, so every observer you register is notified:

```python
from pipelex.pipelex import Pipelex

# Create your observer instance
my_observer = MyCustomObserver(config_param="production")

# Inject it at boot
Pipelex.make(observers={"my_observer": my_observer})
```

!!! warning "Observers must be registered at boot"
    Passing `observers` to `Pipelex.make()` is the only way to register one. The router receives its observer when it is constructed during setup, so there is no supported way to swap an observer on a already-running instance.

Registering your own `observers` dict **replaces** the defaults (a no-op observer and the telemetry observer). Include the built-ins explicitly if you want to keep them alongside your own.

### Step 3: Ensure Proper Integration

The observer is automatically called by the PipeRouterProtocol during pipe execution:

1. `observe_before_run()` is called before any pipe execution
2. `observe_after_successful_run()` is called after successful completion
3. `observe_after_failing_run()` is called when an exception occurs

## Built-in LocalObserver Example

Pipelex includes a `LocalObserver` that writes data to JSONL files:

```python
from pipelex.observer.local_observer import LocalObserver

# Uses default observer directory from config
local_observer = LocalObserver()

# Or specify custom directory
local_observer = LocalObserver(storage_dir="/path/to/custom/dir")

# Inject the observer
Pipelex.make(observers={"local": local_observer})
```

The LocalObserver creates separate JSONL files for each event type:

- `before_run.jsonl` - Pre-execution data
- `after_successful_run.jsonl` - Success events
- `after_failing_run.jsonl` - Failure events

## Advanced Use Cases

### Database Observer
```python
import asyncio
import aiohttp
from pipelex.observer.observer_protocol import ObserverProtocol, PayloadType

class DatabaseObserver(ObserverProtocol):
    def __init__(self, db_connection_string: str):
        self.db_connection = db_connection_string
        # Initialize database connection

    async def observe_before_run(self, payload: PayloadType) -> None:
        # Store in database
        await self._store_event("before_run", payload)

    async def observe_after_successful_run(self, payload: PayloadType) -> None:
        await self._store_event("after_successful_run", payload)

    async def observe_after_failing_run(self, payload: PayloadType) -> None:
        await self._store_event("after_failing_run", payload)

    async def _store_event(self, event_type: str, payload: PayloadType):
        # Your database storage logic here
        pass
```

### Metrics Observer
```python
class MetricsObserver(ObserverProtocol):
    def __init__(self, metrics_client):
        self.metrics = metrics_client

    async def observe_before_run(self, payload: PayloadType) -> None:
        pipe_job = payload["pipe_job"]
        self.metrics.increment(f"pipe.{pipe_job.pipe.code}.started")

    async def observe_after_successful_run(self, payload: PayloadType) -> None:
        pipe_job = payload["pipe_job"]
        self.metrics.increment(f"pipe.{pipe_job.pipe.code}.success")

    async def observe_after_failing_run(self, payload: PayloadType) -> None:
        pipe_job = payload["pipe_job"]
        self.metrics.increment(f"pipe.{pipe_job.pipe.code}.failed")
```

## Best Practices

1. **Keep observers lightweight**: Avoid blocking operations in observer methods
2. **Handle errors gracefully**: Observer failures shouldn't break pipe execution
3. **Use async/await properly**: All observer methods are async
4. **Consider storage performance**: High-frequency pipes generate lots of data
5. **Structure your data**: Plan how you'll query and analyze the collected data later

## Integration with Pipelex Initialization

For automatic observer setup, integrate with your Pipelex initialization:

```python
from pipelex.pipelex import Pipelex

from my_project.observers import MyCustomObserver

def setup_pipelex():
    # Initialize Pipelex with the observer wired in from the start
    return Pipelex.make(observers={"my_observer": MyCustomObserver()})
```

The observer system provides powerful insights into your pipeline execution patterns and is essential for monitoring, debugging, and optimizing your Pipelex methods.

## Content Generator

# Content Generator Injection

!!! warning "Under Construction"
    This section is currently under development.


## Pipe Router

# Pipe Router Injection

!!! warning "Under Construction"
    This section is currently under development.


# Under the Hood

## Overview

# Under the Hood

Welcome to the technical deep-dives of Pipelex. This section is for contributors, curious developers, AI agents, and anyone doing due diligence on how Pipelex works internally.

## What You'll Find Here

- **Architecture Overview** - The two-layer design and how components fit together
- **The Pipelex Kernel** - Operator semantics as importable functions, and how to call them without a loaded method
- **Build-time Elaboration** - How shorthand directives like `structuring_method = "preliminary_text"` are rewritten into concrete pipe trees before any pipe runs
- **Execution Graph Tracing** - How pipeline executions are captured as graphs for visualization
- **Image Handling in LLM Prompts** - How images flow from inputs to LLM calls
- **Reasoning Controls** - How reasoning effort/budget flows to each provider's SDK
- **Error Model** - How errors are classified, carried across every layer, and reported to humans, agents, and HTTP APIs
- **StuffArtefact & Image Rendering** - How template access and image extraction work
- **Test Profile Configuration** - How to configure which models are used in tests
- **Dry Run Mock Generation** - How mock objects satisfy field validation constraints
- **Init CLI Flows** - How `pipelex init` sets up the configuration directory
- **Runtime Bridge & Transport** - The backend-neutral seam that carries a pipe run across process boundaries — boundary DTOs, LibraryCrate propagation, deferred hydration, and per-call isolation
- **Content Generation Across Boundaries** - How dynamic classes and large payloads cross a worker boundary
- **Technical Design Decisions** - Why we chose X over Y
- **Module Deep-Dives** - Detailed explanations of specific subsystems

!!! info "Not Required for Using Pipelex"
    You don't need to read this section to use Pipelex effectively. The [Home](../index.md) section covers everything you need to build methods.

!!! tip "Looking for distributed execution?"
    For running Pipelex methods as durable, distributed workflows, see the user-facing [Distributed Execution](../distributed-execution/index.md) overview. The pages below cover the runtime mechanics that make distributed execution work under the hood.

---

## Start Exploring

- [:material-sitemap: Architecture Overview](./architecture-overview.md){ .md-button .md-button--primary }
- [:material-engine-outline: The Pipelex Kernel](./pipelex-kernel.md){ .md-button }
- [:material-source-branch: Build-time Elaboration](./build-time-elaboration.md){ .md-button }
- [:material-graph: Execution Graph Tracing](./execution-graph-tracing.md){ .md-button }
- [:material-image-multiple: Image Handling in LLM Prompts](./image-handling-in-llm-prompts.md){ .md-button }
- [:material-brain: Reasoning Controls](./reasoning-controls.md){ .md-button }
- [:material-code-braces: StuffArtefact & Image Rendering](./stuffartefact-and-image-rendering.md){ .md-button }
- [:material-test-tube: Test Profile Configuration](./test-profile-configuration.md){ .md-button }
- [:material-flask-outline: Dry Run Mock Generation](./dry-run-mock-generation.md){ .md-button }
- [:material-console: Init CLI Flows](./init-cli-flows.md){ .md-button }
- [:material-pipe: Pipe Routing & Execution](./pipe-routing-and-execution.md){ .md-button }
- [:material-alert-circle-outline: Error Model](./error-model.md){ .md-button }
- [:material-transit-connection-variant: Runtime Bridge & Transport](./runtime-bridge-and-transport.md){ .md-button }
- [:material-swap-horizontal: Content Generation Across Boundaries](./distributed-content-generation.md){ .md-button }



## Architecture Overview

# Architecture Overview

Pipelex is a Python framework for building and running **executable AI methods** using a declarative language (`.mthds` files).

---

## Two-Layer Architecture

Pipelex separates concerns into two distinct layers:

1. **High-Level**: Business logic and orchestration
2. **Low-Level**: Cognitive tools (COGT) and AI inference

This separation allows pipeline authors to focus on *what* should happen, while the framework handles *how* to interact with AI providers.

---

## High-Level: Business Logic & Orchestration

### PipeControllers (Orchestrators)

Located in [`pipelex/pipe_controllers/`](https://github.com/Pipelex/pipelex/tree/main/pipelex/pipe_controllers)

Controllers manage execution flow without performing work themselves:

- **PipeSequence** - Execute pipes one after another
- **PipeParallel** - Execute pipes concurrently
- **PipeBatch** - Process collections of items
- **PipeCondition** - Branch based on conditions

### PipeOperators (Workers)

Located in [`pipelex/pipe_operators/`](https://github.com/Pipelex/pipelex/tree/main/pipelex/pipe_operators)

Operators perform concrete actions:

- **PipeLLM** - Generate text or structured data via LLMs
- **PipeExtract** - Extract content from documents (OCR, parsing)
- **PipeImgGen** - Generate images
- **PipeFunc** - Execute custom Python functions
- **PipeCompose** - Compose content from templates or construct structured objects
- **PipeSearch** - Search the web for information

What an operator *does* — model-deck resolution, prompt assembly, generation, memory write-back — lives one layer down in [`pipelex/kernel/`](https://github.com/Pipelex/pipelex/tree/main/pipelex/kernel), as plain functions that take everything method-specific as an explicit argument and read only the kernel-layer services a boot stands up, such as the model deck and the content generator. The operator classes above hold what is specific to being a step in a method: blueprint resolution, input validation, error context, and execution-graph tracing. The split exists so the same semantics can be called without a loaded method, and so there is one implementation rather than two that drift. See [The Pipelex Kernel](./pipelex-kernel.md) for the layering contract, what a programmatic caller imports, and how it boots.

### Core Domain

Located in [`pipelex/core/`](https://github.com/Pipelex/pipelex/tree/main/pipelex/core)

- **Concepts** - Semantic types with meaning (not just data types)
- **Stuffs** - Knowledge objects combining a concept type with content
- **Working Memory** - Runtime storage for data flowing through pipes
- **Bundles** - Complete pipeline definitions loaded from `.mthds` files

---

## Low-Level: Cognitive Tools (COGT) & Inference

### COGT Layer

Located in [`pipelex/cogt/`](https://github.com/Pipelex/pipelex/tree/main/pipelex/cogt)

The COGT layer abstracts AI provider details from business logic:

- **LLM Workers** - Prompt construction, structured output, templating
- **Extract Workers** - Document processing and OCR
- **Image Generation Workers** - Image creation
- **Search Workers** - Web search and information retrieval
- **Model Catalog & Model Deck** - Manages available models, aliases, and presets
- **Content Generation** - Unified generation interface

!!! note "Why COGT?"
    COGT stands for "Cognitive Tools". This layer lets you swap AI providers without touching your pipeline definitions.

### Plugin System

Two packages, one for the mechanism and one for the built-in adapters:

- [`pipelex/plugins/`](https://github.com/Pipelex/pipelex/tree/main/pipelex/plugins) — the plugin **mechanism**: the `PipelexPlugin` contract, the registrar every plugin registers into, and the capability registries. This is what an out-of-tree plugin imports, and what the `pipelex.plugins.kernel` / `pipelex.plugins.interpreter` entry points resolve against.
- [`pipelex/providers/`](https://github.com/Pipelex/pipelex/tree/main/pipelex/providers) — the built-in **provider adapters**, one directory per vendor, each handling that vendor's API specifics:

    - OpenAI
    - Anthropic
    - Google (Gemini)
    - Mistral
    - AWS Bedrock
    - And more...

The dependency runs one way: adapters depend on the mechanism, never the reverse. Each adapter translates Pipelex's unified interface into provider-specific API calls.

---

## What Keeps The Layers Apart: The Two Hubs

The two layers above would be a diagram rather than an architecture if nothing enforced the split. What enforces it is the **hub** — the mechanism every component uses to reach a shared dependency (the config, a model deck, the pipe library) without importing the module that owns it.

There are two hubs, and the boundary between them *is* the boundary between the layers:

- [**`pipelex/runtime_hub.py`**](https://github.com/Pipelex/pipelex/tree/main/pipelex/runtime_hub.py) — process-scoped infrastructure. Config, console, secrets, storage, telemetry, the model deck, the inference workers, the content generator, the plugin registries. Configured once at boot; identical for every method the process runs.
- [**`pipelex/interpreter_hub.py`**](https://github.com/Pipelex/pipelex/tree/main/pipelex/interpreter_hub.py) — library-scoped method machinery. The library manager and the concept/domain/pipe libraries, the current-library binding, the pipe router, the pipeline manager, the PipeFunc executor. Tied to the method that is loaded.

One rule governs them:

!!! note "The one arrow"
    `interpreter_hub` imports `runtime_hub`. **`runtime_hub` never imports `interpreter_hub`.**

The practical consequence is that the kernel layer cannot reach the interpreter layer. Importing the inference stack loads no `libraries`, `pipe_operators`, `pipe_controllers`, or `codegen` module at all — so anything that just wants a secret, the console, or the model deck (a health check, `pipelex --version`, a plugin's registration module) does not pay for the method interpreter, and a change to a pipe blueprint structurally cannot perturb the import graph of `cogt`.

That is not a convention held up by review: `make check-hub-layering` fails the build if a kernel-layer module imports — or merely names in a string — the interpreter hub, and an import-closure test pins the property itself in a subprocess.

`pipelex/core/` sits on both sides of the line, deliberately. Its data model — concepts, domains, stuffs, working memory, the input/output specs — belongs to the kernel layer: it describes what a method's values *are*, needs no loaded method, and takes the concept or pipe it needs as an injected argument. Everything in `core/` that names a **`Pipe`** belongs to the interpreter layer, because a pipe is the interpreter's own object.

Contributors: the full specification — what lives on each hub, how to place a new symbol, and how the boundary is enforced — is in [Hub Layering](../contribute/hub-layering.md).

---

## How It All Fits Together

```mermaid
flowchart TB
    subgraph MTHDS[".mthds Pipeline Files"]
        direction LR
        D1["Declarative method definitions"]
    end

    subgraph HL["HIGH-LEVEL: Business Logic"]
        direction TB
        subgraph Controllers["PipeControllers"]
            C1["Sequence"]
            C2["Parallel"]
            C3["Batch"]
            C4["Condition"]
        end
        subgraph Operators["PipeOperators"]
            O1["PipeLLM"]
            O2["PipeExtract"]
            O3["PipeFunc"]
            O4["PipeImgGen"]
            O5["PipeCompose"]
            O6["PipeSearch"]
        end
        subgraph Core["Core Domain"]
            CR1["Concepts"]
            CR2["Stuffs"]
            CR3["Working Memory"]
            CR4["Bundles"]
        end
        Controllers --> Operators
        Operators --> Core
    end

    subgraph LL["LOW-LEVEL: COGT & Inference"]
        direction TB
        subgraph COGT["COGT Layer"]
            CG1["LLM Workers"]
            CG2["Extract Workers"]
            CG3["Search Workers"]
            CG4["Model Catalog"]
            CG5["Content Gen"]
        end
        subgraph Plugins["Plugins"]
            P1["OpenAI"]
            P2["Anthropic"]
            P3["Google"]
            P4["Mistral"]
            P5["Bedrock"]
        end
        COGT --> Plugins
    end

    subgraph API["AI Provider APIs"]
        A1["External Services"]
    end

    MTHDS --> HL
    HL --> LL
    LL --> API
```

---

## Next Steps

- [:material-cog: Explore Configuration Internals](../contribute/configuration-defaults-and-overrides.md){ .md-button }


## The Pipelex Kernel

# The Pipelex Kernel

This page is for contributors working on Pipelex internals, and for anyone embedding the kernel directly rather than running `.mthds` methods. For how the operator classes above it fit into the whole, see [Architecture Overview](./architecture-overview.md).

What a `PipeLLM` step actually *does* — resolve a model off the deck, derive a prompting style, assemble the prompt, generate, write the result into memory — used to be reachable only through a fully booted interpreter with a method loaded. [`pipelex/kernel/`](https://github.com/Pipelex/pipelex/tree/main/pipelex/kernel) holds that semantics as plain functions, so it has **one implementation with two kinds of caller**:

- the interpreter's operator classes (`PipeLLM`, `PipeExtract`, `PipeImgGen`, `PipeSearch`, `PipeCompose`, `PipeFunc`), which resolve blueprints, validate inputs, wrap errors and trace, then call the kernel;
- any **programmatic caller** embedding the kernel, which calls the same functions on a process with zero `.mthds` loaded.

Single-sourcing is the whole point. Two callers with two copies of "what an LLM step means" drift, and nothing tells you when they have.

---

## The boot contract

Every kernel call must be servable on `RuntimeBoot.make()` ([`pipelex/runtime_boot.py`](https://github.com/Pipelex/pipelex/blob/main/pipelex/runtime_boot.py)) — the **kernel-only** composition root, with no interpreter constructed and no library loaded.

```python
from pipelex.runtime_boot import RuntimeBoot
from pipelex.kernel.pipelex_kernel import PipelexKernel

RuntimeBoot.make()
kernel = PipelexKernel.make(user_id="my-service")
```

That boot stands up the model deck, the content generator, the class registry, the reporting delegate and the plugin registries — the machinery inference needs. It does **not** stand up the library manager, the pipe router or the pipeline manager, because a kernel caller has no method to load.

`needs_inference=False` boots keyless (no credentials, no model-deck validation) and sets the forced-DRY flag: every run the process initiates is coerced to `run_mode=DRY`, so the leaves mock instead of calling a provider. `PipelexKernel.make` applies that rule through the same `runtime_hub.resolve_run_mode_for_boot` the pipe tier uses — a second copy of the rule at a second factory is how the two would drift apart.

---

## Layering: what the kernel may and may not touch

| May | May not |
|---|---|
| `pipelex.runtime_hub` — the model deck, the content generator, the reporting delegate | `pipelex.interpreter_hub`, directly or transitively |
| `pipelex.core`, `pipelex.cogt`, `pipelex.tools`, `pipelex.tracing` | `pipelex.libraries`, `pipelex.pipe_operators`, `pipelex.pipe_controllers`, `pipelex.pipe_run`, `pipelex.pipeline`, `pipelex.mthds_parsing`, … |
| Definition-site imports | `pipelex.exceptions` or any other cross-layer re-export aggregate |
| Module-top-level imports | Function-local imports (invisible to the static graph *and* to the import-closure test at once) |

The **caller-facing** API is stricter still: hub-free. Everything method-specific arrives as an explicit argument — the concept, the concrete output class, the resolved setting, the working memory — and never through an ambient lookup. Concept compatibility, when a kernel path needs it at all, goes through the pure tiers (`Concept.are_compatible_by_declaration`, `are_structure_classes_compatible`), never through `ConceptLibrary.is_compatible`.

Four gates hold this, and each covers something the others miss — see [Hub Layering](../contribute/hub-layering.md) for the full picture:

| Gate | What it proves |
|---|---|
| `pipelex-dev check-hub-layering` | No kernel *module* imports the interpreter hub |
| `tests/unit/pipelex/test_kernel_layer_import_closure.py` | A kernel entry point *imports* clean |
| `tests/unit/pipelex/test_kernel_layer_exceptions_aggregate_gate.py` | No kernel module reaches the exceptions aggregate — imports and bare strings alike |
| `tests/unit/pipelex/kernel/test_kernel_boot_contract.py` | Every kernel entry point **runs** on a keyless boot, swept afterwards — except the three `resolve_*_setting` helpers, which read the model deck (a separate question from this one) |

Only the last one can see a function-local interpreter import, and it is **per-function**: it catches one inside `run_search` only by calling `run_search`. Every new kernel entry point owes it an arm.

---

## What a programmatic caller imports

Module-level functions carry the semantics. `PipelexKernel` is a thin façade over the LLM pair, holding the per-run state a caller would otherwise thread through every call; every other operator is called directly.

Both façade calls take the concept and the output class the caller wants, defaulting to `Text` and `TextContent` when it wants neither. `llm_text` accepts them because a text step is not always a *native*-`Text` step: a method may declare its output as a concept refining `Text`, and a façade that hardcoded the native one would write a different concept into memory than the interpreter writes from the same authored declaration.

| Module | Entry points |
|---|---|
| `pipelex.kernel.pipelex_kernel` | `PipelexKernel.make`, `.llm_text`, `.llm_object`, `.make_step_metadata` |
| `pipelex.kernel.llm_ops` | `resolve_llm_setting_for_text` / `_for_object`, `concrete_llm_model_handle`, `derive_templating_style`, `derive_structure_prompt`, `generate_object_content`, `run_llm_text`, `run_llm_object` |
| `pipelex.kernel.extract_ops` | `resolve_extract_setting`, `build_extract_job_params`, `run_extract` |
| `pipelex.kernel.img_gen_ops` | `resolve_img_gen_setting`, `resolve_default_size`, `build_img_gen_job_params`, `run_img_gen` |
| `pipelex.kernel.search_ops` | `resolve_search_setting`, `run_search` |
| `pipelex.kernel.compose_ops` | `build_compose_context`, `build_composed_content`, `run_compose_template` |
| `pipelex.kernel.func_ops` | `call_registered_function`, `run_func` |
| `pipelex.kernel.memory_ops` | `shape_inputs`, `store_result`, `extract_main_content` / `extract_named_content`, `extract_main_content_as_list` / `extract_named_content_as_list` |
| `pipelex.kernel.llm_prompt_content` | `LlmPromptContent`, `assemble_llm_prompt` |
| `pipelex.kernel.img_gen_prompt` | `assemble_img_gen_prompt` |
| `pipelex.kernel.prompt_references` | `ImageReference` / `ImageReferenceKind`, `DocumentReference` / `DocumentReferenceKind` |
| `pipelex.kernel.*_results` | The typed result envelopes |

The two `assemble_*` functions are there because `run_llm_text` and `run_img_gen` both take a *ready* prompt. A caller that could not build one would be holding an operator it cannot reach, which is what image generation was until `assemble_img_gen_prompt` existed: its only builder was an interpreter-layer blueprint. What they own is the part a caller must not re-derive — resolving `ImageReference` and `DocumentReference` out of working memory, and, on the image side, keeping the `[Image N]` tokens numbered from the same registry that orders `input_images`, since a mismatch mislabels which image the prompt is describing and nothing downstream can detect it.

There are **no re-exports**: `pipelex/kernel/__init__.py` holds doctrine and nothing else, and every symbol is imported from the module that defines it. For this package that is a layering property rather than a style one — a module that re-exports across layers is a layer boundary with the sign filed off.

Every kernel function is **fully keyword-only**, with zero entries in `subject_grants.toml`. Call sites name every argument.

---

## The memory boundary

`WorkingMemory` is threaded explicitly: a call takes it and returns it. The contract, which both kinds of caller must read the same way:

!!! warning "Treat the returned memory as the result"
    A kernel call may mutate the memory it was passed **and** returns it. Callers must use the returned one and must not rely on the two being the same object — inline execution aliases them today, and a serialization boundary will not.

`pipelex.kernel.memory_ops` holds the three ends of that boundary — shape in, write back, read out:

- **`shape_inputs`** — interpret raw values against the specs declared for them (Smart Inputs: a bare string becomes the declared concept, a dict validates against a structured one, a list shapes element-wise). It takes a `ConceptProviderAbstract` explicitly, because resolving concepts is what a loaded method's library is for and the kernel must stay callable without one. The interpreter hands over its concept library; a library-free caller supplies its own provider (the boot-contract test shows the smallest one that works — native concepts from `ConceptFactory`, compatibility from the declaration tier, structure classes from the class registry a boot fills).
- **`store_result`** — the write-back every operator's ops end with, and the one place the memory contract is implemented.
- **`extract_main_content` / `extract_named_content`** — the typed read. Needed even though every result envelope already carries the produced content, because those fields are annotated with the base `StuffContent`: pass the class you asked for and get it back narrowed.
- **`extract_main_content_as_list` / `extract_named_content_as_list`** — the same typed read for a call that produced several objects. A multiple-output call stores one `ListContent`, which the single-content reads cannot narrow: the bare item class raises, and `ListContent[item_type]` is rejected by design. These verify every item against `item_type`, so the list comes back typed all the way down.

```python
memory = shape_inputs(inputs={"topic": "kernels"}, concept_provider=provider, input_specs=specs)
result = await kernel.llm_object(memory=memory, output_class=Summary, concept=summary_concept, model=model, user="Summarize $topic", result="summary")
summary = extract_main_content(memory=result.memory, content_type=Summary)
```

Reach for the list pair whenever the call asked for several — `is_multiple_output=True` or `fixed_nb_output=n`:

```python
result = await kernel.llm_object(memory=memory, output_class=Summary, concept=summary_concept, model=model, user="Summarize $topic", result="summaries", is_multiple_output=True)
summaries = extract_main_content_as_list(memory=result.memory, item_type=Summary).items
```

---

## Run-scoped state, and who owns the usage lifecycle

`PipelexKernel` holds run-scoped identity, plus the one seam that identity needs:

- **`job_metadata`** — the run-level metadata. It is not what a step runs under: every call mints a per-step copy through `make_step_metadata()`, carrying a fresh `pipe_run_id` and inheriting the trace context, so trace and usage attribution stay per-step. This mirrors the interpreter's pass-down-a-modified-copy pattern. `make_step_metadata(pipe_code=…)` names the pipe the step is running — mirroring what the interpreter stamps on its live and dry paths — so log correlation, usage accounting, and the per-step labelling a distributed backend derives see a named step. When no `pipe_code` is supplied the key is omitted from the update rather than passed as `None`, so a run-level `pipe_code` is never silently erased; the direct-call façade (`llm_text`/`llm_object`) stays deliberately anonymous, because the caller ran no pipe.
- **`cogt_run_params`** — the execution-mode contract (`run_mode`, and the DRY-only `is_mock_usage` sub-flag) that every cogt leaf reads off the assignment it is handed.
- **`step_id_source`** — where each step's `pipe_run_id` comes from, defaulting to a fresh `uuid4`, which is what every in-process run wants. It exists for a kernel hosted inside a replay-based executor, where ids minted from a non-replay-safe source take different values each re-execution; such a host injects its own replay-safe source via `PipelexKernel.make(step_id_source=…)`. It is a seam, not state — the kernel never inspects what it returns.

Nothing derived from config or the model deck is cached on the instance — resolved settings and prompting styles are computed per call, because cached derived state would shadow a later config or deck change and break per-call variation.

**Cost and usage reporting is the caller's lifecycle, not the kernel's.** The interpreter's run machinery opens a graph tracer, builds an event log, registers it on the report delegate and closes all three in a `finally`, because it has a run boundary to hang that on. A kernel call has no such boundary — it is one step, and a caller may make one or a thousand. So the kernel takes a `TraceContext` and does exactly one thing with it: stamp it onto every step's `JobMetadata`, which is what the cogt leaf reads to decide whether to emit a usage event.

Everything else is yours:

```python
from pipelex.runtime_hub import get_report_delegate
from pipelex.system.trace_context import TraceContext
from pipelex.tracing.in_memory_event_log import InMemoryEventLog
from pipelex.tracing.usage_aggregator import UsageAggregator

event_log = InMemoryEventLog()
trace_context = TraceContext(graph_id=run_id, data_inclusion=data_inclusion, emit_graph_events=False, emit_usage_events=True)
get_report_delegate().set_event_log(context_key=trace_context.lookup_key, event_log=event_log, workflow_id="direct", pipeline_run_id=run_id)
try:
    kernel = PipelexKernel.make(user_id="my-service", trace_context=trace_context)
    ...
    tokens_usages = UsageAggregator.aggregate(event_log.read_events(run_id))
finally:
    get_report_delegate().clear_event_log(context_key=trace_context.lookup_key)
```

Passing a `trace_context` adopts its `graph_id` as the run's `pipeline_run_id`. The two are one identity: letting them diverge would scatter a single run's usage events across two ids, because the registered-context emit path stamps the event log's id while the runner fallback stamps the metadata's — and a read-back keyed on either would silently miss the other's.

`pipelex.tracing` holds both halves a caller needs (`make_event_log` for a configured backend, `UsageAggregator` for the read-back) and is kernel-layer, so none of this costs the boot contract. The records that come out are the same [`TokensUsage` wire records](./tokens-usage-wire-records.md) an `/execute` response carries — pinned by an integration test that runs the same step through both callers and compares them.

---

## What the kernel deliberately does not cover

Two arms of the interpreter stayed interpreter-side, both because moving them would cost more than the caller gains:

- **`PipeCompose`'s construct mode.** Its semantics are `StructuredContentComposer` over a `ConstructBlueprint`, and a blueprint is an MTHDS language artifact with a language-side consumer. A programmatic caller holds real Python and builds its structured object directly rather than describing the construction declaratively. The template path *is* fully extracted, and the one thing both paths share — the three-layer context ordering — is single-sourced in `build_compose_context`.
- **`PipeFunc`'s pluggable executor seam.** The protocol and its DTOs are typed on interpreter models (`PipeRunParams`, `LibraryCrate`), so the kernel cannot name them. What running a function *means* — registry lookup, async-vs-sync dispatch, content coercion — is single-sourced in `call_registered_function`, which both the in-process executor and the kernel's `run_func` ride. What stays outside is *where* the function runs, which is configured deployment machinery rather than operator semantics. So a kernel `run_func` always runs in this process.

Beyond those: controllers are out of scope entirely (`pipelex/pipe_controllers/` is the interpreter's), the kernel is not separately installable from PyPI, and "activity-shaped" is a design constraint on the call signatures rather than a distributed-execution deliverable.


## Build-time Elaboration

# Build-time Elaboration

Some MTHDS directives are **shorthands**: they describe what the user wants in one line, but the executable form needs several pipes wired together. Rather than handle them in the runtime, Pipelex rewrites them at load time, before any pipe runs. The runtime sees only normal pipes — `PipeLLM`, [`PipeStructure`](../building-methods/pipes/pipe-operators/PipeStructure.md), [`PipeSequence`](../building-methods/pipes/pipe-controllers/PipeSequence.md), and so on.

Today, exactly one directive is elaborated: `structuring_method = "preliminary_text"` on a [`PipeLLM`](../building-methods/pipes/pipe-operators/PipeLLM.md). This page documents the mechanism so contributors and AI agents can reason about it.

## Where it runs

`BundleElaborator.elaborate(bundle)` runs inside `MthdsParser.make_pipelex_bundle_blueprint()`, right after the bundle dict has been validated into a `PipelexBundleBlueprint`. Every code path that loads a bundle from a `.mthds` file goes through it.

The elaborator has a fast-path short-circuit: if no pipe in the bundle carries `structuring_method = "preliminary_text"`, the input bundle is returned unchanged (identity-preserving). Loading a bundle without preliminary-text pipes pays no overhead.

## What `preliminary_text` becomes

Given a user-authored pipe like:

```toml
[pipe.review_restaurant]
type = "PipeLLM"
description = "Write a structured review of a restaurant from a transcript"
inputs = { transcript = "Text" }
output = "RestaurantReview"
structuring_method = "preliminary_text"
prompt = """
Write a thorough review of this restaurant based on the transcript:

@transcript
"""
```

the elaborator replaces the single entry `review_restaurant` with three blueprints:

- **`review_restaurant__draft_text`** — a `PipeLLMBlueprint` that inherits the original `inputs`, `system_prompt`, `prompt`, and `model`. Its output is **always literal `Text`**, regardless of the original output's multiplicity. `structuring_method` is reset to `None` so the elaborator never recurses.
- **`review_restaurant__structure`** — a `PipeStructureBlueprint` with `inputs = { draft_text = "Text" }`, `output = original.output` (preserving multiplicity — `Foo`, `Foo[]`, `Foo[N]`), and `model = original.model_to_structure`. When `model_to_structure` is `None`, `PipeStructure` falls back to the deck's `for_object` LLM choice at runtime — same behavior as the legacy text-then-object path.
- **`review_restaurant`** (replaced) — a `PipeSequenceBlueprint` wrapping the two synthetic steps, keeping the original `inputs`, `output`, and `description`. The original `pipe_code` is preserved so anything that calls or references the pipe (other pipes, `main_pipe`, the run API) keeps working unchanged.

The two synthetic codes are recorded in `bundle.elaboration_metadata`, a side-table on `PipelexBundleBlueprint` that maps each synthetic code to its `parent_pipe_code` and a `step_role` (`DRAFT_TEXT` or `STRUCTURE`). The wrapping sequence is **not** registered — it is the user-facing pipe.

## Why a side-table

`elaboration_metadata` is declared with `Field(exclude=True)`, so it never serializes back to `.mthds`, TOML, or JSON. The runtime `PipeAbstract` has no `parent_pipe_code` field; the user-facing per-pipe blueprint surface is unpolluted by the elaboration concern.

The side-table is the durable source of truth that downstream tools (graph viewer, CLI listings, distributed traces) can consult to render synthetic pipes specially. Today they don't — synthetic pipes appear as regular pipes in logs and traces — but the metadata is in place so that opt-in can land later without another round of plumbing.

### Lifetime

`elaboration_metadata` is **process-local**. It survives `model_copy` (used by the elaborator itself), but any `model_dump` → `model_validate` round-trip drops the side-table — that is what `exclude=True` buys us. The elaborator's own re-validate pass (`model_validate(elaborated.model_dump(by_alias=True))`) discards the rehydrated bundle and returns the original `model_copy`-built one with metadata intact, precisely because of this.

Practical consequences:

- Within a single process, after `BundleElaborator.elaborate(...)`, the metadata is available everywhere the bundle goes.
- Across any serialization boundary — kajson dump, library cache, Temporal payload, MTHDS export — the metadata is gone. Downstream consumers that need it today have to re-elaborate.
- The dependency loader handles one specific consequence: when a manifest restricts exports, synthetic helpers of exported parents are still loaded, even though they are never named in the manifest. This is implemented inline in `LibraryManager._load_single_dependency`.

When a future consumer (graph viewer, persistent observability store) wants the metadata across boundaries, dropping `exclude=True` is the deliberate next step.

## Multiplicity rule

Step 1 always produces a single `Text`, even when the original output was `Foo[]` or `Foo[3]`. Step 2 is the one that fans out: `PipeStructure` inspects its declared `output` and either calls `make_object` (single) or `make_object_list` (list, with optional fixed `nb_items`). This matches the deleted `make_text_then_object_list` behavior verbatim — one preliminary text, structured into N objects.

## Image inputs

The original `inputs` dict (including any image variables) flows through to step 1. Step 2's input dict is hard-wired to `{ "draft_text": "Text" }`. The structuring template references only `{{ text }}`. Image variables therefore appear only on step 1, where they belong — the elaborator does not need explicit drop logic.

## Pre-checks and validation

Three layers guard against authoring mistakes around the output concept:

1. **Construction-time (string)** — `PipeLLMBlueprint.validate_preliminary_text_output` (a `model_validator(mode="after")`) rejects a literal Text output (`"Text"`, `"native.Text"`, `"Text[]"`, `"Text[N]"`) combined with `structuring_method = "preliminary_text"`. This fires during `model_validate`, before the elaborator runs, so the user gets the error at parse time with a normal Pydantic validation failure.
2. **Defense-in-depth (string)** — `BundleElaborator._elaborate_preliminary_text` re-runs the same string check. Only reachable if a caller bypasses validation via `model_construct`; the test suite exercises it, but nothing in the framework relies on the bypass.
3. **Library-time (concept)** — `PipeStructure.validate_output_with_library` runs once the elaborated `PipeStructure` resolves its concept against the loaded library. This is the only layer that catches a *domain concept* that `refines = "Text"` — those slip past the string-level guards because they don't read as `Text` in the bundle source. A clean separation: string-level guards run at parse time; concept-level guards run at library load.

The elaborator additionally:

- Verifies that the synthetic codes (`<pipe_code>__draft_text`, `<pipe_code>__structure`) don't already exist in the bundle.
- Verifies that the synthetic codes pass `is_pipe_code_valid` (snake_case).
- After producing the new bundle, re-runs `PipelexBundleBlueprint.model_validate(elaborated.model_dump(...))` so bundle-level validators (concept refs, pipe refs, `main_pipe`) re-check the synthetic pipes. Any `ValidationError` here is wrapped in `PipelexUnexpectedError`, with a message naming the bundle's `domain` and the sorted list of synthetic pipe codes (`BundleElaboratorError` is used only for the earlier synthesis-time checks during the same `elaborate()` call).
- Asserts that no synthesized blueprint itself carries `structuring_method = "preliminary_text"` — a recursive-elaboration guard. Today's synthesis explicitly sets `None`; the guard protects against future elaboration kinds copying fields wholesale.

## Cost and runtime semantics

A `preliminary_text` pipe issues **two LLM calls per invocation** (one for the draft text, one for the structuring step). The reporting layer counts both — the integration tests assert exactly two `LLMTokensUsage` records per run.

Runtime, including the Temporal data path, has no knowledge of `structuring_method`. Temporal sees only the elaborated form (`PipeSequence` + `PipeLLM` + `PipeStructure`); the existing `PipeOperator` machinery covers it.

## When to add a new elaboration kind

`BundleElaborator` is intentionally narrow today. Adding a second kind means:

1. A new directive on a blueprint (e.g. another field on `PipeLLMBlueprint` or a new field on a different blueprint).
2. A construction-time validator that rejects nonsensical combinations of the directive with other fields.
3. A new private method on `BundleElaborator` (`_elaborate_<kind>`) that synthesizes the replacement pipes and registers metadata.
4. Extending the `_is_<kind>_pipe` `TypeGuard` set + dispatch in `elaborate`.

When the second kind lands, the dispatch logic should be promoted to a small registry keyed by directive — but not before. One concrete consumer is enough; two is a pattern.

## Related

- [`PipeLLM` › Structuring Method (preliminary_text)](../building-methods/pipes/pipe-operators/PipeLLM.md#structuring-method-preliminary-text) — the user-facing surface.
- [`PipeStructure`](../building-methods/pipes/pipe-operators/PipeStructure.md) — the operator that backs step 2.
- [`PipeSequence`](../building-methods/pipes/pipe-controllers/PipeSequence.md) — the controller that wraps the two synthetic steps.


## Codegen Projections

# Codegen Projections

Codegen turns one **normalized library crate** — the flat, fully-qualified, self-contained, fingerprinted snapshot of a resolved library (see [`pipelex resolve`](../tools/cli/build/structures.md) and the [Library Crate Format](https://mthds.ai)) — into typed, documented artifacts for each consumer. The crate is the single authority: codegen never re-derives meaning from raw bundles and never calls a model.

This page describes the projection engine in `pipelex/codegen/` and the contract it implements: deterministic projections off the crate, stamped generated files, a sibling `codegen.lock`, and an offline drift check whose verdict rides the exit code.

## Two resolved layers, then emitters

The engine reads the crate through two neutral, language-agnostic layers, so every emitter consumes the same resolved shape rather than re-deriving the semantic mapping:

- **Resolved fields** (`pipelex/core/concepts/resolved_fields.py`) — one structure field becomes a `ResolvedType` tree (`text`/`number`/`concept`/`list`/`dict`/`literal`/…). Inline `choices` become a literal; a bare concept ref is promoted to its domain; natives are flagged. Where the source under-specifies a shape — a `list` with no `item_type`, a `concept` field with no ref — the resolved type is `ANY` carrying an explicit **imprecision marker** (never a guess).
- **Resolved concepts** (`pipelex/codegen/resolved_concepts.py`) — one crate concept becomes a `ResolvedConcept`: its type name inputs (domain, code, collision flag), its description, its refinement base, and either its resolved fields, or a structureless/opaque marker. Class naming is decided here once: a concept code that is unique across the crate stays bare (`Report`); a code that collides across domains is domain-qualified so a definition and every reference agree.

An emitter then walks the `ResolvedLibrary` and renders text. Emitters live in `pipelex/codegen/emitters/`, dispatched by `emit_types(crate, target=...)`.

## Targets

The `types` projection ranges over the crate's concept set (one type per qualified concept). Three targets ship today:

- **`python-structures`** — Pipelex runtime `StuffContent` subclasses (the runtime idiom, the successor to `pipelex build structures`). Native references map to the runtime content classes (`TextContent`, …); native concepts themselves are not re-emitted. Each concept lands on the same content class the runtime would resolve it to: a structured concept on `StructuredContent`, a concept refining a native on that native's content class, a structureless one on `TextContent`.
- **`python-pydantic`** — plain `pydantic.BaseModel` types with no Pipelex imports. Every concept is emitted uniformly, including the materialized natives, so the module depends only on `pydantic` and the standard library.
- **`ts-zod`** — a pure TypeScript + Zod types file (`types.ts`, imports only `zod`) plus a `binder.ts` companion. `types.ts` holds Zod schemas and their inferred types; type names are the concept codes; **field keys are the crate's snake_case wire names verbatim**. Concept references use `z.lazy(() => XSchema)` so declaration order is irrelevant and cycles are handled.

### The ts-zod types file and its binder

`types.ts` stays dependency-free and portable (only `zod`). Its object keys are **wire-native** (snake_case), so a schema validates a wire payload *directly*. `binder.ts` is the thin companion exposing one typed `parse<Name>` / `serialize<Name>` pair per concept — each a direct `Schema.parse`, with no key-remapping layer. A pipe's IO types are concepts, so a pipe's output parser / input serializer is just the binder pair for those concept types — the binder is the concept-set-wide realization of the per-pipe parse/serialize helpers.

Wire-native keys (D10) are a deliberate correctness choice: a camelCase-keyed schema would need a binder that remaps keys, and a *generic* deep remap cannot tell a schema-declared field key from arbitrary data inside a `z.record()` / `z.unknown()` value (e.g. `native.JSON`'s `json_obj` map) — so it would silently rename the caller's actual data. Keeping keys wire-native removes that hazard entirely; if camelCase ergonomics are wanted later, they must ride a *schema-aware* transform, not a blind key remap.

For **`python-pydantic`**, no key mapping is ever needed either: wire names are already snake_case Python names, so parse/serialize are the native `Model.model_validate(data)` / `model.model_dump(mode="json")`.

### Refinement and native bases

A concept that refines another keeps its `refines` link when the base is **native-backed** (the refinement chain bottoms out at a native such as `Text` or `Number`), because the native is materialized into the crate and the base carries real runtime behavior; the emitter then renders inheritance (`class Summary(TextContent)` / `class Summary(Text)` / `z.lazy(() => TextSchema)`), which round-trips to the correct base class. A concept that refines an **in-crate structured** base has that base's effective structure flattened in during normalization.

Native materialization itself is faithful-or-absent per native: a content-class field maps when it has an unambiguous blueprint form — primitives, dicts, lists, references to other natives, and nested non-native models, which serialize as JSON objects and therefore map to a `dict` blueprint with unspecified value types (declared imprecision the emitters surface, e.g. `native.JSON`'s `json_obj` map). A field with no honest blueprint form at all (for example, a non-optional union) leaves the whole native structureless rather than guessing a partial shape.

## The CLI surface

Two command families drive the engine:

- **`pipelex resolve [PATH]… [-f json|toml] [-L DIR]…`** — assembles the closure (working bundles + the local `.mthds/methods/` cache), requires it to be **valid**, and emits the normalized crate to stdout. The verdict rides the exit code, mirroring the bare `validate` group: `0` resolved, `1` the library is invalid (a negative verdict — no crate), `2` no verdict (empty closure / not found).
- **`pipelex codegen <kind> …`** — the two-axis projection family (`kind` × `--target`):
    - `pipelex codegen types --target ts-zod|python-pydantic|python-structures [-o DIR] [PATH]… [-L DIR]…` — projects the crate's concept set for a target and writes each emitted file under the output directory.
    - `pipelex codegen inputs [--pipe <ref>] [-f json|toml] [--explicit] [-o FILE] [PATH]… [-L DIR]…` — projects a runnable inputs template for one pipe (Smart Inputs light shape by default, `--explicit` for the envelope), selected by qualified `--pipe` and defaulting to the closure's declared `main_pipe`.

Both codegen commands load and normalize the crate through the same shared helper the resolver uses (`pipelex/cli/commands/crate_loading.py`), so they share the resolve/validate exit-code contract.

The **agent CLI mirrors** the family for machine consumers: `pipelex-agent codegen types --target <flavor>` runs the same engine and emission layer but presents through the agent CLI's two-stream envelopes (`--format markdown|json` on stdout for success, `--error-format` on stderr for errors — markdown default), and `pipelex-agent codegen check` is the same offline drift check with the verdict as a structured envelope: up-to-date is a success envelope, drift is a `CodegenDriftError` error envelope enumerating `drifts[]` by category (exit `1`), and a missing/unreadable lock is a no-verdict error (exit `2`). There is no `pipelex-agent codegen inputs` — the existing `pipelex-agent inputs` group already surfaces that projection. The legacy `build` family rides the same engine: `pipelex build inputs` renders through the same `input_renderer` engine as `codegen inputs` (they cannot diverge), and `pipelex build structures` is a thin alias of `codegen types --target python-structures` (the old always-qualified per-file generator is gone — the alias emits the single stamped `structures.py` + `codegen.lock`, bare-when-unique names). `pipelex build runner` emits its `structures/` scaffold through the same projection and spells the script's imports and example inputs with the emitted class names (`from structures.structures import …`), so the script and the module always agree.

## The trust chain: stamps, lock, and the offline check

Drift — generated code that no longer matches the method it was generated from — is the tax every codegen system eventually charges. The engine refuses to pay it. Every file `pipelex codegen types` writes carries a machine-parseable **stamp** header, the artifact set is recorded in a sibling **`codegen.lock`**, and `pipelex codegen check` verifies both — entirely offline.

### Stamps

`pipelex/codegen/stamp.py` prepends a fenced comment block (in the target's comment syntax — `#` for Python, `//` for TypeScript) to each generated file:

```python
# >>> pipelex-codegen-stamp >>>
# crate_fingerprint: 1336e999…293f46
# engine_version: 0.38.0
# projection: types / python-pydantic
# options: {}
# content_hash: 1d074a49…6f42e4
# <<< pipelex-codegen-stamp <<<
```

The `crate_fingerprint` is the crate's semantic hash, so reformatting or commenting a `.mthds` file never changes a stamp — only a change to the method's effective type surface does. The `content_hash` is a SHA-256 over the body **below** the stamp, so a lone file can testify that it has not been hand-edited, without the engine, the network, or the lock.

### Lock

`pipelex/codegen/lock.py` writes `codegen.lock` (human-diffable TOML) recording each generated artifact's path and body hash, plus the crate fingerprint and engine version the set was built against. The lock catches the one drift class per-file stamps cannot: a deleted concept whose stale generated file lingers on disk. It is a Pipelex-owned artifact, distinct from the standard's `methods.lock` (which pins remote dependency versions).

### Offline check

`pipelex codegen check [DIR]` (`pipelex/codegen/check.py`) is pure hashing — **no engine, no network, no API key** — so any client (this CLI, an SDK, a short CI script) implements it identically. It reports drift by category:

- **missing** — an artifact in the lock is absent on disk;
- **modified** — a file's body no longer matches its locked hash;
- **hand-edited** — a file's stamp is stripped or its recorded content hash no longer matches the body;
- **orphan** — a stamped generated file on disk that the lock does not track.

The verdict rides the exit code (mirroring `resolve` / `validate`): `0` current, `1` drift present, `2` no lock found. Regeneration against the engine is a **dev action**; the offline check is the **CI action** — so template improvements never redden a consumer's CI.

### Idempotent emission

`pipelex/codegen/emission.py` ties it together in two layers. `build_stamped_projection` is the **pure** core: it stamps each emitted body and assembles the matching `codegen.lock` content — no filesystem access — so it is the single source of truth for what a projection *is* on disk. `write_stamped_projection` rides it to materialize locally: it writes each file only when its content changed (write-if-changed — no mtime churn, clean diffs, watch-mode friendly), removes any previously-tracked stamped file that dropped out of the set, and rewrites the lock. Only files the tool itself stamped are ever removed, so a hand-authored file sharing the output directory is never touched. `codegen inputs` applies the same write-if-changed rule to its single (unstamped) template file, so a full regeneration pass over a committed consumer is a true no-op when everything is current.

### Lint-clean by construction

A stamp's `content_hash` is a raw SHA-256 over the body bytes. So if the emitted code is not already what your formatter wants, the first `ruff check --fix` / `ruff format` (or `prettier --write`) over your tree rewrites those bytes and `pipelex codegen check` reports the file as **hand-edited** — accusing you of the one thing you did not do.

The emitters therefore emit exactly what the formatter would write. Generated Python uses builtin generics (`list[str]`, `dict[str, int]`), `X | None` over `Optional[X]`, double-quoted `Literal` members, and isort-grouped imports — merged, sorted, and, past the width threshold below, already wrapped into the parenthesized magic-trailing-comma form ruff itself would produce; generated TypeScript matches Prettier's blank-line and import-wrapping behavior. Running your formatter over a generated tree is a no-op, and the stamp stays valid.

**You should not need to exclude generated paths from your linter.** If you carry such an exclusion from an older version, drop it.

Every import is registered by whoever writes the name it imports, never seeded up front. That matters for crate shapes which use fewer of them than you would expect: a method declaring only structureless concepts emits no `Field(...)` at all, and one whose concepts all refine a native never reaches the runtime root base. An import left unused is an `F401` — a fix ruff applies automatically, so the line would simply vanish and take the stamp with it.

A projection with nothing to emit for a target is its **header alone** — no imports, no trailing blank lines. That case is ordinary rather than degenerate: `python-structures` skips native concepts, since they already exist in the runtime, so a method that declares no concepts of its own leaves that target with no class to write. The header is inert under any formatter, whereas an import block with nothing left to use it is an unused-import finding sitting above a collapsible blank-line run — both of which a formatter would rewrite, breaking the stamp.

#### Line width, which is your setting and not ours

A description or choice list is authored text with no length bound, so a generated line can exceed whatever `line-length` you lint at — and your formatter would then wrap the call, rewriting the bytes.

Because we cannot know your width, anything too long is emitted **already wrapped, with a trailing comma**. That comma is load-bearing: Black and ruff both read it as a deliberate choice to keep the construct exploded, and will not rejoin it at any width. Short lines stay on one line, so ordinary artifacts read exactly as before.

The threshold is 88 columns — ruff's own default, chosen because it is the *tightest* width you are likely to use, and only the tightest threshold is safe. If you lint Python at **fewer than 88 columns**, short generated lines can still be rewrapped; either raise `line-length` for the generated path or keep an exclusion there.

`E501` is the one finding that can survive on a generated file, on a line holding a single long string literal — an authored description or choice value. There is no wrapped form: breaking a string would alter the author's text. It has no automatic fix, so it never changes bytes and never breaks a stamp, and it is not in ruff's default rule set, so most consumers never see it.

#### Docstrings

A concept description becomes the generated class's docstring, rendered the way a human writes one: summary on the first line, continuation lines indented to the class body, closing delimiter on its own line. That shape is not cosmetic — it is the only one that `ruff format` and the pydocstyle rules both leave alone. Rendered flat, a multi-line description trips `D207`/`D209` and edge whitespace trips `D210`, all of which ruff fixes automatically, rewriting the bytes.

So `__doc__` carries the class-body indentation, exactly as a hand-written docstring does. What holds exactly is the value you actually read:

```python
inspect.getdoc(GeneratedClass) == inspect.cleandoc(authored_description)
```

Exact bytes survive wherever they are consumed programmatically rather than read as prose — `Field(description=...)` keeps the description verbatim, and so does the crate.

#### TypeScript assumes Prettier's defaults

The `ts-zod` target is emitted to match Prettier's **default** configuration: 80-column print width, double quotes, semicolons. A long concept name or choice list is pre-wrapped the way Prettier would wrap it, for the same reason the Python side pre-explodes long calls.

If your Prettier config changes `printWidth`, `singleQuote`, or `semi`, the generated files will not match it, and `prettier --write` will rewrite them. Either run codegen output through your own Prettier before stamping, or keep an exclusion for the generated path.

One ruff setting is required, because it cannot be fixed in the emitted bytes:

```toml
[tool.ruff.lint.flake8-type-checking]
runtime-evaluated-base-classes = [
  "pydantic.BaseModel",
  "pipelex.core.stuffs.structured_content.StructuredContent",
]
```

Without it, `TC003` asks you to move `from datetime import date` into an `if TYPE_CHECKING:` block. Applying that **breaks** the generated models — pydantic resolves annotations at runtime to build validators, so the import must stay at runtime. The setting tells ruff these classes are runtime-evaluated, and the finding goes away.

The remaining findings are artifacts of *how* you lint, not of file content. None of them has an automatically-applied fix, so none can change bytes:

- `INP001` ("implicit namespace package") fires when you point ruff at a bare directory with no `__init__.py`. Add one, or lint the package that contains it.
- `E501` on a single long string literal, as described above.
- `D301` ("use `r\"\"\"`") on the rare description that carries both `\"\"\"` and `'''`, or a control character — the only inputs with no verbatim rendering, which therefore have to be escaped. Ruff classes that fix as *unsafe*, so `ruff check --fix` leaves it alone.
- Import grouping for the `python-structures` target assumes `pipelex` is a third-party dependency — which it is in your tree. The Pipelex repo itself, where `pipelex` is first-party, is the one place ruff wants the opposite order; no generated artifact is committed there, so the consumer's grouping is the one the emitter targets.

## Serving the engine over HTTP

The same engine backs the `pipelex-api` routes (`POST /v1/resolve`, `POST /v1/codegen`, and the re-pointed `/v1/build/*` — the route envelopes are documented in `pipelex-api`'s `docs/codegen.md`). Two host-facing cores make that possible without any CLI plumbing:

- **`pipelex.pipeline.resolve_bundle.resolve_crate_from_contents`** resolves **in-memory** MTHDS contents (strings, with optional per-content sources) into the normalized crate. It mirrors `validate_bundle`'s in-memory arm — the same `translate_to_validate_bundle_error` cascade, so an invalid library raises the one shared `ValidateBundleError` and a resolve verdict cannot drift from a validate verdict — and the same **loaded-on-success contract**: the library is left loaded and current for the host to read live pipes from, and the host owns its teardown. Resolution is static (no dry-run sweep), matching `pipelex resolve`.
- **`build_stamped_projection`** (above) gives the host the stamped artifact set plus the lock as pure content. A client that writes the served files and lock verbatim reproduces a local run byte-for-byte — the offline `codegen check` passes on the written tree exactly as it would locally. There is deliberately no server-side check route: the check is offline by design.

## Surfacing imprecision, never guessing

A deterministic tool that guesses is a liability. Where a resolved type carries an imprecision marker, the emitter surfaces it rather than inventing a shape:

- Python emitters append an inline `# imprecise: <reason>` comment on the field.
- The ts-zod emitter emits `z.unknown()` plus a JSDoc `@imprecise <reason>` tag.

Two concept-level cases are surfaced the same honest way:

- A **structureless** concept (no structure, no refinement) projects as an opaque type — an empty model for `python-pydantic`, `z.unknown()` for ts-zod — with the imprecision stated in its docstring. Its *base class* is not a guess, though: `python-structures` emits it on `TextContent`, because that is what the runtime resolves the same declaration to — "describe it in prose and get text back" is what a structureless concept means to an author, and the interpreter's text-vs-object dispatch reads the base class to decide which call to make. A projection that emitted the root base instead would answer that question differently from the runtime for the same authored concept, silently. That inherited `text` field is what makes this target's structureless class the one exception to the pass-through rule below.
- A **Python-class-backed** concept (`structure = "<ClassName>"`, whose shape lives only in hand-written Python, not in MTHDS) is surfaced as opaque — the bare class name is never silently emitted into a portable crate. This is the one opaque shape that keeps the root base: its content class genuinely is not visible to the crate, so promoting it to `TextContent` would be the guess the rule above forbids.

Opaque is **pass-through, never lossy**: the ts-zod `z.unknown()` hands the wire object through verbatim, and the Python emitters set `model_config = ConfigDict(extra="allow")` on opaque classes so `model_validate` keeps every field (pydantic's default `extra="ignore"` would silently strip the content). The owner of a Python-class-backed concept recovers the typed object by validating with their own class (`MyLegacyClass.model_validate(payload)`); every other consumer gets the honest untyped object.

The one exception is the shape described above: a **structureless** concept on `python-structures` inherits `TextContent.text`, which is required, so an object-shaped payload raises instead of landing in `extra`. That is not a regression against the runtime — the class `ConceptFactory` builds for the same declaration is a `TextContent` subclass too, so it rejects the same payload. Matching the runtime is the point; a projection that accepted a payload the runtime refuses would be the more expensive lie. Consumers who want the old permissive behaviour are describing a concept that is not structureless, and should give it a structure.

## The extension-file story

Generated code is never edited — hand edits are overwritten on regeneration, and the trust chain treats them as drift. Customization lives in **sibling extension files** that survive regeneration, and each generated file's header says so:

- **Python** — subclass the generated type from a sibling module:

    ```python
    # my_types_ext.py
    from .structures import Report

    class MyReport(Report):
        ...
    ```

- **TypeScript** — augment the generated type from a sibling module via declaration merging.

Every generated file carries an `AUTOGENERATED — DO NOT EDIT` header naming its projection and pointing at this mechanism.


## Dry Run Mock Generation

# Dry Run Mock Generation

Dry runs validate pipeline structure without executing inference. This requires generating mock `StuffContent` objects that satisfy Pydantic field constraints. The `DryRunFactory` system produces format-compliant mock values for fields with validation rules (e.g., snake_case identifiers, PascalCase concept codes).

---

## Why Mock Generation Matters

Pydantic models in Pipelex often enforce format constraints via `field_validator` or `model_validator`:

```python
class BundleHeaderSpec(StructuredContent):
    domain_code: str  # Must be snake_case
    main_pipe: str    # Must be snake_case, must exist in pipe dict

class ConceptSpec(StructuredContent):
    concept_code: str  # Must be PascalCase
```

Standard mock generators (like Polyfactory) produce random strings like `"uygNjiAuDMOtZEyibgHw"` which fail validation. The dry run system addresses this at two levels:

1. **Field-level**: Generate values matching expected formats (snake_case, PascalCase, concept refs, etc.)
2. **Model-level**: Bypass validators using `factoVry_use_construct=True`

---

## When Dry Run Mock Generation Is Used

```mermaid
flowchart TD
    A[pipelex validate --all] --> B[BundleValidator.validate_pipes]
    B --> C[WorkingMemoryFactory.make_mock_inputs]
    C --> D[DryRunFactory.make_dry_run_factory]
    D --> E[Polyfactory with custom providers]

    F[PipeLLM dry run] --> G[cogt leaf: dry_llm_gen_object in dry_mock.py]
    G --> H[DryRunFactory.make_dry_run_factory]
    H --> E

    I[PipeCompose dry run] --> J[StructuredContentComposer.compose]
    J --> K[model_validate with resolved values]
```

A dry run does not swap in a special content generator: `run_mode=DRY` rides `CogtRunParams` on every cogt assignment, and each inference leaf (`llm_generate`, the `*_and_store` image/extract leaves, search, templating) branches to its `dry_*` mock helper in `dry_mock.py` — identically whether the leaf runs inline or inside a Temporal activity.

| Trigger | Entry Point | Mock Generation |
|---------|-------------|-----------------|
| `pipelex validate` | `BundleValidator.validate_pipes()` | `WorkingMemoryFactory.make_mock_inputs()` |
| `PipeLLM` output in dry mode | `dry_llm_gen_object()` / `dry_llm_gen_object_list()` (cogt leaf, `dry_mock.py`) | `DryRunFactory` via the caller's class in-process, via the schema-reconstructed class on a worker |
| `PipeSearch` structured output in dry mode | `dry_search_gen_structured_object()` in-process / `dry_search_gen_structured()` on a worker (cogt leaf, `dry_mock.py`) | Same split as `PipeLLM`; the worker arm dumps its mock to a dict, because the leaf's result crosses the boundary as data |
| `PipeFunc` output in dry mode | `WorkingMemoryFactory.make_mock_content()` | `DryRunFactory` (with explicit field constraints) |
| `PipeCompose` in dry mode | `StructuredContentComposer.compose()` | Uses resolved values from working memory; falls back to a `DryRunFactory` mock of the output class only when a value error hits under mock inputs |

---

## Architecture

### MockFormat Enum

Located at `pipelex/cogt/content_generation/dry_run_factory.py`, the `MockFormat` enum defines all supported mock value formats:

```python
class MockFormat(StrEnum):
    SNAKE_CASE = "snake_case"
    PASCAL_CASE = "pascal_case"
    CONCEPT_REF = "concept_ref"
    IGNORE = "ignore"
    DICT_SNAKE_KEY_PASCAL_VALUE = "dict_snake_key_pascal_value"
    DICT_SINGLE_EXTRACT_INPUT = "dict_single_extract_input"
```

### DryRunFactory

The factory dynamically creates a Polyfactory `ModelFactory` subclass with field-specific providers:

```python
class DryRunFactory:
    @classmethod
    def generate_snake_case_code(cls) -> str:
        suffix = "".join(random.choices(string.ascii_lowercase, k=4))
        return f"mock_{suffix}"  # e.g., "mock_abcd"

    @classmethod
    def generate_pascal_case_code(cls) -> str:
        suffix = "".join(random.choices(string.ascii_lowercase, k=4))
        return f"Mock{suffix.capitalize()}"  # e.g., "MockAbcd"

    @classmethod
    def generate_concept_ref(cls) -> str:
        return f"{cls.generate_snake_case_code()}.{cls.generate_pascal_case_code()}"
        # e.g., "mock_abcd.MockCdef"

    @classmethod
    def make_dry_run_factory(
        cls,
        object_class: type[BaseModelTypeVar],
        *,
        snake_case_field_names: set[str] | None = None,
        pascal_case_field_names: set[str] | None = None,
    ) -> type[ModelFactory[BaseModelTypeVar]]:
        ...
```

---

## Generated Mock Value Formats

| Format | Generator | Example Output | Used For |
|--------|-----------|----------------|----------|
| `SNAKE_CASE` | `generate_snake_case_code()` | `mock_abcd` | `domain_code`, `pipe_code`, `the_field_name` |
| `PASCAL_CASE` | `generate_pascal_case_code()` | `MockAbcd` | `concept_code` |
| `CONCEPT_REF` | `generate_concept_ref()` | `mock_abcd.MockCdef` | `concept_ref` field (domain.ConceptCode format) |
| `IGNORE` | Sets field to `Ignore()` | None/default | `default_value`, `structure` fields |
| `DICT_SNAKE_KEY_PASCAL_VALUE` | `generate_dict_snake_key_pascal_value()` | `{mock_abcd: MockCdef}` | `inputs` dict in PipeSpec |
| `DICT_SINGLE_EXTRACT_INPUT` | `generate_dict_single_extract_input()` | `{mock_abcd: "Image"}` | `inputs` dict in PipeExtract |
| Random string | Polyfactory default | `uygNjiAuDMOtZEyibgHw` | All other string fields |

---

## Declaring MockFormat on Fields

The `DryRunFactory` auto-detects format constraints from Pydantic `Field` definitions using the `mock_format` key in `json_schema_extra`:

```python
from pydantic import Field
from pipelex.cogt.content_generation.dry_run_factory import MockFormat

class ConceptStructureSpec(StructuredContent):
    # Snake case field
    the_field_name: str = Field(
        description="Field name. Must be snake_case.",
        json_schema_extra={"mock_format": MockFormat.SNAKE_CASE}
    )

    # Concept reference field (domain.ConceptCode format)
    concept_ref: str | None = Field(
        default=None,
        description="For type='concept', the concept reference.",
        json_schema_extra={"mock_format": MockFormat.CONCEPT_REF},
    )

    # Field to ignore during mock generation (use default/None)
    default_value: Any | None = Field(
        default=None,
        json_schema_extra={"mock_format": MockFormat.IGNORE}
    )
```

---

## Using Field Examples for Enum-like Values

For fields that should pick from a set of valid values (like enum members or known strings), use the `examples` parameter. The factory's `__use_examples__: True` configuration makes Polyfactory randomly select from provided examples:

```python
class ConceptSpec(StructuredContent):
    # Refines should be one of the native concepts
    refines: str | None = Field(
        default=None,
        examples=["Text", "Image", "Document", "TextAndImages", "Number", "Page"],
    )

class PipeComposeSpec(PipeSpec):
    # Target format should be a valid TargetFormat value
    target_format: TargetFormat | str | None = Field(
        default=None,
        description="Target format for the output (template mode)",
        examples=list(TargetFormat),  # Polyfactory picks randomly from these
    )
```

---

## Cross-Field Dependencies with PostGenerated

For fields that depend on values generated for other fields, use Polyfactory's `PostGenerated` directive. The factory automatically handles the `main_pipe` field which must reference a key from the generated `pipe` dict:

```python
# Inside DryRunFactory.make_dry_run_factory():
if "main_pipe" in object_class.model_fields and "pipe" in object_class.model_fields:
    class_attrs["main_pipe"] = PostGenerated(cls._main_pipe_from_pipe_dict)
```

The callback receives all previously generated values and can compute the dependent field:

```python
@staticmethod
def _main_pipe_from_pipe_dict(_field_name: str, values: dict[str, Any]) -> str:
    pipe_dict: dict[str, Any] | None = values.get("pipe")
    if pipe_dict and len(pipe_dict) > 0:
        pipe_keys: list[str] = list(pipe_dict.keys())
        return random.choice(pipe_keys)
    # Fallback to a mock value if pipe dict is empty/not available
    return "mock_" + "".join(random.choices(string.ascii_lowercase, k=4))
```

---

## Implementation Details

### Mock Input Generation

When `WorkingMemoryFactory.make_mock_inputs()` creates mock working memory:

```python
@classmethod
def make_mock_content(cls, typed_named_stuff_spec: TypedNamedStuffSpec) -> StuffContent:
    mock_factory = DryRunFactory.make_dry_run_factory(
        object_class=typed_named_stuff_spec.structure_class,
        snake_case_field_names=SNAKE_CASE_FIELD_NAMES,
        pascal_case_field_names=PASCAL_CASE_FIELD_NAMES,
    )
    return mock_factory.build(factory_use_construct=True)
```

The `factory_use_construct=True` flag bypasses `field_validator` and `model_validator` during object creation, preventing validation errors from random nested values.

### LLM Output Generation

When the dry LLM leaf (`dry_llm_gen_object` in `dry_mock.py`) generates mock outputs, it resolves the output class exactly the way the live leaf does — via `resolve_object_class` — then builds via `build_mock_object()`:

```python
item_class = resolve_object_class(object_assignment=..., object_class=...)
return build_mock_object(item_class)
```

`object_class` is the caller's live class when there is one (the in-process case), and `None` when only the serialized assignment is in hand (the distributed-worker case), in which case the class is rebuilt from the assignment's JSON schema. Keeping the two run modes on one resolution matters: if only the live leaf used the real class, the mock would be built against a *weaker* class than the one the provider is constrained by — the mock would be less faithful than the thing it mocks.

No explicit field constraints are passed — the factory auto-detects `MockFormat` from field definitions. On the rebuilt-class path, `json_schema_extra` hints and custom validators are dropped by the schema round-trip: classes with exotic format constraints should declare `examples` / `mock_format`, otherwise re-validation against the original class raises a typed `DryRunObjectFidelityError`. On the live-class path those invariants are present at build time instead, so an unsatisfiable one surfaces as a typed `DryRunMockBuildError` naming the same remedy.

### PipeCompose Resolution

`StructuredContentComposer.compose()` does **not** generate mocks. Instead, it resolves field values from working memory and validates the result:

```python
async def compose(self) -> StuffContent:
    field_values = await self._resolve_all_fields()
    try:
        return self.output_class.model_validate(field_values)
    except ValidationError as exc:
        formatted_error = format_pydantic_validation_error(exc)
        msg = f"Cannot validate {self.output_class.__name__}: {formatted_error}"
        raise StructuredContentComposerValidationError(msg) from exc
```

In dry run mode, the working memory already contains properly formatted mock values (generated by `WorkingMemoryFactory`), so validation typically succeeds.

There is one fallback: when composition fails with a `StructuredContentComposerValueError` (e.g. a dotted path that mock data cannot resolve, such as a field on a description-only concept) *and* the run uses mock inputs, `PipeCompose` builds a `DryRunFactory` mock of the output class instead of failing. Type-incompatibility and validation errors still raise, and with real inputs (e.g. `pipelex validate`) unresolvable paths raise too — those are genuine bugs, not mock artifacts.

---

## Behavior Matrix

| Scenario | Format Constraints | Validators Bypassed |
|----------|-------------------|---------------------|
| `WorkingMemoryFactory.make_mock_content()` | Yes (from `json_schema_extra` + explicit params) | Yes (`factory_use_construct`) |
| `dry_llm_gen_object()` (cogt leaf) | Yes (from `json_schema_extra` surviving the schema round-trip) | No (validators run; failures raise typed `DryRunMockBuildError`) |
| `StructuredContentComposer.compose()` | N/A (uses resolved values) | No (validates) |

---

## Extending Field Constraints

### Adding a New MockFormat Value

1. **Add to the MockFormat enum** in `dry_run_factory.py`:

    ```python
    class MockFormat(StrEnum):
        ...
        KEBAB_CASE = "kebab_case"
    ```

2. **Add a generator method**:

    ```python
    @classmethod
    def generate_kebab_case_code(cls) -> str:
        suffix = "".join(random.choices(string.ascii_lowercase, k=4))
        return f"mock-{suffix}"
    ```

3. **Handle the format in `make_dry_run_factory()`**:

    ```python
    for field_name in detected_formats[MockFormat.KEBAB_CASE]:
        if field_name in object_class.model_fields:
            class_attrs[field_name] = Use(cls.generate_kebab_case_code)
    ```

### Using the New Format on Fields

```python
class MySpec(StructuredContent):
    my_kebab_field: str = Field(
        description="A kebab-case identifier",
        json_schema_extra={"mock_format": MockFormat.KEBAB_CASE}
    )
```

!!! warning "Field name matching is exact"
    The field name must exactly match a key in `object_class.model_fields`. No glob patterns or inheritance traversal.

---

## File Reference

| File | Purpose |
|------|---------|
| `pipelex/cogt/content_generation/dry_run_factory.py` | `DryRunFactory` class with `MockFormat` enum and generators |
| `pipelex/cogt/content_generation/dry_mock.py` | Leaf-level dry/mock helpers (`dry_llm_gen_*`, `build_mock_object`, `stamp_mock_main_coordination`) |
| `pipelex/core/memory/working_memory_factory.py` | `WorkingMemoryFactory.make_mock_content()` with field constraints |
| `pipelex/pipe_operators/compose/structured_content_composer.py` | Composes `StructuredContent` from working memory (no mocks) |
| `pipelex/pipeline/dry_run_pipeline.py` | `dry_run_pipeline()` orchestration |
| `pipelex/pipe_run/dry_run_in_process.py` | `dry_run_pipe_in_process()` — the in-process, never-dispatching twin |

---

## Next Steps

- [Architecture Overview](./architecture-overview.md) — Understand the two-layer design
- [Test Profile Configuration](./test-profile-configuration.md) — Configure model sets for testing


## Execution Graph Tracing

# Execution Graph Tracing

Pipelex captures pipeline executions as directed graphs for visualization and debugging. The `pipelex/graph/` module provides tracing infrastructure, a canonical data model (GraphSpec), and multiple rendering backends (Mermaid, ReactFlow).

---

## Design Principle

Pipeline executions form implicit graphs: pipes call other pipes, data flows between them. The graph module makes this structure explicit:

1. **Trace at runtime**: Instrument pipe execution to capture nodes (pipes) and edges (relationships)
2. **Store canonically**: GraphSpec is a versioned, renderer-agnostic JSON model
3. **Render as needed**: Transform GraphSpec into Mermaid diagrams or ReactFlow visualizations

```
Pipe Execution → GraphTracer → GraphSpec → Renderers → HTML/Mermaid
```

!!! info "Non-Intrusive Design"
    Graph tracing is opt-in. When disabled, a no-op tracer is used with zero overhead. The tracer is injected via `TraceContext` in `JobMetadata`, not global state.

---

## Usage Variants

| Scenario | CLI | API | Result |
|----------|-----|-----|--------|
| Generate execution graph | `pipelex run pipe my_pipe --graph` | `PipelexMTHDSProtocol(execution_config=...).execute(...)` | GraphSpec JSON + HTML viewers |
| Force include full data | `--graph --graph-full-data` | `data_inclusion.stuff_json_content=True` | Data embedded in IOSpec |
| Force exclude data | `--graph --graph-no-data` | The four stuff/error `data_inclusion` flags → `False` (`pipe_and_concept_registry` unaffected) | Previews only |
| Dry run with graph | `--dry-run --graph` | `PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.DRY, execution_config=...)` | Graph of mock execution |

!!! info "Full Data Included by Default"
    The default configuration includes full data in graphs (`stuff_json_content`, `stuff_text_content`, `stuff_html_content`, `error_stack_traces`, and `pipe_and_concept_registry` are all `true`). Use `--graph-full-data` or `--graph-no-data` only to override project-specific settings — the flags toggle the first four; `pipe_and_concept_registry` is set only via config.

---

## Interfaces

### CLI Commands

```bash
# Run pipeline and generate graph
pipelex run pipe my_pipe --graph

# Include full serialized data
pipelex run pipe my_pipe --graph --graph-full-data

# Exclude data (previews only)
pipelex run pipe my_pipe --graph --graph-no-data

# Dry run with graph tracing
pipelex run pipe my_pipe --dry-run --graph --mock-inputs
```

### API

```python
from pipelex.pipeline.runner import PipelexMTHDSProtocol
from pipelex.system.pipe_run_mode import PipeRunMode

# Execute with graph tracing via config
runner = PipelexMTHDSProtocol(
    execution_config=config.with_execution_overrides(generate_graph=True),
)
response = await runner.execute(
    pipe_code="my_pipe",
)
pipe_output = response.pipe_output

# Dry run with graph: the same runner in DRY mode with mock inputs — no separate code path.
dry_runner = PipelexMTHDSProtocol(
    pipe_run_mode=PipeRunMode.DRY,
    execution_config=config.with_execution_overrides(generate_graph=True, mock_inputs=True),
)
response = await dry_runner.execute(pipe_code="my_pipe")
graph_spec = response.pipe_output.graph_spec
```

!!! tip "Dry run from MTHDS content"
    To dry-run an entire bundle straight from MTHDS content and get back a `GraphSpec`, use `dry_run_pipeline(mthds_contents=...)` (`pipelex/pipeline/dry_run_pipeline.py`) — the shared entrypoint behind the CLI graph commands, which wires the same DRY-mode runner for you. It owns its graph transport (a scoped in-memory event log), so it produces the graph regardless of the host's `tracing_config` and never writes trace files as a side effect.

### Outputs

| Output | File | Purpose |
|--------|------|---------|
| `graphspec_json` | `graphspec.json` | Canonical graph representation |
| `mermaidflow_mmd` | `mermaidflow.mmd` | Mermaid flowchart code |
| `mermaidflow_html` | `mermaidflow.html` | Standalone Mermaid viewer |
| `reactflow_html` | `reactflow.html` | Interactive ReactFlow viewer |

---

## Architecture

```mermaid
flowchart TB
    subgraph EXECUTION["Pipeline Execution"]
        direction TB
        PIPE["PipeAbstract.run_pipe()"]
        JOB["JobMetadata"]
        PIPE --> JOB
    end

    subgraph TRACING["Graph Tracing"]
        direction TB
        MGR["GraphTracerManager<br/>(singleton)"]
        TRACER["GraphTracer"]
        CTX["TraceContext"]
        MGR --> TRACER
        TRACER --> CTX
    end

    subgraph MODEL["Canonical Model"]
        direction TB
        SPEC["GraphSpec"]
        NODE["NodeSpec"]
        EDGE["EdgeSpec"]
        IO["IOSpec"]
        SPEC --> NODE
        SPEC --> EDGE
        NODE --> IO
    end

    subgraph ANALYSIS["Pre-computed Analysis"]
        direction TB
        GA["GraphAnalysis"]
        TREE["containment_tree"]
        STUFF["stuff_registry"]
        GA --> TREE
        GA --> STUFF
    end

    subgraph RENDER["Renderers"]
        direction TB
        MF["MermaidflowFactory"]
        RF["ReactFlow HTML Generator"]
        MF --> HTML1["Mermaid HTML"]
        RF --> HTML2["ReactFlow HTML"]
    end

    JOB --> CTX
    CTX --> TRACER
    TRACER --> SPEC
    SPEC --> GA
    GA --> MF
    GA --> RF
```

---

## Core Components

### GraphSpec

The canonical, versioned data model for execution graphs. Designed for JSON serialization and renderer-agnostic storage.

```python
class GraphSpec(BaseModel):
    graph_id: str
    created_at: datetime
    pipeline_ref: PipelineRef
    nodes: list[NodeSpec]
    edges: list[EdgeSpec]
    usage: GraphUsageSpec | None
    meta: dict[str, Any]

    def to_json(self) -> str:
        return self.model_dump_json(by_alias=True, indent=2)
```

`usage` (and its per-node counterpart `NodeSpec.usage`) carries the run's inference usage attributed to graph position — see [Per-node Usage Attribution](per-node-usage-attribution.md) for the three usage states and the invariants a consumer must branch on.

`meta.format` is always `"mthds"` on newly emitted Pipelex graphs. `meta.mode` records provenance for renderers and shared tooling: Pipelex execution graphs emit `"dry"` for dry-run/mock execution and `"live"` for real execution. The shared renderer also accepts `"static"` for graphs produced by a static MTHDS graph builder.

### Node Types

| NodeKind | Description |
|----------|-------------|
| `PIPE_CALL` | Generic pipe invocation |
| `CONTROLLER` | PipeController (Sequence, Parallel, etc.) |
| `OPERATOR` | PipeOperator (LLM, Extract, etc.) |
| `INPUT` | Pipeline input node |
| `OUTPUT` | Pipeline output node |
| `ARTIFACT` | Generated artifact |
| `ERROR` | Error node |

### Edge Types

| EdgeKind | Description |
|----------|-------------|
| `CONTROL` | Execution flow between pipes |
| `DATA` | Data flow (stuff passed between pipes) |
| `CONTAINS` | Parent-child containment (controller → children) |
| `SELECTED_OUTCOME` | Condition outcome selection |
| `BATCH_ITEM` | Batch fan-out: input list → item extracted for each batch iteration |
| `BATCH_AGGREGATE` | Batch fan-in: item outputs → aggregated output list |
| `PARALLEL_COMBINE` | Branch outputs → combined output in PipeParallel |

Every `EdgeSpec` also carries an `optional` boolean (default `false`). On a `DATA` edge it marks that the producer's output is declared optional (`?` presence marker) — the value flowed on this run but may be absent on others. Renderers can use it to draw the edge distinctly (e.g. dashed).

### Node Status

| NodeStatus | Description |
|------------|-------------|
| `SCHEDULED` | Not yet started |
| `RUNNING` | Currently executing |
| `SUCCEEDED` | Completed successfully |
| `FAILED` | Execution failed |
| `SKIPPED` | Lifted (skipped) because a plain input resolved absent — a successful outcome, not an error |
| `CANCELED` | Canceled before completion |

A `SKIPPED` node also carries `skip_reason` (a human-readable sentence naming the absent input, e.g. `skipped because input 'source' is absent`). A lifted pipe with a plural output still writes a real empty-list value, so its node registers that output and downstream `DATA` edges resolve normally; a lifted singular output is a recorded absence with no output spec.

---

## Implementation

### Tracing Flow

```python
# 1. Manager opens tracer for pipeline run
manager = GraphTracerManager.get_or_create_instance()
trace_context = manager.open_tracer(
    graph_id=pipeline_run_id,
    data_inclusion=config.data_inclusion,
    pipeline_ref_domain="my_domain",
    pipeline_ref_main_pipe="my_pipe",
)

# 2. Context flows through JobMetadata to each pipe
job_metadata = JobMetadata(
    pipeline_run_id=pipeline_run_id,
    trace_context=trace_context,
)

# 3. Each pipe reports start/end to tracer
node_id, child_context = manager.on_pipe_start(
    trace_context=trace_context,
    pipe_code="extract_text",
    pipe_type="PipeExtract",
    node_kind=NodeKind.OPERATOR,
    started_at=datetime.now(timezone.utc),
    input_specs=[...],
)

# 4. On completion, report success with output
manager.on_pipe_end_success(
    lookup_key=trace_context.lookup_key,
    node_id=node_id,
    ended_at=datetime.now(timezone.utc),
    output_spec=IOSpec(...),
)

# 5. Manager closes tracer and returns GraphSpec
graph_spec = manager.close_tracer(pipeline_run_id)
```

### Event-Log Transport and the Scoped Override

Trace events travel through an `EventLogProtocol` backend (`pipelex/tracing/`): the tracer emits events into it during the run (write side, wired in `pipeline_run_setup`), and `assemble_tracing` reads them back after the run to build the `GraphSpec` and usage aggregates (read side, triggered from `PipeRun.run`). Both sides normally build their backend instance independently from `tracing_config` via `make_event_log` — NDJSON files or DynamoDB bridge the two instances through external storage.

The assembled usage rides back on `pipe_output.tokens_usages` (with any assembly failure on `usage_assembly_error`), which the sync `/execute` response returns directly. For delivery-enabled runs (a storage target set), the delivery executor (`pipelex/pipe_run/delivery_executor.py`) also persists it as a `tokens_usages.json` result artifact — `{"tokens_usages": [...], "usage_assembly_error": null}` — next to `working_memory.json`, the `main_stuff.*` renders, and the graph outputs, so a durable client polling result files gets the same usage records a sync caller does. The artifact is written unconditionally on every successful result delivery: explicit nulls mean usage assembly was off for that run. A failed run stores no result files at all, so an absent `tokens_usages.json` means either the run failed or it was delivered before the artifact existed — tell those apart from the delivery status, not from the file's presence.

For fully in-process runs, `pipelex.runtime_hub.scoped_event_log` pins one shared instance for both sides instead:

```python
from pipelex.runtime_hub import scoped_event_log
from pipelex.tracing.in_memory_event_log import InMemoryEventLog

with scoped_event_log(InMemoryEventLog()):
    response = await runner.execute(...)  # graph assembles in memory
```

Semantics:

- The override is ContextVar-scoped (mirrors `scoped_pipe_router`), so concurrent runs with separate scopes never cross-contaminate, and the prior value is restored on exit.
- A set override **implies tracing-enabled**: it is honored even when `tracing_config.is_enabled` is `False`, on both the write side and the read side's early-return.
- Lifecycle: the read side does not `close()` the scoped instance and the machinery never calls `cleanup()` on it — but the write-side tracer DOES call `close()` on its event log at teardown, before the read side assembles. A scoped event log's `close()` must therefore be idempotent or a no-op (as `InMemoryEventLog`'s is); scoping a backend whose `close()` releases a real resource would break its own assembly read.

This is what lets a graph-producing dry-run trace entirely in memory (no NDJSON file, no DynamoDB round-trip). Both dry-run entrypoints rely on it: `dry_run_pipe_in_process` (`pipelex/pipe_run/dry_run_in_process.py` — the graph arm of protocol `validate` and of the single Temporal validation activity) and `dry_run_pipeline` (`pipelex/pipeline/dry_run_pipeline.py`) itself — these functions exist to produce a graph, so they install their own scoped `InMemoryEventLog` rather than depending on the host having tracing configured (a host with `tracing_config.is_enabled = false`, like pipelex-api's `/validate` in direct mode, still gets its graph). A run that nonetheless finishes without a graph raises the typed `DryRunGraphNotProducedError`.

### TraceContext Propagation

TraceContext is a serializable context that flows through pipe execution:

```python
class TraceContext(BaseModel):
    graph_id: str                           # Unique graph identifier
    parent_node_id: str | None              # Parent pipe's node ID
    node_sequence: int                      # Counter for generating node IDs
    data_inclusion: DataInclusionConfig     # What data to capture

    def copy_for_child(self, child_node_id: str, *, next_sequence: int) -> TraceContext:
        """Create context for nested pipe execution."""
        return TraceContext(
            graph_id=self.graph_id,
            parent_node_id=child_node_id,
            node_sequence=next_sequence,
            data_inclusion=self.data_inclusion,
        )
```

### Producer/Consumer Paradigm

Data flow in the graph follows a producer/consumer model:

- **Producers**: Nodes that create or output data (stuff). When a pipe outputs a value, it becomes the producer of that data item.
- **Consumers**: Nodes that receive or use data as input. When a pipe takes a value as input, it becomes a consumer of that data item.

Each piece of data is identified by a **digest** (a unique hash). This allows the tracer to track where data originates and where it flows, even when the same value passes through multiple pipes.

```
Producer Node ──(outputs)──► Stuff (digest: abc123) ──(inputs)──► Consumer Node
```

This paradigm enables:

- Automatic DATA edge generation without explicit wiring
- Visualization of data lineage across the execution graph
- Debugging by tracing which pipe produced unexpected output

### Data Flow Edge Generation

DATA edges are generated at teardown by correlating input/output digests:

```python
def _generate_data_edges(self) -> None:
    for consumer_node_id, node_data in self._nodes.items():
        for input_spec in node_data.input_specs:
            if input_spec.digest is None:
                continue
            producer_node_id = self._stuff_producer_map.get(input_spec.digest)
            if producer_node_id and producer_node_id != consumer_node_id:
                self.add_edge(
                    source_node_id=producer_node_id,
                    target_node_id=consumer_node_id,
                    edge_kind=EdgeKind.DATA,
                    label=input_spec.name,
                )
```

---

## GraphAnalysis

Pre-computed analysis layer that extracts common information for renderers:

```python
analysis = GraphAnalysis.from_graphspec(graph_spec)

# Lookups
node = analysis.nodes_by_id["node_123"]
children = analysis.get_children("controller_node")
is_root = analysis.is_root("node_123")

# Data flow
stuff_info = analysis.get_stuff_info(digest="abc123")
producer = analysis.get_producer(digest="abc123")
consumers = analysis.get_consumers(digest="abc123")
```

| Attribute | Purpose |
|-----------|---------|
| `nodes_by_id` | Fast node lookup by ID |
| `containment_tree` | Parent → children mapping |
| `child_node_ids` | All nodes that have parents |
| `controller_node_ids` | Nodes with children |
| `root_nodes` | Top-level nodes |
| `stuff_registry` | Digest → StuffInfo |
| `stuff_producers` | Digest → producer node ID |
| `stuff_consumers` | Digest → consumer node IDs |

---

## Rendering

### Mermaidflow

Converts GraphSpec to Mermaid flowchart syntax with controller subgraphs:

```python
from pipelex.graph.mermaidflow.mermaidflow_factory import MermaidflowFactory

mermaidflow = MermaidflowFactory.make_from_graphspec(
    graph_spec,
    graph_config=graph_config,
    direction=FlowchartDirection.TOP_DOWN,
    include_subgraphs=True,
)

print(mermaidflow.mermaid_code)
```

**Output structure:**

- Controllers rendered as subgraphs
- Operators rendered as rectangles inside subgraphs
- Stuff nodes (data items) rendered as stadium shapes
- DATA edges connect producers → stuff → consumers

### ReactFlow HTML

ReactFlow HTML is generated directly from GraphSpec — no intermediate ViewSpec layer. The HTML generator embeds GraphSpec as JSON and the client-side JavaScript handles dataflow analysis, layout, and rendering.

---

## Configuration

### GraphConfig

```toml
# pipelex.toml (default values)
[pipelex.pipeline_execution_config.graph_config]

[pipelex.pipeline_execution_config.graph_config.data_inclusion]
stuff_json_content = true       # Include full serialized data
stuff_text_content = true       # Include ASCII text representation
stuff_html_content = true       # Include HTML representation
error_stack_traces = true       # Include full stack traces
pipe_and_concept_registry = true  # Include pipe and concept registries in the GraphSpec

[pipelex.pipeline_execution_config.graph_config.graphs_inclusion]
graphspec_json = true           # Generate GraphSpec JSON
mermaidflow_mmd = true          # Generate Mermaid code
mermaidflow_html = true         # Generate Mermaid HTML
reactflow_html = true           # Generate ReactFlow HTML
```

### MermaidRenderingConfig

| Option | Description |
|--------|-------------|
| `direction` | Flowchart direction (`top_down`, `left_to_right`) |
| `is_include_data_edges` | Show data flow edges |
| `is_include_contains_edges` | Show containment edges |
| `is_include_selected_outcome_edges` | Show condition-result (selected outcome) edges |
| `is_show_stuff_codes` | Show digest in stuff labels |
| `style.theme` | Mermaid theme (default, base, dark, forest, neutral) |

### ReactFlowRenderingConfig

| Option | Description |
|--------|-------------|
| `is_use_cdn` | Load the graph viewer assets from a CDN (jsDelivr) instead of inlining them |
| `layout_direction` | Flowchart layout direction (`top_down`, `left_to_right`), converted internally to Dagre's TB/LR |
| `nodesep` | Node separation in pixels |
| `ranksep` | Rank separation in pixels |
| `edge_type` | Edge style (bezier, smoothstep, step, straight) |
| `initial_zoom` | Initial viewport zoom |
| `pan_to_top` | Pan the viewport to the top on load |
| `default_title` | Title of the generated HTML page |
| `show_batch_controller` | Render controller pipes as grouping containers |
| `style.theme` | UI theme (light, dark, system) |
| `style.palette` | Node color palette (`yellow_blue`, `dracula`) |

---

## IOSpec Data Capture

IOSpec captures input/output data for nodes:

```python
class IOSpec(BaseModel):
    name: str                    # Variable name
    concept: str | None          # Concept code
    content_type: str | None     # MIME type
    preview: str | None          # Truncated preview (max 200 chars)
    size: int | None             # Content size
    digest: str | None           # Unique identifier for data flow
    data: str | dict[str, Any] | list[str] | list[dict[str, Any]] | None  # Full serialized content
    data_text: str | None        # ASCII text representation
    data_html: str | None        # HTML representation
    extra: dict[str, Any]        # Extra markers, e.g. `optional` set on a declared-optional (`?`) output
```

!!! warning "Preview Truncation"
    Previews are automatically truncated to 200 characters. Stack traces are truncated to 2000 characters. Use `--graph-full-data` to capture complete content.

---

## Validation

GraphSpec validation enforces invariants:

```python
from pipelex.graph.validation import validate_graphspec

validate_graphspec(graph_spec)
```

| Invariant | Error |
|-----------|-------|
| Duplicate node IDs | `GraphSpecValidationError` |
| Duplicate edge IDs | `GraphSpecValidationError` |
| Edge references non-existent node | `GraphSpecValidationError` |
| Failed node without error spec | `GraphSpecValidationError` |

---

## Syntax Quick Reference

| Pattern | Purpose |
|---------|---------|
| `GraphSpec.to_json()` | Serialize to JSON string |
| `GraphAnalysis.from_graphspec(g)` | Pre-compute analysis |
| `MermaidflowFactory.make_from_graphspec(...)` | Generate Mermaid |
| `generate_reactflow_html(graphspec, config=...)` | Generate ReactFlow HTML |
| `generate_graph_outputs(g, graph_config=..., pipe_code=...)` | Generate all outputs |

---

## Files Reference

| File | Purpose |
|------|---------|
| `pipelex/graph/graphspec.py` | Canonical GraphSpec model |
| `pipelex/tracing/usage_attribution.py` | Per-node usage accumulation and subtree rollup |
| `pipelex/graph/graph_tracer.py` | GraphTracer implementation |
| `pipelex/graph/graph_tracer_manager.py` | Singleton manager for tracers |
| `pipelex/graph/graph_tracer_protocol.py` | Protocol + NoOp implementation |
| `pipelex/system/trace_context.py` | Serializable tracing context — sits below `graph/` because it rides in every job's metadata |
| `pipelex/system/data_inclusion_config.py` | Data-capture flags carried by the trace context, surfaced in the TOML under `graph_config.data_inclusion` |
| `pipelex/graph/graph_analysis.py` | Pre-computed graph analysis |
| `pipelex/graph/graph_factory.py` | Output generation factory |
| `pipelex/graph/graph_config.py` | Configuration models |
| `pipelex/graph/validation.py` | GraphSpec validation |
| `pipelex/graph/exceptions.py` | Graph-specific exceptions |
| `pipelex/graph/mermaidflow/` | Mermaid rendering |
| `pipelex/graph/reactflow/` | ReactFlow rendering |

---

## Next Steps

- [:material-sitemap: Architecture Overview](./architecture-overview.md){ .md-button .md-button--primary }
- [:material-image-multiple: Image Handling in LLM Prompts](./image-handling-in-llm-prompts.md){ .md-button }


## TokensUsage Wire Records

# TokensUsage Wire Records

`TokensUsageRecord` (`pipelex/reporting/usage_records.py`) is the ONLY shape of a run's token usage that crosses the client boundary. Internally, the runtime tracks usage with full-fidelity models (`AnyTokensUsage` carrying a `JobMetadata`); those internals never reach a client — the converter trims them at the two terminal emission points.

## The two surfaces

Both surfaces carry the same record shape:

- **Blocking `/execute` response** — `pipe_output.tokens_usages` on the response body. The API server dumps its full response, then calls `apply_tokens_usage_wire_shape(response_dump, pipe_output=...)` to replace the usage dumps with wire records.
- **Durable `tokens_usages.json` result artifact** — written by the delivery executor as `{"tokens_usages": [...] | null, "usage_assembly_error": string | null}` and relayed verbatim by the platform's runs-results endpoint.

## Record fields

One record per inference call, in completion order:

| field | type | semantics |
|---|---|---|
| `model_type` | string | Kind of inference: known values `llm`, `img_gen`, `extract`, `search`. Open set. |
| `inference_model_name` | string | Human model name (e.g. `gpt-4o`). |
| `inference_model_id` | string | Provider/platform model id (e.g. `gpt-4o-2024-11-20`). |
| `pipe_code` | string \| null | Pipe that made the call — per-pipe cost attribution. |
| `job_category` | string \| null | Known values: `llm_job`, `img_gen_job`, `extract_job`, `search_job`, `jinja2_job`, `mock_job`. Open set. |
| `unit_job_id` | string \| null | Known values: `llm_gen_text`, `llm_gen_object`, `img_gen_text_to_image`, `extract_pages`, `search_sourced_answer`, `search_structured`. Open set. |
| `nb_tokens_by_category` | object&lt;string, int&gt; | Raw provider-reported token counts, keyed by `TokenCategory` values (`input`, `input_cached`, `output`, ...). Open set. `input` is the joined total; `input_cached` is a subset of it, not additive. |
| `cost` | number \| null | Computed USD cost of this call. `null` when the model has no rate table (own-GPU, mock, dry-run). |
| `started_at` | string \| null | ISO 8601. |
| `completed_at` | string \| null | ISO 8601. Duration is client-derivable; not shipped. |

Enum-ish fields are **open sets** on the wire: consumers must treat them as plain strings, so runtime enum churn is non-breaking.

## What never crosses the wire

The converter deliberately drops runtime plumbing and the rate table: `job_metadata` (the object and any nesting of it), `unit_costs`, `user_id`, `session_id`, `request_id`, `pipe_run_id`, `otel_context`, `trace_context`, `content_generation_job_id`, and `pipeline_run_id` (redundant — the envelope already identifies the run). `TokensUsageRecord` is `extra="forbid"` server-side so an accidental new field fails loudly instead of silently widening the contract.

The record shape is also invariant to server-side observability settings: telemetry/tracing on or off, a consumer always gets structurally identical records.

## Cost semantics

`cost` is computed by `compute_tokens_usage_cost` (`pipelex/cogt/usage/cost_registry.py`) from the rates stamped on the internal record at inference time — the same cost engine and the same canonical total as the CLI cost table: input_non_cached + input_cached + output component costs, with the cached-discount fallback (cached input at 50% of the input rate when no explicit cached rate exists). Categories the cost engine excludes from totals (audio, reasoning, prediction) are excluded here too. A run-level total is the sum of per-record `cost` values.

`cost: null` means "unrated" (no rate table: own-GPU models, mock/dry runs) — distinct from `0.0`, which is a rated call that cost nothing.

## Null semantics

- `tokens_usages: null` — usage assembly was off for the run. (The durable file is absent entirely when the run failed — a failed run stores no result files — or, hosted, when it was delivered before the artifact existed.)
- `tokens_usages: []` — assembly on, no inference happened.
- `usage_assembly_error` non-null — assembly *broke* (event-read failure), distinct from "off". It is a bare diagnostic string, mirroring `graph_assembly_error`.

## Where the trim happens (and where it must NOT)

The trim runs only at the terminal, client-facing emitters:

1. `DeliveryExecutor._generate_usage_file` (durable artifact write, worker-side).
2. `apply_tokens_usage_wire_shape` on the `/execute` response dump (called by the API server).

Every internal crossing keeps full fidelity: the runtime-bridge `tokens_usages_dump` (rehydrated back into typed records by the host), `PipeOutput` over the Temporal kajson wire, and the usage-event telemetry stream (NDJSON / event log), which keeps emitting full records including `job_metadata` — that stream is operator-side observability, never returned to clients.

For this reason the trim must never be implemented as a `field_serializer`/`model_serializer` on `PipeOutput`: it would silently trim internal transport too.

## Related pages

- [Cost Tracking & Reporting](../features/cost-tracking.md) — the in-process cost table and CSV export (full internal records, unaffected by this wire contract).
- [Execution Graph Tracing](execution-graph-tracing.md) — the sibling `graph_spec` artifact and the trace event streams.
- [Per-node Usage Attribution](per-node-usage-attribution.md) — the *other* projection of the same usage events, which keeps `node_id` and folds calls per graph node.


## Per-node Usage Attribution

# Per-node Usage Attribution

A run produces two artifacts that describe the same thing from different angles: `pipe_output.graph_spec` (the shape of the run) and `pipe_output.tokens_usages` (what it cost). Both are built from **one** trace-event stream, and every usage event in that stream already names the graph node it belongs to. This page is about the projection that keeps that link — `GraphSpec.usage` and `NodeSpec.usage` — and about the three states a usage number can be in, which is the part consumers get wrong.

## Two projections of one event stream

Every inference call emits a `UsageReportEvent` carrying `node_id` (from `trace_context.parent_node_id`) and the full internal `AnyTokensUsage`. That single stream is read twice:

- **`UsageAggregator`** (`pipelex/tracing/usage_aggregator.py`) projects it into a flat list — one record per call, `node_id` dropped. That list becomes `pipe_output.tokens_usages`, trimmed to [`TokensUsageRecord`](tokens-usage-wire-records.md) at the client boundary, and feeds the console cost table and the CSV.
- **`usage_attribution`** (`pipelex/tracing/usage_attribution.py`) projects it into per-node totals — `node_id` kept, calls folded together. Those totals become `NodeSpec.usage` and `GraphSpec.usage`, assembled by `GraphSpecAssembler`.

The two projections are deliberately separate — `tokens_usages` is a shipped, `extra="forbid"` client contract, and a graph is not the place to re-litigate its record shape. What they are *not* allowed to do is disagree on arithmetic, so both compute every dollar through the same `compute_tokens_usage_cost` (`pipelex/cogt/usage/cost_registry.py`) and define a token total the same way `AggregatedCosts.total_nb_tokens` does.

```mermaid
flowchart TB
    CALL["inference call completes"]
    EVENT["UsageReportEvent<br/>{ node_id, tokens_usage }"]

    subgraph LOG["event log — one stream, ordered by (workflow_id, sequence)"]
        STREAM["PipeStart · PipeEndSuccess · Edge · UsageReport"]
    end

    subgraph ATTRIBUTED["attributed projection"]
        direction TB
        ASSEMBLER["GraphSpecAssembler<br/>node_id kept, calls folded"]
        GRAPH["NodeSpec.usage<br/>GraphSpec.usage"]
    end

    subgraph FLATTENED["flat projection"]
        direction TB
        AGGREGATOR["UsageAggregator<br/>node_id dropped, one record per call"]
        FLAT["PipeOutput.tokens_usages"]
        SINKS["cost table · CSV · API wire"]
    end

    CALL --> EVENT
    EVENT -- "the link exists here" --> STREAM
    STREAM --> ASSEMBLER
    STREAM --> AGGREGATOR
    ASSEMBLER --> GRAPH
    AGGREGATOR --> FLAT
    FLAT --> SINKS
```

## When attribution happens, and why it waits

`GraphSpecAssembler` folds usage events during pass 1, keyed by the `node_id` the event *named* — with no node lookup. Resolution against the real node set happens in pass 2. This is not an optimisation: a `UsageReportEvent` can legitimately be read **before** the `PipeStartEvent` of the node it names, because the two can come from different workers and the stream is ordered by `(workflow_id, sequence)`. "Is this node real?" is only answerable once every event has been seen.

Pass 2 then does three things: resolves each accumulated total onto its node (or into the unattributed bucket), rolls subtrees up the `parent_node_id` chain, and computes the run total.

## The three states of a usage number

This is the part that matters for anything rendering a graph. A missing number means one of three things, and they must not be conflated — `$0.00` on an unrated node is a lie, and a controller that reports nothing because it was read through the wrong half looks like a bug.

| State | How it is encoded | What a consumer may claim |
|---|---|---|
| **Not collected** | `usage is None` | nothing — no measurement was taken |
| **Ran no inference** | `usage` present, `inference_calls == 0` | nothing (a controller may still report its subtree) |
| **Unrated** | `cost is None` with `inference_calls > 0` | that calls were made; **no price** |
| **Partial** | `inference_calls > rated_inference_calls > 0` | a price marked as a lower bound: `≥ $0.0043` |
| **Rated** | `rated_inference_calls == inference_calls > 0` | the price: `$0.0043` |

Unrated is not an edge case: `compute_tokens_usage_cost` returns `None` whenever the model carries no rate table, and dry/mock runs hardcode an empty rate table — so **every dry-run graph is unrated**.

**A note on token counts, for anyone rendering them.** The counts are faithful to what the run reported, which is not the same as being meaningful to a reader. Two cases to handle before putting a number on screen: a dry run reports one synthetic call per would-be inference with zero tokens (or, under `--mock-usage`, with *invented* non-zero tokens — never display those, they measure nothing), and a `PipeSearch` reports its provider's billing unit in the token field, so a single search shows ~2,000,000 "tokens" for a correct price of $0.005. The `cost` is the number that survives both; `mthds-ui` accordingly displays cost only, and only for a real run.

## The invariants

`NodeUsageSpec` (`pipelex/graph/graphspec.py`) states four invariants in its docstring. They are the contract; each has a named test in `tests/unit/pipelex/tracing/test_usage_attribution.py`.

1. **`usage is None` is all-or-nothing across a graph.** It means no usage was reported anywhere in the run — collection was off, or the run made zero inference calls. As soon as one usage event was seen, **every** node carries a spec, zeroed where nothing ran. A controller, a lifted pipe and a `PipeFunc` all get `inference_calls=0`, never `usage=None`. So the field never distinguishes "this node was not measured" from "that node was".
2. **`cost is None` ⟺ `rated_inference_calls == 0`.** Nothing else. "Made no call" and "made only unrated calls" both land here, and `inference_calls` tells them apart.
3. **`inference_calls > rated_inference_calls > 0` means `cost` is a lower bound**, not a total — some of this node's calls carried no rate table. A UI that renders it as a complete figure is silently wrong about money.
4. **`total_tokens` is input_joined + output**, the same definition as `AggregatedCosts.total_nb_tokens`. It is **not** the sum of `nb_tokens_by_category`: `input_cached` is a *subset* of `input`, not additive, so summing the dict double-counts. Never sum the dict; read `total_tokens`.

## The model that ran, vs the model that was asked for

A GraphSpec names a model in three places, and only one of them is the outcome:

| Rung | Where | Example |
|---|---|---|
| Authored choice | pipe blueprint `llm_choices.for_text` | `$writing-factual` (a preset) |
| Requested handle | `execution_data.resolved_model` | `@default-premium` (still an alias) |
| **What actually ran** | `NodeUsageSpec.by_model[].inference_model_name` | `claude-4.6-sonnet` |

`execution_data.resolved_model` is captured from `LLMSetting.model` (`pipe_llm.py`), which is a *handle*: alias resolution happens later, at inference time. The name overstates what it holds — roughly a third of the nodes in a typical corpus carry an unresolved `@alias` there. Only `by_model` survives alias resolution, deck defaults, and any fallback or retry that landed somewhere other than what was requested.

**`by_model` is a list because one node routinely uses more than one model.** A `PipeLLM`'s text pass and its object-structuring pass resolve separately (that is what `resolved_model_for_object` is about), and a retry can land elsewhere again. Collapsing them into "the model" would be wrong in exactly the way a single `cost` across mixed rated and unrated calls is wrong. Entries are ordered most-used first, ties broken by name, so a consumer can take `by_model[0]` as the dominant model without sorting. Per-model `cost` follows invariant 2: `None` iff no call to that model was priced.

The same breakdown rolls up: `subtree_by_model` tells you every model a controller's whole branch used.

## Own and subtree

Every node carries both halves: its own inference, and its own plus every descendant's (`subtree_*`). A controller (`PipeSequence`, `PipeBatch`, `PipeParallel`) runs no inference itself, so its own numbers are always zero and only the subtree half says anything — read a controller through its own half and it reports nothing at all. Both tokens and cost roll up, so a consumer that shows either one has the branch figure available without walking the graph.

The rollup is computed once, in the assembler, for the same reason `AggregatedCosts` computes its totals once: several consumers read the same GraphSpec, and they must not each re-derive it and disagree.

```text
  seq_node       calls 0 (0 rated)   subtree 3 calls (2 rated)  subtree_cost 0.0071  subtree_tokens 4210
    ├── llm_a    calls 1 (1 rated)   cost 0.0043   tokens 2130
    ├── llm_b    calls 1 (1 rated)   cost 0.0028   tokens 2080
    └── func_c   calls 0 (0 rated)   cost None     tokens 0      ← ran, made no inference
```

`func_c` has a spec present with `cost=None` and `inference_calls=0` — invariants 1 and 2 together say "collected, ran nothing", never "unknown".

`roll_up` tolerates two malformed-parentage cases rather than blowing up, because a partial cross-worker event read can produce either: a `parent_node_id` naming an absent node makes the child a root (never a `KeyError`), and a cycle in the parentage chain is broken by an in-progress guard and logged (never an infinite walk). The walk is memoized and uses an explicit stack, so it is O(nodes) and independent of the interpreter's recursion limit.

## The run total and the unattributed bucket

`GraphSpec.usage` is a `GraphUsageSpec` with two `NodeUsageSpec` fields:

- **`total`** — every usage the run reported, attributed or not. This is the graph's comparand for the cost report's own total.
- **`unattributed`** — the part that named no live node: the `UNATTRIBUTED_NODE_ID` (`"unknown"`) fallback stamped when an inference runs outside any pipe context, or a `node_id` whose `PipeStartEvent` never made it into the stream. Surfaced as its own bucket rather than dropped, so the graph's total can never silently disagree with the cost report's.

On a healthy run `unattributed` is zero. A non-zero one is a real signal worth looking at.

### Cross-checking against the cost report

The two are computed from the same events but encode "unrated" differently: `CostRegistry.aggregate_costs` treats an unrated usage as zero-cost, while `NodeUsageSpec.cost` is `None`. Comparing them naively manufactures a disagreement. The correct check is:

- `graph.usage.total.cost` equals the cost report's total **when `total.rated_inference_calls == total.inference_calls`**;
- when they differ, the graph total is a lower bound and the report's total is the wrong comparand — compare `rated_inference_calls` against the report's row count instead.

## The in-process tracer does not carry usage

There are two GraphSpec builders. `GraphSpecAssembler` produces every consumed spec. The in-process `GraphTracer.teardown()` output is discarded at all three of its call sites (`runner.py`, `pipe_run.py`, `dry_run_in_process.py` — each reads `pipe_output.graph_spec`, i.e. the assembler's), and `GraphTracerProtocol` has no method to receive usage at all.

So usage is deliberately assembler-only. That gap is *asserted* rather than normalized away: `tests/unit/pipelex/tracing/test_assembler_equivalence.py` runs a usage-bearing scenario through both builders, asserts they stay structurally equivalent, and asserts the divergence explicitly (assembler populated, tracer `None`). If someone later plumbs usage into the tracer, that test fails and tells them to delete the assertion — which is what a test documenting an intentional gap should do.

## Files reference

| File | Purpose |
|------|---------|
| `pipelex/graph/graphspec.py` | `NodeUsageSpec`, `GraphUsageSpec` and the four invariants |
| `pipelex/tracing/usage_attribution.py` | `UsageAccumulator`, `roll_up`, `attribute_usage` |
| `pipelex/tracing/graphspec_assembler.py` | Folds `UsageReportEvent`, attributes in pass 2 |
| `pipelex/tracing/usage_aggregator.py` | The other projection — the flat `tokens_usages` list |
| `pipelex/cogt/usage/cost_registry.py` | `compute_tokens_usage_cost` — the one cost engine |

## Related pages

- [Execution Graph Tracing](execution-graph-tracing.md) — the GraphSpec model and the trace event streams.
- [TokensUsage Wire Records](tokens-usage-wire-records.md) — the client-facing shape of the other projection.
- [Cost Tracking & Reporting](../features/cost-tracking.md) — the console table and CSV export.


## Image Handling in LLM Prompts

# Image Handling in LLM Prompts

This document describes how Pipelex handles images in PipeLLM prompts. The system implements a **prompt template-driven inclusion model** where images are sent to the LLM if and only if the prompt templates explicitly reference them.

Both `prompt` (user prompt) and `system_prompt` support image references using the same syntax.

---

## Design Principle

Images are included based on **what the prompt template references**, not on **what the input types contain**.

| Scenario | Images Sent? |
|----------|-------------|
| Input is `Image`, prompt template uses `@image` or `$image` | Yes |
| Input is `Page` with nested images, prompt template uses `@page` or `$page` | No |
| Input is `Page` with nested images, prompt template uses `{{ page \| with_images }}` | Yes |

This design prevents accidental image leakage and gives prompt template authors explicit control over what visual content reaches the LLM.

!!! info "Why Prompt Template-Driven?"
    Sending images to LLMs costs tokens and processing time. Prompt template-driven inclusion ensures you only pay for images you actually need the LLM to see.

---

## Three Reference Kinds

The system recognizes three distinct ways images can be referenced in prompt templates:

```
ImageReferenceKind
├── DIRECT       → Variable is ImageContent itself
├── DIRECT_LIST  → Variable is list[ImageContent] or Image[]
└── NESTED       → Variable is struct with nested images, using | with_images filter
```

### DIRECT References

When a prompt template variable directly points to an `Image` type:

```toml
[pipe.describe_photo]
inputs = { photo = "Image" }
prompt = "Describe this photo: @photo"
```

The image is automatically included. The `@photo` (or `$photo`) reference renders as `[Image 1]` in the prompt text.

### DIRECT_LIST References

When a prompt template variable points to an `Image[]` (list of images):

```toml
[pipe.analyze_gallery]
inputs = { photos = "Image[]" }
prompt = "Analyze these photos: $photos"
```

All images in the list are included. The `$photos` (or `@photos`) reference renders as:
```
[Image 1]
[Image 2]
[Image 3]
```

### NESTED References

When a struct contains images but isn't itself an image type, you must explicitly request image extraction:

```toml
[pipe.describe_document]
inputs = { doc = "Document" }
prompt = "{{ doc | with_images }}"
```

Without `| with_images`, only the text representation is sent. With it, nested images are extracted and included.

---

## System Prompt Support

Images can be referenced in both `system_prompt` and `prompt` using identical syntax:

```toml
[pipe.analyze_with_context]
inputs = { context_image = "Image", query_image = "Image" }
system_prompt = "You are analyzing images. Here is context: $context_image"
prompt = "Now analyze this image: $query_image"
```

### Global Numbering

When images appear in both prompts, they share a **global sequential numbering**:

1. **System prompt images are extracted first** - they get lower numbers (`[Image 1]`, `[Image 2]`, etc.)
2. **User prompt images are extracted second** - they continue the sequence (`[Image 3]`, `[Image 4]`, etc.)

This ensures consistent numbering across the entire prompt sent to the LLM.

### Example

```toml
[pipe.compare_styles]
inputs = { reference = "Image", subject = "Image" }
system_prompt = "Use this reference image for style comparison: $reference"
prompt = "Analyze the style of this image: $subject"
```

Results in:

- System prompt: `"Use this reference image for style comparison: [Image 1]"`
- User prompt: `"Analyze the style of this image: [Image 2]"`

Both images are sent to the LLM in order: `[Image 1]` (reference), `[Image 2]` (subject).

---

## The `| with_images` Filter

The `with_images` filter is the key mechanism for extracting images from complex structures.

### What It Does

1. Walks the structure recursively
2. Finds all `ImageContent` instances
3. Registers each image with a sequential number
4. Returns the text representation with `[Image N]` tokens inline

### Example Output

Given a `Page` with text and images:

```python
PageContent(
    text_and_images=TextAndImagesContent(
        text=TextContent(text="Welcome to the guide"),
        images=[ImageContent(url="...")]
    ),
    page_view=ImageContent(url="...")
)
```

The filter produces:

```
text_and_images: Welcome to the guide
[Image 1]
page_view: [Image 2]
```

### When to Use It

| Structure | Without Filter | With Filter |
|-----------|---------------|-------------|
| `Page` | Text only | Text + images |
| `Document` | Text only | Text + images |
| `list[Article]` | Text only | Text + all nested images |
| Custom struct with images | Text only | Text + images |

---

## Architecture

### Component Overview

```mermaid
flowchart TB
    subgraph FT["FACTORY TIME"]
        direction TB
        BP["PipeLLMBlueprint"]
        TA["TemplateImageAnalyzer"]
        IR["ImageReference[]"]

        BP -->|"template + inputs"| TA
        TA -->|"analyzes"| IR
    end

    subgraph RT["RUNTIME"]
        direction TB
        WM["Working Memory"]
        REG["ImageRegistry"]
        FLT["with_images filter"]
        LP["LLMPrompt"]

        WM -->|"values"| FLT
        FLT -->|"registers"| REG
        REG -->|"images"| LP
    end

    FT -->|"image_references"| RT
```

### Factory Time: Prompt Template Analysis

When a PipeLLM is created from a blueprint, the `TemplateImageAnalyzer` examines both `prompt` and `system_prompt` templates:

1. **Parse prompt template AST** - Extract all variable references with their filters
2. **Resolve types** - Look up each variable's type from input specifications
3. **Determine reference kind** - Based on type and filters applied
4. **Pre-compute nested paths** - For NESTED references, identify where images live in the structure

```python
# Stored in PipeLLM after analysis
user_image_references = [
    ImageReference(
        variable_path="page",
        kind=ImageReferenceKind.NESTED,
        nested_image_paths=["text_and_images.images", "page_view"]
    )
]
system_image_references = [
    ImageReference(
        variable_path="context_image",
        kind=ImageReferenceKind.DIRECT,
        nested_image_paths=None
    )
]
```

### Runtime: Image Collection

When the prompt is built:

1. **Create registry** - Fresh `ImageRegistry` for this prompt
2. **Extract system prompt images first** - Direct and list references from `system_prompt` are processed first, getting lower numbers
3. **Extract user prompt images second** - Direct and list references from `prompt` continue the numbering sequence
4. **Inject registry into context** - Registry available to Jinja2 filters for nested image extraction
5. **Render both templates** - `with_images` filter populates registry during rendering
6. **Collect images** - Retrieve all registered images after rendering
7. **Build prompt** - Both texts have tokens, images in separate list

---

## Data Flow

```mermaid
flowchart TB
    subgraph FT["FACTORY TIME"]
        direction TB
        PT[/"PipeLLM Blueprint"/]
        TA["TemplateImageAnalyzer"]
        IR[("ImageReference[]")]

        PT -->|"prompt + inputs"| TA
        TA -->|"analyzes"| IR
    end

    subgraph RT["RUNTIME"]
        direction TB
        WM[("Working Memory")]
        REG["ImageRegistry"]
        RENDER["with_images filter"]
        LP[/"LLMPrompt"/]

        WM -->|"values"| RENDER
        RENDER -->|"registers"| REG
        REG -->|"images"| LP
    end

    IR -->|"image_references"| RT
```

**Factory Time**: The `TemplateImageAnalyzer` parses both `system_prompt` and `prompt` templates, finds variables with image types or the `| with_images` filter, looks up their types, and pre-computes nested image paths.

**Runtime**: System prompt images are extracted first, then user prompt images, ensuring global sequential numbering. Values with nested images are passed through the `with_images` filter, which registers images to the `ImageRegistry` and returns text with `[Image N]` tokens. The final `LLMPrompt` contains both texts and the collected images.

---

## Image Registry

The `ImageRegistry` manages image numbering during prompt construction.

### Key Properties

- **0-based indices, 1-based display** - `register_image` returns a 0-based index; the 1-based `[Image N]` display number is produced at render time
- **Sequential** - Images numbered in order of registration
- **Deduplicated** - Same URL gets same index

```python
class ImageRegistry:
    def register_image(self, image: Any) -> int:
        """Register an image and return its 0-based index. Same URL = same index."""
        if image.url in self._image_urls:
            # Find existing index
            for index, existing in enumerate(self._images):
                if existing.url == image.url:
                    return index
        self._images.append(image)
        self._image_urls.add(image.url)
        return len(self._images) - 1
```

### Deduplication Example

If the same image appears in multiple places:

```python
# First registration
registry.register_image(img_a)  # Returns 0

# Second registration of same URL
registry.register_image(img_a)  # Returns 0 (not 1)

# Different image
registry.register_image(img_b)  # Returns 1
```

---

## Working with StuffArtefact

Values from working memory arrive wrapped in `StuffArtefact`, a thin delegation adapter that provides template-friendly access to content fields.

### Template Access

StuffArtefact delegates attribute access to the underlying content:

```python
# In template: {{ page.title }}
# StuffArtefact delegates to: page._stuff.content.title
```

### Filter Handling

The `with_images` filter uses the `ImageRenderable` protocol to handle StuffArtefact transparently:

```python
# StuffArtefact implements ImageRenderable
if isinstance(value, ImageRenderable):
    return value.render_with_images(registry=registry, text_format=text_format)

# StuffArtefact.render_with_images() delegates to content
def render_with_images(self, *, registry, text_format) -> str:
    return self._stuff.content.render_with_images(registry=registry, text_format=text_format)
```

!!! note "ImageRenderable Protocol"
    The `ImageRenderable` protocol uses `@runtime_checkable` to enable `isinstance()` checks without importing concrete types—avoiding circular imports between the Jinja2 layer and domain layer.

For detailed information on StuffArtefact's delegation pattern and the ImageRenderable protocol, see [:material-code-braces: StuffArtefact & Image Rendering](./stuffartefact-and-image-rendering.md).

---

## Validation

The system validates image usage at both factory time and runtime:

### Factory Time

| Condition | Error |
|-----------|-------|
| `\| with_images` on a type with no nested images (including a direct `Image`) | `WithImagesFilterError`: "Filter '\| with_images' used on variable 'X' but the type has no nested images. Remove the filter or use a type with nested images." |

### Runtime

| Condition | Error |
|-----------|-------|
| `with_images` on undefined value | "Cannot use with_images filter on undefined value" |
| `with_images` on non-ImageRenderable type (e.g., string) | "The with_images filter received a X which does not implement the ImageRenderable protocol. This filter requires StuffArtefact, StuffContent subclasses, or lists containing such types. Did you accidentally apply another filter first that converted to string?" |

The runtime check catches cases where filter chaining converts structured data to a string before `with_images` runs (e.g., `{{ pages | tag | with_images }}`).

---

## Prompt Template Syntax Reference

### Direct Image

```toml
inputs = { photo = "Image" }
prompt = "$photo"
```

### Image List

```toml
inputs = { gallery = "Image[]" }
prompt = "$gallery"
```

### Multiple Image Lists

```toml
inputs = { before = "Image[]", after = "Image[]" }
prompt = """
Before: $before

After: $after
"""
```

### Nested Images

```toml
inputs = { report = "Report" }
prompt = "{{ report | with_images }}"
```

### Mixed

```toml
inputs = { cover = "Image", pages = "Page[]" }
prompt = """
Cover: $cover

Pages:
{{ pages | with_images }}
"""
```

!!! warning "Filter Chaining: Order Matters"
    The `with_images` filter extracts images from structured data and returns a string with `[Image N]` tokens. The `tag` filter wraps its input in tags (```...``` or XML). **Order matters** when chaining these filters.

    **What works:**

    - ✅ `{{ pages | with_images }}` - extracts images with tokens
    - ✅ `{{ pages | tag }}` - formats text output (no images)
    - ✅ `{{ pages | with_images | tag }}` - extracts images, then wraps result in tags
    - ✅ `{{ pages | first | with_images }}` - non-terminal filter before `with_images` is fine

    **What doesn't work:**

    - ❌ `{{ pages | tag | with_images }}` - `tag` stringifies first, so `with_images` receives a string and can't extract images

    **Rule of thumb:** `with_images` must receive structured data to extract images. Place it before any filter that converts to string (like `tag`).

---

## Files Reference

### Core Implementation

| File | Purpose |
|------|---------|
| `pipelex/kernel/prompt_references.py` | `ImageReference` and `ImageReferenceKind` models |
| `pipelex/kernel/llm_prompt_content.py` | Run-time prompt assembly — image collection, registry, placeholders |
| `pipelex/pipe_operators/shared/template_image_analyzer.py` | Factory-time template analysis |
| `pipelex/tools/jinja2/image_registry.py` | Runtime image tracking |
| `pipelex/tools/jinja2/jinja2_with_images_filter.py` | The `with_images` filter implementation |

### Supporting Files

| File | Purpose |
|------|---------|
| `pipelex/tools/jinja2/jinja2_required_variables.py` | `VariableReference` for filter detection |
| `pipelex/pipe_operators/llm/llm_prompt_blueprint.py` | The parsed `.mthds` prompt, holding the image references and mapping down to the kernel |

---

## Next Steps

- [:material-book-open: Learn about PipeLLM](../building-methods/pipes/pipe-operators/PipeLLM.md){ .md-button }
- [:material-sitemap: Architecture Overview](./architecture-overview.md){ .md-button }


## Reasoning Controls

# Reasoning Controls

Pipelex provides a unified abstraction for controlling LLM reasoning (chain-of-thought / extended thinking) across providers. This page describes how reasoning parameters flow from user configuration through to provider-specific SDK calls.

---

## How to Use Reasoning

There are two ways to enable reasoning on a `PipeLLM` pipe: set `reasoning_effort` (a symbolic level) or `reasoning_budget` (an explicit token count). They are mutually exclusive — see [Mutual Exclusivity](#mutual-exclusivity) for details.

### Inline LLM Setting

Add reasoning directly in the `model` table of a pipe definition:

```toml
[pipe.analyze_contract]
type = "PipeLLM"
model = { model = "claude-4.5-sonnet", temperature = 0.1, reasoning_effort = "high" }
```

### LLM Preset

Define a reusable preset in your LLM deck, then reference it with the `$` prefix:

```toml
# In .pipelex/inference/deck/1_llm_deck.toml
[llm.presets]
deep-analysis = { model = "@default-premium", temperature = 0.1, reasoning_effort = "high" }
```

```toml
# In a .mthds file
[pipe.analyze_contract]
type = "PipeLLM"
model = "$deep-analysis"
```

### Using reasoning_budget

Instead of a symbolic effort level, you can specify an explicit token budget:

```toml
model = { model = "claude-4.5-sonnet", temperature = 0.1, reasoning_budget = 16384 }
```

`reasoning_budget` is supported by Anthropic and Google. OpenAI and Mistral raise `LLMCapabilityError`.

### Model-Specific Examples

Different models use different thinking modes under the hood. Pipelex handles the translation automatically — you always use `reasoning_effort` or `reasoning_budget`.

**Claude 4.6 Opus — adaptive mode**

The provider's SDK dynamically adjusts reasoning depth. `reasoning_effort` controls how aggressively it reasons:

```toml
# Adaptive: the SDK decides how many tokens to spend on reasoning
model = { model = "claude-4.6-opus", temperature = 0.1, reasoning_effort = "high" }
```

You can also override with an explicit budget, which forces `enabled` mode:

```toml
model = { model = "claude-4.6-opus", temperature = 0.1, reasoning_budget = 16384 }
```

**Gemini 2.5 Pro — manual mode**

Effort is translated to a `thinking_budget` token count:

```toml
# Manual: effort "medium" -> thinking_budget = 5000 tokens
model = { model = "gemini-2.5-pro", temperature = 0.3, reasoning_effort = "medium" }
```

**Gemini 3.1 Pro — adaptive mode**

Effort maps to a `ThinkingLevel` enum sent to the Google SDK:

```toml
# Adaptive: effort "high" -> ThinkingLevel.HIGH
model = { model = "gemini-3.1-pro", temperature = 0.3, reasoning_effort = "high" }
```

**GPT-5.2 — manual mode**

Effort maps directly to OpenAI's `reasoning_effort` parameter:

```toml
# Manual: effort "max" -> reasoning_effort = "xhigh" in the SDK call
model = { model = "gpt-5.2", temperature = 0.1, reasoning_effort = "max" }
```

!!! note "Structured Generation"
    Reasoning parameters are not supported for structured generation. See [Structured Generation](#structured-generation) for details.

!!! tip "Test Coverage"
    These examples are exercised in `tests/integration/pipelex/cogt/test_llm_reasoning.py`.

---

## Core Concepts

### ReasoningEffort

The `ReasoningEffort` enum (`pipelex/cogt/llm/llm_job_components.py`) defines the following levels:

| Level | Value | Description |
|-------|-------|-------------|
| `NONE` | `"none"` | Disable reasoning entirely |
| `MINIMAL` | `"minimal"` | Lowest reasoning effort |
| `LOW` | `"low"` | Light reasoning |
| `MEDIUM` | `"medium"` | Moderate reasoning |
| `HIGH` | `"high"` | Heavy reasoning |
| `XHIGH` | `"xhigh"` | Above `HIGH`, below `MAX`; maps to provider-specific xhigh value where supported (e.g. OpenAI) |
| `MAX` | `"max"` | Maximum reasoning budget |

### ThinkingMode

The `ThinkingMode` enum (`pipelex/cogt/llm/thinking_mode.py`) defines how a model handles reasoning at the SDK level:

| Mode | Meaning |
|------|---------|
| `none` | Model does not support reasoning. Attempting to use reasoning params raises `LLMCapabilityError`. |
| `manual` | Pipelex translates effort to a provider-specific value (token budget, effort string, or prompt mode). |
| `adaptive` | The provider's SDK dynamically adjusts reasoning depth. Only Anthropic and Google (Gemini 3) support this today. |

Each model spec in the backend TOML files declares a `thinking_mode`. This is a required field on `InferenceModelSpec` — models without reasoning capabilities set `thinking_mode = "none"` (or inherit it from `[defaults]`).

### Mutual Exclusivity

`reasoning_effort` and `reasoning_budget` are mutually exclusive. Both `LLMSetting` and `LLMJobParams` enforce this via a `model_validator`:

- **`reasoning_effort`** — A symbolic level (`NONE` through `MAX`). Pipelex resolves it to the provider-specific format.
- **`reasoning_budget`** — A raw token count passed directly to providers that accept it (Anthropic, Google). OpenAI and Mistral reject this with `LLMCapabilityError`.

---

## Data Flow

```mermaid
---
config:
  layout: dagre
  theme: base
---
flowchart TB
    A["LLMSetting<br>(MTHDS talent or API)"] -->|make_llm_job_params| B["LLMJobParams<br>reasoning_effort / reasoning_budget"]
    B --> C{Provider Worker}

    C -->|OpenAI Completions| D["_resolve_reasoning_effort()<br>-> effort string"]
    C -->|OpenAI Responses| D2["_resolve_reasoning()<br>-> Reasoning dict"]
    C -->|Anthropic| E["_build_thinking_params()<br>-> _ThinkingParams"]
    C -->|Google| F["_build_thinking_config()<br>-> ThinkingConfig"]
    C -->|Mistral| G["_resolve_prompt_mode()<br>-> prompt_mode"]
    C -->|Bedrock (aioboto3)| H["_validate_no_reasoning_params()<br>-> LLMCapabilityError if set"]
```

---

## Provider Mappings

Each provider has an `effort_to_level_map` configured in its provider subconfig within `pipelex.toml`. These maps translate `ReasoningEffort` values to provider-specific level strings. The special value `"disabled"` means reasoning should be skipped entirely (the accessor returns `None`).

### OpenAI (Completions & Responses)

OpenAI models use `thinking_mode = "manual"` and map `ReasoningEffort` to the `reasoning_effort` parameter via `openai_config.effort_to_level_map`:

```toml
[cogt.llm_config.openai_config.effort_to_level_map]
none = "none"
minimal = "minimal"
low = "low"
medium = "medium"
high = "high"
xhigh = "xhigh"
max = "xhigh"
```

| ReasoningEffort | OpenAI value |
|-----------------|-------------|
| `NONE` | `"none"` |
| `MINIMAL` | `"minimal"` |
| `LOW` | `"low"` |
| `MEDIUM` | `"medium"` |
| `HIGH` | `"high"` |
| `XHIGH` | `"xhigh"` |
| `MAX` | `"xhigh"` |

!!! note
    OpenAI's `"none"` is a valid API value (sent to the SDK), not disabled. This is different from the `"disabled"` convention used by other providers.

OpenAI does not support `reasoning_budget` or `thinking_mode = "adaptive"`. Both raise `LLMCapabilityError`.

When reasoning is active, `temperature` is omitted from the SDK call (OpenAI requires this).

### Anthropic

Anthropic supports both `manual` and `adaptive` thinking modes. The effort mapping is configured via `anthropic_config.effort_to_level_map`:

```toml
[cogt.llm_config.anthropic_config.effort_to_level_map]
none = "disabled"
minimal = "low"
low = "low"
medium = "medium"
high = "high"
xhigh = "xhigh"
max = "max"
```

| ReasoningEffort | Anthropic level |
|-----------------|----------------|
| `NONE` | `None` (thinking disabled) |
| `MINIMAL` | `"low"` |
| `LOW` | `"low"` |
| `MEDIUM` | `"medium"` |
| `HIGH` | `"high"` |
| `XHIGH` | `"xhigh"` |
| `MAX` | `"max"` |

Both modes first check `anthropic_config.effort_to_level_map` to gate reasoning. If the map returns `"disabled"` (e.g., for `NONE` effort), thinking is disabled entirely — no `thinking` parameter is sent to the SDK.

**ADAPTIVE mode** uses `{"type": "adaptive"}` with an `OutputConfigParam(effort=...)` where the effort value comes from the level map.

**MANUAL mode** resolves effort to a token budget via the `effort_to_budget_maps` config (keyed by `prompting_target`), then sends `{"type": "enabled", "budget_tokens": N}`. The budget is capped to `min(budget, max_tokens - 1)` to satisfy Anthropic's API constraint.

**`reasoning_budget`** (explicit) always uses `{"type": "enabled", "budget_tokens": N}` regardless of thinking mode. The same `min(budget, max_tokens - 1)` cap is applied.

!!! note
    `MINIMAL` and `LOW` both map to `"low"` in the level map. In ADAPTIVE mode they produce identical behavior. In MANUAL mode they are differentiated by the budget map (512 vs 1024 tokens). This matches the granularity that each mode supports.

When thinking is active, `temperature` is suppressed (Anthropic requires `temperature=1` or omission with thinking).

### Google Gemini

Google models use either `thinking_mode = "manual"` (Gemini 2.5 series) or `thinking_mode = "adaptive"` (Gemini 3 series). Both modes use `google_config.effort_to_level_map` as a gate:

```toml
[cogt.llm_config.google_config.effort_to_level_map]
none = "disabled"
minimal = "minimal"
low = "low"
medium = "medium"
high = "high"
xhigh = "high"
max = "high"
```

If the level map returns `"disabled"` (e.g., for `NONE` effort), thinking is disabled with `thinking_budget=0` regardless of mode.

!!! note
    `MAX` maps to `"high"` because Google's `ThinkingLevel` enum tops out at `HIGH` — there is no higher level.

**ADAPTIVE mode** (Gemini 3) sends a `thinking_level` value (e.g., `ThinkingLevel.LOW`, `ThinkingLevel.MEDIUM`, `ThinkingLevel.HIGH`) mapped from the `effort_to_level_map`. The Google SDK dynamically adjusts reasoning depth based on this level. No `thinking_budget` is set in adaptive mode.

!!! note
    `MINIMAL` and `LOW` map to distinct levels in the level map (`"minimal"` and `"low"`). In ADAPTIVE mode they produce different results — `ThinkingLevel.MINIMAL` vs `ThinkingLevel.LOW`. In MANUAL mode they are further differentiated by the budget map (512 vs 1024 tokens).

**MANUAL mode** (Gemini 2.5) resolves effort to a `thinking_budget` (token count) via the `effort_to_budget_maps` config:

| ReasoningEffort | thinking_budget |
|-----------------|----------------|
| `NONE` | `0` (disabled via level map) |
| `MINIMAL` | `512` |
| `LOW` | `1024` |
| `MEDIUM` | `5000` |
| `HIGH` | `16384` |
| `XHIGH` | `32768` |
| `MAX` | `65536` |

**`reasoning_budget`** (explicit) passes through directly as `thinking_budget`. When `max_tokens` is known, the budget is capped to `min(budget, max_tokens - 1)`.

!!! note
    An explicit `reasoning_budget` always produces a `thinking_budget`-based config, even when the model uses `thinking_mode = "adaptive"`. This overrides the `thinking_level` approach that adaptive mode normally uses.

Temperature is passed normally to the Google API regardless of reasoning mode.

!!! note "VertexAI Backend"
    Reasoning controls are not implemented for Gemini models on the VertexAI backend because Google favors the newer Gen-AI SDK. The VertexAI backend uses `sdk = "openai"` (the OpenAI-compatible endpoint), which routes through the OpenAI worker and does not expose Google's native thinking controls (`thinking_budget` / `thinking_level`). For Gemini reasoning support, use the `google` backend with the native Gen-AI SDK. Or better yet, use the Pipelex Gateway.

### Mistral

Mistral models use `thinking_mode = "manual"`. The effort mapping is configured via `mistral_config.effort_to_level_map`:

```toml
[cogt.llm_config.mistral_config.effort_to_level_map]
none = "disabled"
minimal = "reasoning"
low = "reasoning"
medium = "reasoning"
high = "reasoning"
xhigh = "reasoning"
max = "reasoning"
```

| ReasoningEffort | Mistral behavior |
|-----------------|-----------------|
| `NONE` | `prompt_mode` omitted (no reasoning) |
| `MINIMAL` through `MAX` | `prompt_mode = "reasoning"` |

Mistral does not support `reasoning_budget` or `thinking_mode = "adaptive"`. Both raise `LLMCapabilityError`.

Temperature is passed normally to the Mistral API regardless of reasoning mode.

### Bedrock (aioboto3 native models)

Bedrock native models using the `bedrock_aioboto3` SDK do not support reasoning parameters. Any `reasoning_effort` or `reasoning_budget` raises `LLMCapabilityError`.

!!! note
    Claude models accessed through Bedrock use the `bedrock_anthropic` SDK variant and go through the Anthropic worker, which does support reasoning.

### Gateway and Proxy Backends

Gateway and proxy backends (Azure OpenAI, Portkey, BlackBoxAI, Pipelex Gateway) route API calls through an intermediary but use the same provider worker classes as direct backends. Their reasoning capabilities depend on the `sdk` field in each model's backend TOML, which determines which worker handles the request.

- **Azure OpenAI** uses `sdk = "azure_openai_responses"`, routing through the OpenAI Responses worker. Reasoning models declare `thinking_mode = "manual"` and use OpenAI-style `reasoning_effort`.
- **Portkey** uses `portkey_completions` or `portkey_responses` SDKs, both routing through OpenAI workers. All models — including Anthropic and Google models proxied via Portkey — follow OpenAI reasoning semantics.
- **BlackBoxAI** uses `sdk = "openai"` or `"openai_responses"`. Proxied models follow OpenAI reasoning semantics.

!!! note
    When a provider's models are accessed through a gateway using an OpenAI-compatible SDK, the reasoning controls follow OpenAI semantics (`reasoning_effort`) rather than the provider's native semantics. For native reasoning controls (e.g., Anthropic thinking budgets, Google thinking levels), use the direct provider backend.

---

## Effort-to-Level Configuration

Each provider has an `effort_to_level_map` in its subconfig within `pipelex.toml` that maps `ReasoningEffort` values to provider-specific level strings. Every `ReasoningEffort` key must be present in each map (enforced by a validator).

The special value `"disabled"` causes the accessor to return `None`, signaling that reasoning should be skipped. OpenAI uses `"none"` as a valid API value instead (not `"disabled"`).

The level is resolved at runtime via `<ProviderConfig>.get_reasoning_level()` in each plugin's config module (e.g., `pipelex/providers/openai/openai_config.py`). Each config class returns the provider's native SDK type.

## Effort-to-Budget Configuration

For providers that use token budgets (Anthropic MANUAL, Google MANUAL), `ReasoningEffort` is resolved to a token count via the `effort_to_budget_maps` in `pipelex.toml`:

```toml
[cogt.llm_config.effort_to_budget_maps.anthropic]
none = 0
minimal = 512
low = 1024
medium = 5000
high = 16384
xhigh = 32768
max = 65536

[cogt.llm_config.effort_to_budget_maps.gemini]
none = 0
minimal = 512
low = 1024
medium = 5000
high = 16384
xhigh = 32768
max = 65536
```

The map is keyed by `prompting_target` (from the model spec). A validated mapping must contain entries for all `ReasoningEffort` values (including `none`, even though it is unreachable at runtime — the level map gates `NONE` as disabled before the budget lookup).

The budget is resolved at runtime via `LLMConfig.get_reasoning_budget()` (`pipelex/cogt/config_cogt.py`).

---

## Backend TOML Configuration

Each model declares its reasoning capability via `thinking_mode` in the backend TOML:

```toml
# Model that supports reasoning
[claude-4-sonnet]
thinking_mode = "manual"

# Model with adaptive reasoning
["claude-4.6-opus"]
thinking_mode = "adaptive"

# Google Gemini 3 with adaptive reasoning
["gemini-3.1-pro"]
thinking_mode = "adaptive"

# Model without reasoning (or inherited from defaults)
[gpt-4o-mini]
thinking_mode = "none"
```

Backends that have no reasoning-capable models set a default:

```toml
[defaults]
thinking_mode = "none"
```

---

## Structured Generation

Reasoning parameters (`reasoning_effort` and `reasoning_budget`) are not supported for structured generation (`_gen_object`). If either parameter is set when calling structured generation, an `LLMCapabilityError` is raised with the message "does not support reasoning parameters for structured generation".

This applies to all providers (OpenAI, Anthropic, Google, Mistral). Bedrock native models already reject all reasoning parameters before reaching `_gen_object`.

## NONE Semantics

The behavior of `ReasoningEffort.NONE` varies by provider:

- **OpenAI**: Sends `reasoning_effort="none"` to the API, which is a valid API value that minimizes reasoning.
- **Anthropic**: Disabled via `effort_to_level_map` gate — no `thinking` parameter is sent.
- **Google**: Disabled via `effort_to_level_map` gate, sets `thinking_budget=0`.
- **Mistral**: Omits `prompt_mode` (no reasoning).

---

## Error Handling

All reasoning-related errors use `LLMCapabilityError` (`pipelex/cogt/exceptions.py`):

| Scenario | Error |
|----------|-------|
| `reasoning_effort` on a `thinking_mode = "none"` model | "does not support reasoning" |
| `reasoning_budget` on a provider that doesn't support it | "does not support reasoning_budget" |
| `thinking_mode = "adaptive"` on OpenAI or Mistral | "adaptive ... not supported" |
| Any reasoning param on Bedrock (aioboto3) models | "does not support reasoning parameters" |
| Reasoning params during structured generation | "does not support reasoning parameters for structured generation" |
| Both `reasoning_effort` and `reasoning_budget` set | `ValueError` / `LLMSettingValueError` (mutual exclusivity) |

---

## File Reference

| File | Purpose |
|------|---------|
| `pipelex/cogt/llm/llm_job_components.py` | `ReasoningEffort` enum, `LLMJobParams` with mutual exclusivity validator |
| `pipelex/cogt/llm/thinking_mode.py` | `ThinkingMode` enum |
| `pipelex/cogt/llm/reasoning_config_base.py` | Shared helpers: `EffortToLevelMap`, `validate_effort_to_level_map()`, `get_reasoning_level_str()` |
| `pipelex/cogt/llm/llm_setting.py` | `LLMSetting` with reasoning fields and `make_llm_job_params()` |
| `pipelex/cogt/config_cogt.py` | `LLMConfig` with `get_reasoning_budget()` and effort-to-budget map validation |
| `pipelex/providers/openai/openai_config.py` | `OpenAIConfig` with `get_reasoning_level()` returning `ChatCompletionReasoningEffort \| None` |
| `pipelex/providers/anthropic/anthropic_config.py` | `AnthropicConfig` with `get_reasoning_level()` returning `AnthropicEffortLevel \| None` |
| `pipelex/providers/google/google_config.py` | `GoogleConfig` with `get_reasoning_level()` returning `genai_types.ThinkingLevel \| None` |
| `pipelex/providers/mistral/mistral_config.py` | `MistralConfig` with `get_reasoning_level()` returning `MistralPromptMode \| None` |
| `pipelex/cogt/model_backends/model_spec.py` | `InferenceModelSpec.thinking_mode` field |
| `pipelex/providers/openai/openai_completions_llm_worker.py` | OpenAI Completions reasoning resolution |
| `pipelex/providers/openai/openai_responses_llm_worker.py` | OpenAI Responses reasoning resolution |
| `pipelex/providers/anthropic/anthropic_llm_worker.py` | Anthropic thinking params builder |
| `pipelex/providers/google/google_llm_worker.py` | Google thinking config builder |
| `pipelex/providers/mistral/mistral_llm_worker.py` | Mistral prompt mode resolution |
| `pipelex/providers/bedrock/bedrock_llm_worker.py` | Bedrock reasoning validation |
| `pipelex/pipelex.toml` | Default effort-to-budget maps and effort-to-level maps |

---

## Next Steps

- [Architecture Overview](./architecture-overview.md) — Understand the two-layer design
- [Test Profile Configuration](./test-profile-configuration.md) — Configure model sets for testing


## StuffArtefact & Image Rendering

# StuffArtefact & Image Rendering

StuffArtefact is a thin delegation adapter that wraps `Stuff` objects for Jinja2 template access. The `ImageRenderable` protocol enables image extraction from nested content without circular imports between the template layer and domain layer.

---

## Design Principle

Two problems, one pattern:

1. **Template Access**: Templates need to access content fields via `{{ my_stuff.field_name }}` without knowing Pydantic internals
2. **Image Extraction**: The `with_images` filter needs to traverse content hierarchies without importing concrete types

**Solution**: StuffArtefact delegates attribute access to underlying content. ImageRenderable uses `@runtime_checkable` Protocol for duck-typed image extraction.

```
StuffArtefact wraps Stuff → delegates to StuffContent → Protocol enables traversal
```

!!! info "Protocol over Inheritance"
    ImageRenderable uses duck typing. Any class with a `render_with_images()` method satisfies the protocol—no base class required.

---

## Template Access Patterns

| Pattern | Result |
|---------|--------|
| `{{ doc.title }}` | Content field value |
| `{{ doc.pages }}` | Nested content (list, struct, etc.) |
| `{{ doc._stuff_name }}` | Metadata: variable name |
| `{{ doc._content_class }}` | Metadata: content class name |
| `{{ doc._concept_code }}` | Metadata: concept code |
| `{{ doc \| with_images }}` | Text with `[Image N]` tokens |
| `{{ doc \| tag }}` | Tagged output (no images) |

---

## Image Extraction Behavior

| Content Type | `render_with_images()` Behavior |
|-------------|--------------------------------|
| `ImageContent` | Self-registers, returns `[Image N]` |
| `ListContent` | Iterates items, recurses into nested |
| `TextAndImagesContent` | Renders text first, then images |
| `StructuredContent` | Iterates model fields, recurses into nested lists/tuples/dicts |
| `StuffArtefact` | Delegates to underlying content |

---

## Architecture

```mermaid
flowchart TB
    subgraph TEMPLATE["Jinja2 Template"]
        VAR["{{ doc | with_images }}"]
    end

    subgraph FILTER["with_images Filter"]
        direction TB
        CHECK["isinstance(value, ImageRenderable)?"]
        CALL["value.render_with_images(registry=registry, text_format=text_format)"]
        CHECK --> CALL
    end

    subgraph ARTEFACT["StuffArtefact"]
        DELEGATE["Delegates to _stuff.content"]
    end

    subgraph CONTENT["Content Implementations"]
        direction TB
        IMG["ImageContent: register + return token"]
        LIST["ListContent: iterate items"]
        TEXT["TextAndImagesContent: text then images"]
        STRUCT["StructuredContent: iterate model_fields"]
    end

    subgraph REGISTRY["ImageRegistry"]
        direction TB
        STORE["Append to list (0-based)"]
        DEDUP["Skip if URL exists"]
    end

    VAR --> CHECK
    CALL --> DELEGATE
    DELEGATE --> IMG & LIST & TEXT & STRUCT
    IMG --> STORE
    STORE --> DEDUP
```

---

## StuffArtefact Implementation

### Attribute Access Priority

StuffArtefact uses `__getattribute__` to intercept all attribute access:

```python
def __getattribute__(self, key: str) -> Any:
    # 1. Passthrough: _stuff, methods, magic attributes
    if key in _PASSTHROUGH_ATTRS or key.startswith("__"):
        return object.__getattribute__(self, key)

    # 2. Content fields (highest priority for templates)
    content_fields = type(content).model_fields
    if key in content_fields:
        return getattr(content, key)

    # 3. Metadata accessors
    match key:
        case "_stuff_name": return stuff.stuff_name
        case "_content_class": return content.__class__.__name__
        case "_concept_code": return stuff.concept.code
        case "_stuff_code": return stuff.stuff_code
        case "_content": return content

    # 4. Fallback to normal lookup
    return object.__getattribute__(self, key)
```

!!! warning "Content Field Priority"
    Content fields shadow StuffArtefact methods. If your content has a field named `items`, accessing `artefact.items` returns the field value, not the iteration method. Use `artefact.iter_items()` for explicit dict-like iteration.

### Dict-like Access

StuffArtefact supports bracket notation and iteration:

| Method | Purpose |
|--------|---------|
| `artefact["field"]` | Bracket access (via `__getitem__`) |
| `artefact.get("field", default=...)` | Safe access with default |
| `"field" in artefact` | Membership test |
| `artefact.iter_keys()` | Iterate field names |
| `artefact.iter_items()` | Iterate (key, value) pairs |
| `artefact.iter_values()` | Iterate values |

---

## ImageRenderable Protocol

### Definition

```python
from typing import Protocol, runtime_checkable

@runtime_checkable
class ImageRenderable(Protocol):
    """Protocol for types that can render with image extraction."""

    def render_with_images(
        self,
        *,
        registry: ImageRegistry,
        text_format: TextFormat,
    ) -> str:
        """Render to string, registering images to the registry.

        Returns:
            String with [Image N] tokens where images appear.
        """
        ...
```

### Why Protocol?

- **Avoids circular imports**: `tools/jinja2/` can check types from `core/stuffs/` without importing them
- **Duck typing**: No inheritance required—any matching method signature works
- **Runtime checking**: `isinstance(value, ImageRenderable)` works at runtime

---

## Content Implementations

### ImageContent

Self-registers and returns a token:

```python
def render_with_images(self, *, registry, text_format) -> str:
    image_index = registry.register_image(self)
    return f"[Image {image_index + 1}]"
```

### ListContent

Iterates items, delegating to nested ImageRenderable objects:

```python
def render_with_images(self, *, registry, text_format) -> str:
    parts: list[str] = []
    for item in self.items:
        if isinstance(item, ImageRenderable):
            rendered = item.render_with_images(registry=registry, text_format=text_format)
        else:
            rendered = item.rendered_markdown()
        if rendered:
            parts.append(rendered)
    return "\n".join(parts)
```

### TextAndImagesContent

Renders text first, then registers each image:

```python
def render_with_images(self, *, registry, text_format) -> str:
    parts: list[str] = []
    if self.text:
        parts.append(self.text.rendered_for_prompt(text_format=text_format))
    if self.images:
        for image in self.images:
            image_index = registry.register_image(image)
            parts.append(f"[Image {image_index + 1}]")
    return "\n".join(parts)
```

### StructuredContent

Default implementation for structured models iterates Pydantic model fields:

```python
def render_with_images(self, *, registry, text_format) -> str:
    parts: list[str] = []
    for field_name in type(self).model_fields:
        field_value = getattr(self, field_name)
        if field_value is None:
            continue
        rendered = self._render_value_with_images(field_value, registry=registry, text_format=text_format)
        if rendered:
            parts.append(f"{field_name}: {rendered}")
    return "\n".join(parts)
```

The `_render_value_with_images` helper recurses: a nested `ImageRenderable` gets `render_with_images(...)`, plain lists/tuples/dicts are traversed item by item, a non-renderable `StuffContent` falls back to `rendered_for_prompt(text_format=text_format)`, and anything else to `str(value)`. A plain `StuffContent` subclass that is not a `StructuredContent` does not implement the protocol.

---

## Image Numbering

### Registry Behavior

| Operation | Result |
|-----------|--------|
| First image registered | Returns index `0` |
| Second image registered | Returns index `1` |
| Same URL registered again | Returns existing index (deduplication) |
| Display in template | `[Image {index + 1}]` (1-based) |

### Example

```python
registry = ImageRegistry()

idx = registry.register_image(img_a)  # idx=0 → [Image 1]
idx = registry.register_image(img_b)  # idx=1 → [Image 2]
idx = registry.register_image(img_a)  # idx=0 → [Image 1] (same URL)

len(registry.images)  # 2 (deduplicated)
```

---

## The `with_images` Filter

### Execution Flow

```python
@pass_context
def with_images(context: Context, value: Any, _: Any = None) -> str:
    # 1. Validate input
    if isinstance(value, Undefined):
        raise Jinja2ContextError("Cannot use with_images on undefined")

    # 2. Get/create registry from context
    registry = context.get(Jinja2ContextKey.IMAGE_REGISTRY)
    if registry is None:
        registry = ImageRegistry()

    # 3. Get text format
    text_format = TextFormat(context.get(Jinja2ContextKey.TEXT_FORMAT, default=TextFormat.PLAIN))

    # 4. Protocol-based rendering
    if isinstance(value, ImageRenderable):
        return value.render_with_images(registry=registry, text_format=text_format)

    # 5. Handle plain sequences
    if isinstance(value, (list, tuple)):
        return _render_sequence_with_images(value, registry=registry, text_format=text_format)

    # 6. Reject unsupported types
    raise Jinja2ContextError(f"{type(value).__name__} does not implement ImageRenderable")
```

!!! warning "Filter Order Matters"
    `with_images` must receive structured data to extract images. Place it **before** any filter that converts to string.

    | Pattern | Works? |
    |---------|--------|
    | `{{ doc \| with_images }}` | Yes |
    | `{{ doc \| with_images \| tag }}` | Yes |
    | `{{ doc \| tag \| with_images }}` | No—`tag` stringifies first |

---

## Syntax Quick Reference

| Pattern | Purpose |
|---------|---------|
| `{{ x.field }}` | Access content field |
| `{{ x["field"] }}` | Bracket access |
| `{{ x._stuff_name }}` | Variable name metadata |
| `{{ x._content_class }}` | Content class name |
| `{{ x \| with_images }}` | Extract images as tokens |
| `{{ x \| with_images \| tag }}` | Extract then wrap in tags |

---

## Files Reference

| File | Purpose |
|------|---------|
| `pipelex/core/stuffs/stuff_artefact.py` | Thin delegation adapter for Jinja2 |
| `pipelex/tools/jinja2/image_renderable.py` | `@runtime_checkable` Protocol for images |
| `pipelex/tools/jinja2/tag_renderable.py` | `@runtime_checkable` Protocol for tagging |
| `pipelex/tools/jinja2/jinja2_with_images_filter.py` | `with_images` filter |
| `pipelex/tools/jinja2/jinja2_filters.py` | `tag` and `format` filters |
| `pipelex/tools/jinja2/image_registry.py` | Image tracking with deduplication |
| `pipelex/core/stuffs/structured_content.py` | Field-iterating `render_with_images()` for structured models |
| `pipelex/core/stuffs/image_content.py` | Self-registering image token |
| `pipelex/core/stuffs/list_content.py` | List iteration for images |
| `pipelex/core/stuffs/text_and_images_content.py` | Text + images rendering |

---

## Next Steps

- [:material-image-multiple: Image Handling in LLM Prompts](./image-handling-in-llm-prompts.md){ .md-button .md-button--primary }
- [:material-sitemap: Architecture Overview](./architecture-overview.md){ .md-button }


## Test Profile Configuration

# Test Profile Configuration

Pipelex uses a flexible test profile system to control which AI models are included in parametrized tests. This allows developers to quickly iterate with a small set of models during development, while still being able to run comprehensive tests across all available models.

---

## Overview

The test profile system consists of:

1. **Profile definitions** in `.pipelex-dev/test_profiles.toml`
2. **Model collections** organized by provider/type
3. **A preprocessing command** that generates fixture files
4. **Make targets** for easy regeneration

---

## Quick Start

```bash
# Regenerate test fixtures with default (dev) profile
make regenerate-test-models

# Regenerate with a specific profile
make regenerate-test-models TEST_PROFILE=full
make regenerate-test-models TEST_PROFILE=coverage
make regenerate-test-models TEST_PROFILE=ci
```

---

## Profile Configuration File

The configuration lives in `.pipelex-dev/test_profiles.toml`:

```toml
################################################################################
# Collections - Reusable lists of models/backends
################################################################################

[collections.backends]
all = ["pipelex_gateway", "anthropic", "openai", "google", ...]

[collections.llm]
anthropic = ["claude-3-haiku", "claude-4-opus", "claude-4.5-sonnet", ...]
openai = ["gpt-4o-mini", "gpt-4o", "gpt-5", ...]
google = ["gemini-2.5-flash", "gemini-2.5-pro", ...]

[collections.img_gen]
openai = ["gpt-image-1", "gpt-image-1-mini", "gpt-image-1.5"]
fal = ["flux-pro", "flux-2", "flux-2-pro"]

[collections.extract]
from_pdf = ["pypdfium2-extract-pdf", "docling-extract-text", "mistral-ocr", ...]
from_image = ["docling-extract-text", "mistral-ocr", "deepseek-ocr", ...]

################################################################################
# Profiles
################################################################################

[profiles.dev]
description = "Quick dev testing with fast and cheap models"
backends = ["pipelex_gateway", "internal"]
llm_models = ["claude-4.5-haiku", "gpt-4o-mini", "gemini-2.5-flash-lite"]
img_gen_models = ["gpt-image-1-mini"]
extract_models = ["@from_pdf"]

[profiles.full]
description = "All available models"
include_all = true
```

---

## Model List Syntax

The profile system supports several powerful syntax options for specifying models:

| Syntax | Description | Example |
|--------|-------------|---------|
| `"model-name"` | Specific model (direct reference) | `"gpt-4o-mini"` |
| `@collection` | All models from a named collection | `"@anthropic"`, `"@from_pdf"` |
| `backend:*` | All models from a specific backend | `"openai:*"` |
| `pattern*` | Glob pattern matching | `"claude-4*"`, `"gpt-*"` |
| `*` | All available models | `"*"` |
| `!pattern` | Exclude models matching pattern | `"!*-codex"` |

### Examples

```toml
# Use all Anthropic models from the collection
llm_models = ["@anthropic"]

# Use all Claude 4.5 models (glob pattern)
llm_models = ["claude-4.5-*"]

# Use all GPT models except codex variants
llm_models = ["gpt-*", "!*-codex", "!*-codex-*"]

# Combine collection reference with specific models
llm_models = ["@anthropic", "gpt-4o-mini", "gemini-2.5-flash"]

# Use all models from the openai backend
llm_models = ["openai:*"]

# Use all backends
backends = ["@all"]
```

---

## Built-in Profiles

### `dev` (default)

Quick development testing with fast, cheap models from key providers.

```toml
[profiles.dev]
backends = ["pipelex_gateway", "internal"]
llm_models = ["claude-4.5-haiku", "gpt-4o-mini", "gemini-2.5-flash-lite"]
img_gen_models = ["gpt-image-1-mini"]
extract_models = ["@from_pdf"]
```

### `ci`

Minimal set for CI pipelines - no actual inference.

```toml
[profiles.ci]
backends = ["azure_openai"]
llm_models = ["gpt-4o-mini"]
img_gen_models = []
extract_models = []
```

### `coverage`

One model per backend for coverage testing.

```toml
[profiles.coverage]
backends = ["anthropic", "openai", "google", "mistral", "internal"]
llm_models = ["claude-4.5-haiku", "gpt-4o-mini", "gemini-2.5-flash-lite", "mistral-large"]
img_gen_models = ["gpt-image-1", "nano-banana"]
extract_models = ["pypdfium2-extract-pdf"]
```

### `full`

All available models from all backends.

```toml
[profiles.full]
include_all = true
```

### `all_backends`

Test select models across all backends.

```toml
[profiles.all_backends]
backends = ["@all"]
llm_models = ["claude-4.5-haiku", "gpt-4o-mini", "gemini-2.5-flash-lite"]
img_gen_models = ["gpt-image-1-mini"]
extract_models = ["@from_pdf"]
```

---

## How It Works

### 1. Model Discovery

The preprocessing command discovers available models from:

- **Backend TOML files** in `inference/backends/*.toml`, resolved from the project `.pipelex/` directory if present there, otherwise from the global `~/.pipelex/` directory
- **Pipelex Gateway** remote configuration

### 2. Profile Resolution

When you run `make regenerate-test-models TEST_PROFILE=dev`:

1. Collections are loaded from `[collections.*]` sections
2. The specified profile is loaded from `[profiles.dev]`
3. Model specifiers are resolved:
    - `@collection_name` expands to the collection's models
    - `backend:*` expands to all models from that backend
    - Glob patterns (`claude-*`) are matched against known models
    - Exclusions (`!pattern`) are removed
4. Results are filtered by allowed backends

### 3. Fixture Generation

The resolved model/backend pairs are written to:

```
tests/integration/pipelex/fixtures/_generated_model_sets.py
```

This file contains:

```python
from tests.integration.pipelex.fixtures.model_combo import ModelCombo

LLM_COMBOS: list[ModelCombo] = [
    ModelCombo('claude-4.5-haiku', 'pipelex_gateway'),
    ModelCombo('gemini-2.5-flash-lite', 'pipelex_gateway'),
    ModelCombo('gpt-4o-mini', 'pipelex_gateway'),
]

IMG_GEN_COMBOS: list[ModelCombo] = [...]
EXTRACT_COMBOS: list[ModelCombo] = [...]
SEARCH_COMBOS: list[ModelCombo] = [...]
```

---

## CLI Command

For more control, use the CLI directly:

```bash
# Preview what would be generated
pipelex-dev preprocess-test-models --profile dev

# Generate fixtures
pipelex-dev preprocess-test-models --generate-fixtures --profile dev

# Output full availability JSON
pipelex-dev preprocess-test-models --output-json --profile full

# Quiet mode (minimal output)
pipelex-dev preprocess-test-models --generate-fixtures --profile dev --quiet
```

---

## Adding a New Profile

1. Add your profile to `.pipelex-dev/test_profiles.toml`:

    ```toml
    [profiles.my_profile]
    description = "My custom profile for X testing"
    backends = ["anthropic", "openai"]
    llm_models = ["claude-4.5-*", "gpt-4o-mini"]
    img_gen_models = []
    extract_models = []
    ```

2. Regenerate fixtures:

    ```bash
    make regenerate-test-models TEST_PROFILE=my_profile
    ```

---

## Adding Models to Collections

When new models become available, add them to the appropriate collection:

```toml
[collections.llm]
# Add new model to existing manufacturer collection
anthropic = [
    "claude-3-haiku",
    "claude-4.5-sonnet",
    "claude-5-opus",  # New model
]
```

Collections serve as documentation of available models and can be referenced in any profile.

---

## Local Overrides

For personal customizations that shouldn't be committed to the repository, create a `.pipelex-dev/test_profiles_override.toml` file. This file is gitignored and will be deep-merged on top of the base configuration.

This is useful for:

- Testing with specific models not in standard profiles
- Creating personal profiles tailored to your development workflow
- Temporarily modifying collections without affecting the team

### Example Override File

```toml
# .pipelex-dev/test_profiles_override.toml

# Add a personal profile
[profiles.my_local]
description = "My local testing profile"
backends = ["pipelex_gateway"]
llm_models = ["claude-4.5-sonnet", "gpt-4o"]
img_gen_models = []
extract_models = []

# Override an existing profile
[profiles.dev]
llm_models = ["claude-4.5-opus"]  # I prefer testing with opus locally

# Extend a collection
[collections.llm]
my_favorites = ["claude-4.5-sonnet", "gpt-4o", "gemini-2.5-pro"]
```

After creating or modifying your override file, regenerate fixtures:

```bash
make regenerate-test-models TEST_PROFILE=my_local
```

!!! tip "Team Collaboration"
    Since the override file is gitignored, each developer can maintain their own testing preferences without conflicts. The base `.pipelex-dev/test_profiles.toml` remains the shared source of truth for CI and team workflows.

---

## Best Practices

1. **Use `dev` profile during development** - Fast iteration with minimal API calls
2. **Use `coverage` profile for PR testing** - Ensures all backends work
3. **Use `full` profile sparingly** - Only when testing model-specific behavior
4. **Use glob patterns for model families** - `claude-4.5-*` catches all 4.5 variants

---

## Troubleshooting

### No models in profile

If a profile shows 0 models, check:

1. The specified models exist in the backend configurations
2. The backends list includes backends that have those models
3. Collection names are spelled correctly (case-sensitive)

### Empty list vs missing key

- `llm_models = []` - No LLM models included
- Key not present - All LLM models from allowed backends included

### Glob patterns not matching

Glob patterns use `fnmatch` syntax:

- `*` matches any characters
- `?` matches single character
- Patterns are case-sensitive


## Init CLI Flows

# Init CLI Flows

`pipelex init` sets up the `.pipelex/` configuration directory. It handles four independent concerns — config files, inference backends, routing profiles, and telemetry — through a focus-based dispatch system. Each concern owns its own file-copying and customization logic, so they can be run together or individually without interference.

---

## Why This Design

The `.pipelex/` directory contains two categories of files with different lifecycles:

1. **Config files** (`pipelex.toml`, `plxt.toml`) — static templates copied verbatim from the kit (no interactive customization); re-running init overwrites them (full reset).
2. **Inference files** (`inference/backends.toml`, `inference/routing_profiles.toml`, `inference/backends/*.toml`, `inference/deck/*.toml`) — interactive setup, customized per-project based on which AI backends the user selects.

These two categories are managed by separate steps. `init_config()` copies only config files (skipping the `inference/` directory entirely). The inference step handles its own template copying and then runs interactive backend selection and routing customization. Each file is owned by exactly one step — `init_config()` explicitly skips the `inference/` directory via `INIT_SKIP_DIRS`, and skips `telemetry.toml` and `pipelex_service.toml` via `INIT_SKIP_FILES`. This separation ensures that re-running `pipelex init config` never overwrites a user's carefully tuned inference setup.

---

## Interfaces

### CLI Commands

| Command | Focus | What It Does |
|---------|-------|--------------|
| `pipelex init` | `all` | Full setup: config + inference + routing + telemetry |
| `pipelex init config` | `config` | Copy config templates, then trigger inference if first-time |
| `pipelex init inference` | `inference` | Interactive backend selection + routing |
| `pipelex init routing` | `routing` | Routing profile customization only |
| `pipelex init telemetry` | `telemetry` | Telemetry config template copy |
| `pipelex init agreement` | `agreement` | Gateway terms acceptance (no reset) |
| `pipelex init credentials` | `credentials` | Credential setup for enabled backends |

All commands except `agreement` and `credentials` perform a **full reset** (overwrite existing files). Config updates are not yet supported.

### Inputs

- **`focus`** (`InitFocus` enum): Determines which steps run. Derived from the CLI subcommand.
- **`skip_confirmation`** (`bool`): When `True`, skips the interactive confirmation prompt. Used when called from `pipelex doctor --fix`.
- **`local`** (`bool`): When `True`, targets the project-level `.pipelex/` directory instead of the global `~/.pipelex/`. Maps to the `--local` CLI flag.

### Outputs / Side Effects

| Artifact | Produced By | Target Dir | Path |
|----------|-------------|------------|------|
| `pipelex.toml` | `init_config()` | Project or global | `.pipelex/pipelex.toml` |
| `plxt.toml` | `init_config()` | Project or global | `.pipelex/plxt.toml` |
| `backends.toml` | Inference step | Project or global | `.pipelex/inference/backends.toml` |
| `backends/*.toml` | Inference step | Project or global | `.pipelex/inference/backends/` |
| `deck/*.toml` | Inference step | Project or global | `.pipelex/inference/deck/` |
| `routing_profiles.toml` | Inference step | Project or global | `.pipelex/inference/routing_profiles.toml` |
| `telemetry.toml` | Telemetry step | Project or global | `.pipelex/telemetry.toml` |
| `.env` | Credentials step | **Always global** | `~/.pipelex/.env` (mode 0600) |
| `pipelex_service.toml` | Gateway terms acceptance | **Always global** | `~/.pipelex/pipelex_service.toml` |

!!! info "Project vs global"
    Most files are written to the target directory chosen at init time (project `.pipelex/` or global `~/.pipelex/`). The exceptions are `.env` (credentials) and `pipelex_service.toml`, which are **always** written to and read from `~/.pipelex/` — see [Gateway Terms: Always Global](#gateway-terms-always-global).

---

## Architecture

### Overall Flow

```mermaid
flowchart TD
    START([pipelex init]) --> FOCUS{focus?}

    FOCUS -- agreement --> AGREE[_init_agreement]
    AGREE --> DONE([Done])

    FOCUS -- credentials --> CREDS_DIRECT["prompt_credentials<br/>Prompt for missing API keys"]
    CREDS_DIRECT --> DONE

    FOCUS -- "all / config / inference / routing / telemetry" --> NEEDS[determine_needs]
    NEEDS --> CONFIRM{User confirms?}
    CONFIRM -- No --> CANCEL([Cancelled])
    CONFIRM -- Yes --> EXEC[execute_initialization]

    EXEC --> S1{needs_config?}
    S1 -- Yes --> INITCFG["init_config()<br/>Copies non-inference files"]
    INITCFG --> DETECT{backends.toml<br/>existed before?}
    DETECT -- "No (first time)" --> FORCE_INF[Force needs_inference = True]
    DETECT -- Yes --> CHK_INF{check_inference<br/>in focus?}
    CHK_INF -- Yes --> FORCE_INF
    CHK_INF -- No --> GW_CHK[Check gateway terms]

    S1 -- No --> S2

    FORCE_INF --> S2
    GW_CHK --> S2

    S2{needs_inference?}
    S2 -- Yes --> COPY_INF["Copy inference templates<br/>(backends, deck, routing)"]
    COPY_INF --> CUST_BE[customize_backends_config<br/>Interactive backend selection]
    CUST_BE --> CHK_RT{check_routing?}
    CHK_RT -- "No (auto-route)" --> CUST_RT[customize_routing_profile<br/>Auto-routing based on selection]
    CUST_RT --> S25
    CHK_RT -- "Yes (focus=routing)" --> S25

    S2 -- No --> S25

    S25{check_credentials?}
    S25 -- Yes --> CREDS["prompt_credentials<br/>Prompt for missing API keys"]
    S25 -- No --> S3
    CREDS --> S3

    S3{needs_routing?}
    S3 -- Yes --> ROUTE[customize_routing_profile<br/>Standalone routing setup]
    S3 -- No --> S4

    ROUTE --> S4

    S4{needs_telemetry?}
    S4 -- Yes --> TELEM[setup_telemetry<br/>Copy telemetry template]
    S4 -- No --> DONE2([Done])

    TELEM --> DONE2
```

---

## Implementation

### Determine Needs

`determine_needs()` evaluates the current state of `.pipelex/` to decide which steps are required:

```python
nb_missing_config_files = init_config(reset=False, dry_run=True) if check_config else 0
needs_config = check_config and (nb_missing_config_files > 0 or reset)
needs_inference = check_inference and (not path_exists(backends_toml_path) or reset)
needs_routing = check_routing and (not path_exists(routing_profiles_toml_path) or reset)
needs_telemetry = check_telemetry and (not path_exists(telemetry_config_path) or reset)
```

The `check_*` flags are derived from the `focus` parameter:

| Focus | `check_config` | `check_credentials` | `check_inference` | `check_routing` | `check_telemetry` |
|-------|:-:|:-:|:-:|:-:|:-:|
| `all` | Yes | Yes | Yes | No | Yes |
| `config` | Yes | Yes | No | No | No |
| `inference` | No | Yes | Yes | No | No |
| `routing` | No | No | No | Yes | No |
| `telemetry` | No | No | No | No | Yes |

!!! info "Routing is separate from inference"
    `check_routing` is only `True` for `focus=routing`. When `focus=all`, routing is handled automatically as part of the inference step (Step 2), not as a standalone step.

### Step 1: Config Step — `init_config()`

Copies the config template tree from `kit/configs/` to `.pipelex/`, with two skip mechanisms:

```python
INIT_SKIP_FILES: frozenset[str] = GIT_IGNORED_CONFIG_FILES | {TELEMETRY_CONFIG_FILE_NAME, ".DS_Store"}
INIT_SKIP_DIRS: frozenset[str] = frozenset({"inference"})
```

The recursive `copy_directory_structure` function checks both sets before processing each entry:

```python
if item in INIT_SKIP_FILES:
    continue
if os.path.isdir(src_item):
    if item in INIT_SKIP_DIRS:
        continue
    # recurse...
```

### First-Time Inference Detection

After `init_config()` runs, `execute_initialization` decides whether inference setup is needed — even if inference was not in the original focus:

```python
backends_existed_before = path_exists(backends_toml_path)
init_config(reset=reset)
backends_exists_now = path_exists(backends_toml_path)

if not backends_existed_before or (check_inference and backends_exists_now):
    needs_inference = True
```

This handles two scenarios:

| Condition | Meaning | Action |
|-----------|---------|--------|
| `not backends_existed_before` | First-time setup (no inference yet) | Force inference step regardless of focus |
| `check_inference and backends_exists_now` | Inference in focus + existing config | Re-run inference (reset) |

When inference is **not** forced and backends already exist, gateway terms are still checked:

```python
if not needs_inference and backends_existed_before:
    _check_gateway_terms_if_needed(console, backends_toml_path)
```

### Step 2: Inference Step

When `needs_inference` is `True` and `reset` is `True`, the inference step copies its own template files independently:

1. `backends.toml` — main backend registry
2. `backends/*.toml` — per-backend configuration files
3. `deck/*.toml` — model deck configurations
4. `routing_profiles.toml` — routing profile definitions

Then runs interactive customization:

1. `customize_backends_config()` — prompts user to select backends, handles gateway terms, and suggests IDE extension installation via `suggest_extension_install_if_needed()`
2. `customize_routing_profile()` — auto-configures routing based on selected backends (**only when `check_routing` is `False`**, i.e. when routing is not the specific focus)

When `focus=routing`, the inference step skips routing entirely because Step 3 handles it as a standalone operation.

### Step 2.5: Credentials Step

When `check_credentials` is `True` (for `focus` in `all`, `config`, `inference`), `prompt_credentials()` checks which backends are enabled and prompts the user for any missing API keys. Credentials are stored in `~/.pipelex/.env` with mode 0600.

### Step 3: Routing Step

If `needs_routing` is `True` (only for `focus=routing`), runs `customize_routing_profile()` as a standalone step.

### Step 4: Telemetry Step

Copies a telemetry template and prints instructions. No interactive prompts. Which template is copied depends on the target:

- **Global init** (target is `~/.pipelex/`): copies `pipelex/kit/configs/telemetry.toml` — the active template with disabled-by-default settings.
- **Project init** (target is `{project_root}/.pipelex/`): copies `pipelex/kit/configs/telemetry.project.toml` — every setting commented out so the project file does not shadow `~/.pipelex/telemetry.toml` or `~/.pipelex/telemetry_override.toml` during layered loading.

`setup_telemetry()` selects the template via a `for_project: bool` arg passed down from the `--local` flag on `pipelex init`.

---

## Scenario Matrix

| Scenario | Focus | Config Step | Inference Step | Credentials | Routing Step | Gateway Terms | Telemetry Step |
|----------|-------|:-----------:|:--------------:|:-----------:|:------------:|:-------------:|:--------------:|
| Fresh project, full init | `all` | Copies config files | Copies templates + interactive selection | Prompted | Auto (part of inference) | Via `customize_backends_config` | Copies template |
| Fresh project, config only | `config` | Copies config files | Forced (first-time detected) | Prompted | Auto (part of inference) | Via `customize_backends_config` | Skipped |
| Existing project, full re-init | `all` | Overwrites config files | Resets templates + interactive selection | Prompted | Auto (part of inference) | Via `customize_backends_config` | Overwrites template |
| Existing project, config only | `config` | Overwrites config files | Skipped (backends already exist) | Prompted | Skipped | `_check_gateway_terms_if_needed` | Skipped |
| Existing project, inference only | `inference` | Skipped | Resets templates + interactive selection | Prompted | Auto (part of inference) | Via `customize_backends_config` | Skipped |
| Existing project, routing only | `routing` | Skipped | Skipped | Skipped | Resets template + interactive selection | Skipped | Skipped |
| Existing project, credentials only | `credentials` | Skipped | Skipped | Prompted | Skipped | Skipped | Skipped |
| Gateway terms only | `agreement` | Skipped | Skipped | Skipped | Skipped | Direct acceptance | Skipped |

!!! warning "Config-Only on Existing Project"
    Running `pipelex init config` on a project that already has `inference/backends.toml` will overwrite config files (`pipelex.toml`, etc.) but will **not** touch the inference setup. Gateway terms are still checked via `_check_gateway_terms_if_needed`. The user's backend selection and routing are preserved.

---

## File Reference

### Template Directory (`kit/configs/`)

| File / Directory | Copied By | Purpose |
|-----------------|-----------|---------|
| `pipelex.toml` | `init_config()` | Main Pipelex configuration |
| `plxt.toml` | `init_config()` | PLXT tooling configuration |
| `inference/` | Inference step | All inference configuration (see below) |
| `inference/backends.toml` | Inference step | Backend registry (enabled/disabled flags) |
| `inference/backends/*.toml` | Inference step | Per-backend settings (API keys, endpoints) |
| `inference/deck/*.toml` | Inference step | Model deck definitions |
| `inference/routing_profiles.toml` | Inference step | Routing profile definitions |
| `telemetry.toml` | Telemetry step | Telemetry export configuration |
| `pipelex_service.toml` | Agreement step | Gateway service terms tracking |

### Skip Lists

| Constant | Contents | Reason |
|----------|----------|--------|
| `INIT_SKIP_FILES` | All `GIT_IGNORED_CONFIG_FILES` (`pipelex_service.toml`, `pipelex_override.toml`, `telemetry_override.toml`, `telemetry.project.toml`, `pipelex_gateway_models.md`, `pipelex_gateway_models_plain.md`, `x_custom_llm_deck.toml`, `x_custom_extract_deck.toml`) plus `telemetry.toml` and `.DS_Store` | Git-ignored, auto-generated, or managed by other steps |
| `INIT_SKIP_DIRS` | `inference` | Managed independently by inference step |

### Source Modules

| Module | Purpose |
|--------|---------|
| `pipelex/cli/commands/init/command.py` | Orchestration: `init_cmd()`, `execute_initialization()`, `determine_needs()` |
| `pipelex/cli/commands/init/config_files.py` | Config file copying: `init_config()`, skip lists |
| `pipelex/cli/commands/init/backends.py` | Backend customization: `customize_backends_config()`, `get_selected_backend_keys()` |
| `pipelex/cli/commands/init/routing.py` | Routing customization: `customize_routing_profile()` |
| `pipelex/cli/commands/init/telemetry.py` | Telemetry setup: `setup_telemetry()` |
| `pipelex/cli/commands/init/credentials.py` | Credential prompting: `prompt_credentials()`, `write_env_file()`, `read_env_file()` |
| `pipelex/cli/commands/init/ide_extension.py` | IDE extension suggestion: `suggest_extension_install_if_needed()` |
| `pipelex/cli/commands/init/ui/types.py` | `InitFocus` enum definition |

---

## Config Directory Resolution

The `ConfigLoader` (singleton: `config_manager`) resolves where configuration files live. Understanding this logic is essential because bugs in directory resolution cause files to be written or read from the wrong location.

**Source:** `pipelex/system/configuration/config_loader.py`

### Three Directory Properties

| Property | Path | When it exists |
|----------|------|----------------|
| `global_config_dir` | `~/.pipelex/` | Always (created on first use) |
| `project_config_dir` | `{project_root}/.pipelex/` | Only if the directory exists on disk |
| `pipelex_config_dir` | Project if exists, else global | Always (backward-compat alias) |

### Project Root Detection

`find_project_root()` walks up from `cwd` looking for marker files:

- `.pipelex`, `.git`, `pyproject.toml`, `setup.py`, `setup.cfg`, `package.json`, `.hg`

A directory containing only a `.pipelex/` directory is therefore itself a valid project root.

The home directory (`~`) is explicitly excluded — even if it contains a `package.json`, it is never treated as a project root.

### File Resolution Order

`resolve_config_file(relative_path, config_dir)` uses layered resolution:

```mermaid
flowchart TD
    START([resolve_config_file]) --> EXPLICIT{config_dir<br/>provided?}
    EXPLICIT -- Yes --> USE_EXPLICIT["Return config_dir / relative_path"]
    EXPLICIT -- No --> PROJECT{project_config_dir<br/>exists?}
    PROJECT -- Yes --> FILE_EXISTS{file exists in<br/>project dir?}
    FILE_EXISTS -- Yes --> USE_PROJECT["Return project_dir / relative_path"]
    FILE_EXISTS -- No --> USE_GLOBAL["Return global_dir / relative_path"]
    PROJECT -- No --> USE_GLOBAL
```

This means a project-level file **wins** over the global one, but only if it actually exists on disk.

### Global Config Bootstrap

`ensure_global_config_exists()` creates `~/.pipelex/` with kit template files on first use (called automatically during `load_config()`). It skips all `GIT_IGNORED_CONFIG_FILES` (files like `pipelex_service.toml`) and `.DS_Store` — these are never part of the bootstrap copy.

### Config Loading Chain

`load_config()` merges configuration in this order (later values override earlier ones):

1. Package defaults (`pipelex/pipelex.toml`)
2. Global config (`~/.pipelex/pipelex.toml`)
3. Project config (`{project_root}/.pipelex/pipelex.toml`, if different from global)
4. Override files from the effective config dir (local, environment, run mode, override)

---

## Gateway Terms: Always Global

`pipelex_service.toml` is **always** written to and read from `~/.pipelex/`, never from a project directory. This is an intentional invariant.

**Why:** Gateway terms are a user-level agreement, not a per-project setting. Accepting terms once should apply everywhere.

### Write Locations

All writers explicitly target `config_manager.global_config_dir`:

| Context | Module | `mkdir` before write? |
|---------|--------|-----------------------|
| Interactive init (config step) | `init/command.py` → `_check_gateway_terms_if_needed()` | Yes |
| Interactive backends | `init/backends.py` → `customize_backends_config()` | Yes |
| Agreement step | `init/command.py` → `_init_agreement()` | Yes (in `update_service_terms_acceptance`) |
| Agent CLI | `agent_cli/commands/init_cmd.py` → `_configure_backends()` | Yes |

### Read Locations

| Context | Module | How |
|---------|--------|-----|
| Doctor | `doctor_cmd.py` → `check_models()` | `load_pipelex_service_config_if_exists(config_dir=global_config_dir)` |
| Runtime | `runtime_boot.py` setup | `load_pipelex_service_config_if_exists(config_dir=global_config_dir)` |

### Rules

- **`mkdir` before writing**: All write paths ensure the config directory exists before writing. Some callers call `global_config_dir.mkdir(parents=True, exist_ok=True)` explicitly, and `update_service_terms_acceptance()` itself ensures the directory exists, so every path is protected.
- **Git-ignored**: `pipelex_service.toml` is in `GIT_IGNORED_CONFIG_FILES`, so it is never copied by `init_config()` and never synced to a project directory.

**Source:** `pipelex/system/pipelex_service/pipelex_service_agreement.py`, `pipelex/system/pipelex_service/pipelex_service_config.py`

---

## Agent CLI Init

`pipelex-agent init` is the non-interactive counterpart to `pipelex init`. It runs the same steps but takes all inputs from a JSON config argument instead of interactive prompts.

**Source:** `pipelex/cli/agent_cli/commands/init_cmd.py`

### Target Directory Resolution

`_resolve_target_dir(global_)` determines where files are written:

| Flag | Target |
|------|--------|
| Default (no flag) | `{project_root}/.pipelex/` — project root detected via `find_project_root()` |
| `--global` / `-g` | `~/.pipelex/` |

If no project root is found and `--global` is not set, the command fails with an error.

### Config Input

`--config` / `-c` accepts a JSON string (inline or file path) with these optional fields:

| Field | Type | Purpose |
|-------|------|---------|
| `backends` | `list[str]` | Backend keys to enable (e.g. `"openai"`, `"anthropic"`, `"pipelex_gateway"`) |
| `primary_backend` | `str` | Required when 2+ backends selected and `pipelex_gateway` is not among them |
| `accept_gateway_terms` | `bool` | Sets gateway terms acceptance; when omitted, nothing is written and any existing acceptance state is left untouched |

### Flow

```mermaid
flowchart TD
    START([pipelex-agent init]) --> PARSE["Parse --config JSON"]
    PARSE --> RESOLVE["Resolve target dir<br/>(project or --global)"]
    RESOLVE --> STEP1["Step 1: init_config()<br/>Copy config templates (skips inference/)"]
    STEP1 --> STEP15["Step 1.5: Copy inference templates<br/>(backends.toml, backends/*, deck/*, routing_profiles.toml)"]
    STEP15 --> STEP16["Step 1.6: Copy telemetry template<br/>(global: active defaults; project: commented-out)"]
    STEP16 --> STEP2["Step 2: Configure backends<br/>Enable requested backends in backends.toml"]
    STEP2 --> GW{pipelex_gateway<br/>in backends?}
    GW -- Yes --> TERMS["Write gateway terms to ~/.pipelex/<br/>(always global; skipped if accept_gateway_terms omitted)"]
    GW -- No --> STEP3
    TERMS --> STEP3["Step 3: Configure routing<br/>Auto-derive routing profile"]
    STEP3 --> STEP4["Step 4: Mark inference setup completed<br/>(written to ~/.pipelex/)"]
    STEP4 --> STEP5["Step 5: Prime remote-config cache<br/>(no-op if gateway disabled or terms not accepted)"]
    STEP5 --> OUTPUT["Output result<br/>(Markdown, or JSON via --format json)"]
```

!!! note "No credentials step"
    The agent CLI does **not** configure credentials. Use `pipelex-agent doctor` to check credential health, or `pipelex init credentials` for interactive credential setup.

### Key Differences from Interactive Init

| Aspect | `pipelex init` | `pipelex-agent init` |
|--------|---------------|---------------------|
| Prompts | Interactive (Rich prompts) | None — all input from `--config` JSON |
| Gateway terms | Prompted interactively | From `accept_gateway_terms` field in config |
| Output | Rich console output | Markdown by default (`agent_success_formatted()` / `agent_error()`), structured JSON via `--format json` |
| Credentials | Prompted interactively | Not configured — use `pipelex-agent doctor` or `pipelex init credentials` |
| Focus dispatch | Supports individual focus (`config`, `inference`, etc.) | Runs config, inference, routing, telemetry (no credentials) |
| Target dir default | Global (`~/.pipelex/`), use `--local` for project | Project (`{project_root}/.pipelex/`), use `--global` for global |

---

## Doctor Fix: Config Dir Handling

`pipelex doctor` and `pipelex doctor --fix` use the config directory resolution to find and fix configuration issues wherever they live.

**Source:** `pipelex/cli/commands/doctor_cmd.py`

### How Each Check Resolves Files

| Check | Resolution Method | Details |
|-------|-------------------|---------|
| `check_config_files()` | `resolve_config_file()` | Layered resolution (project > global) |
| `check_telemetry_config()` | `resolve_config_file()` | Layered resolution (project > global) |
| `check_backend_credentials()` | `resolve_config_file()` | Finds `backends.toml` wherever it lives |
| `check_backend_files()` | `resolve_config_file()` | Finds `inference/backends/` wherever it lives |
| `check_models()` gateway terms | `global_config_dir` | **Always global** — reads from `~/.pipelex/` |

### Fix Targeting

When `--fix` replaces an outdated backend file, it derives the config directory from the resolved file path:

```python
# File lives at {config_dir}/inference/backends/{name}.toml
# Walk up 3 levels to get the config_dir
resolved_config_dir = Path(backend_file_report.file_path).parent.parent.parent
replace_backend_file(backend_name, config_dir=resolved_config_dir)
```

This ensures the fix targets the same directory where the issue was found — if the broken file was in the project `.pipelex/`, the replacement goes there too, not to `~/.pipelex/`.

---

## Next Steps

- [:material-cog: Configuration Internals](../contribute/configuration-defaults-and-overrides.md){ .md-button }
- [:material-sitemap: Architecture Overview](./architecture-overview.md){ .md-button }


## Pipe Routing & Execution

# Pipe Routing & Execution

Pipelex runs pipes either **direct** (single-process) or **distributed** on a host runtime. Both share the same pipe definitions, library loading, and controller logic. The key difference is *where* the pipe runs and how the PipeJob travels to get there. The distributed host runtime is a durable backend — Temporal-backed execution or a Mistral Workflows integration — delivered through the Pipelex platform (see [Distributed Execution](../distributed-execution/index.md)). Both reach pipe execution through the same framework-agnostic runtime bridge.

---

## Terminology

| Term | Meaning |
|------|---------|
| **Direct execution** | All pipes run in the same Python process. Library, class registry, and pipe resolution are shared in-memory. |
| **Distributed execution** | PipeJob is serialized and sent to a remote worker — a Pipelex Temporal worker, or a Mistral Workflows activity. The worker is a separate process (potentially on a different machine). |

---

!!! note "The runtime bridge"
    Both distributed paths reach pipe execution through the framework-agnostic runtime bridge. The open `pipelex/runtime_bridge/` keeps the boot, serialization, mode/delivery, and worker-side hydration (`primitives/hydration.py`) pieces; the cross-process **dispatch** primitives — classifying controller pipes as child workflows and leaf operators as activities (`pipe_classification`), flushing trace events (`trace_flush`), and delivering results (`delivery`) — were extracted out of open core, and both commercial host-runtime plugins get them from the closed side. Graph and token-usage assembly is not a bridge primitive — it lives in `pipelex/pipe_run/tracing_assembly.py` and runs the same way for local and distributed runs.

---

## Design Principle

Every pipe execution — regardless of mode — follows the same pattern:

1. **Setup**: `pipeline_run_setup()` loads the library, resolves the pipe, initializes working memory, and creates a `PipeJob`
2. **Route**: A router (`PipeRouter` for direct mode, or the booted orchestrator's distributed router) receives the PipeJob and dispatches it
3. **Execute**: The pipe's `run_pipe()` method runs, potentially resolving and executing child pipes

The `PipeJob` is the universal unit of execution. It carries everything needed to run a single pipe.

---

## The PipeJob Model

`PipeJob` encapsulates all information needed to execute a pipe.

| Field | Type | Purpose |
|-------|------|---------|
| `pipe` | `PipeAbstract` | The resolved pipe object (concrete operator or controller) |
| `working_memory` | `WorkingMemory \| None` | Runtime data store — typed `Stuff` objects keyed by variable name |
| `working_memory_raw` | `dict \| None` | Raw JSON dict of WorkingMemory for deferred hydration (Temporal only) |
| `pipe_run_params` | `PipeRunParams` | Execution config: run mode (LIVE/DRY), output multiplicity, pipe stack for cycle detection |
| `job_metadata` | `JobMetadata` | Pipeline run ID, user ID, OTel tracing context, graph tracing context |
| `output_name` | `str \| None` | Override for the output variable name |
| `library_crate` | `LibraryCrate \| None` | Serializable library snapshot for distributed execution |

`PipeJob` is created by `pipeline_run_setup()`, which handles library loading, pipe resolution, working memory initialization, and telemetry setup. For distributed dispatch, a transport-prep step outside open core moves `working_memory` to `working_memory_raw` (deferred hydration) and ensures the crate is attached.

!!! warning "Build `PipeRunParams` through its factory"
    `PipeRunParamsFactory.make_run_params()` is the single writer of the fields that carry a run-scoped invariant: `run_mode`, `pipe_stack_limit` and `batch_max_concurrency`. None of the three has a default — for each, a silently-defaulted value would point at the dangerous direction (real spend, an unbounded pipe stack, a fan-out that launches every branch at once), and for `batch_max_concurrency` the would-be default `None` is *also* the resolved value of an authored `max_concurrency = "unbounded"`, so "never written" and "authored unbounded" would be indistinguishable. Constructing `PipeRunParams` by hand and omitting any of them raises a `ValidationError` — at the constructor and at the wire boundary alike.

---

## Library Loading

Before a pipe can execute, the **library** must be loaded. The library contains:

- **Pipes** — all pipe definitions (operators and controllers), resolved by code via `get_required_pipe()`
- **Concepts** — semantic type definitions that determine what data a `Stuff` object holds
- **Domains** — namespaces that group related pipes and concepts

### Base vs Custom Libraries

- **Base libraries** are loaded from directories listed in `PIPELEXPATH`. They contain shared pipe/concept definitions available to all executions.
- **Custom bundles** are per-request `mthds_content` strings. Each API call can bring its own definitions.

### Dynamic Class Generation

When a `.mthds` file declares a concept like `RawText = "Raw input text..."`, the library loading process:

1. `ConceptFactory._handle_basic_blueprint()` detects the concept declaration
2. `StructureGenerator.generate_from_structure_blueprint()` dynamically creates a Python class (e.g., `RawText` inheriting from `TextContent`)
3. The class is registered with the active `ClassRegistry` (accessed via `hub.get_class_registry()`) so it can be serialized/deserialized

These dynamically-generated classes become the `content` type of `Stuff` objects in `WorkingMemory`.

!!! info "Why Dynamic Classes Matter"
    When `WorkingMemory` crosses to a Temporal worker, `dump_for_transport()` stamps content with pipelex-private `__pipelex_class__` / `__pipelex_module__` markers instead of Kajson's `__class__` / `__module__` — deliberately keeping Kajson's universal decoder out of the loop at the data-converter boundary. Pipelex's hydrator resolves those markers against the per-workflow `ClassRegistry`, the only place these dynamic concept classes are registered.

---

## Direct Execution

In direct execution, everything runs in a single Python process. This is the default mode when no orchestrator plugin is booted.

### Flow

```
pipeline_run_setup()
  ├── Load library (library_manager singleton)
  ├── Generate dynamic concept classes
  ├── Register classes with Kajson
  ├── Resolve pipe via get_required_pipe(pipe_code)
  ├── Initialize WorkingMemory from inputs
  └── Return PipeJob

PipeRouter.run(pipe_job)
  ├── Notify observers (before)
  ├── _run_pipe_job(pipe_job)
  │   └── pipe_job.pipe.run_pipe(...)     ← delegates directly to the pipe
  │       ├── Concrete pipe: execute operator logic (LLM call, template, etc.)
  │       └── Controller: resolve child pipes via get_required_pipe(),
  │           then recursively call child.run_pipe()
  ├── Notify observers (after)
  └── Return PipeOutput
```

### Router Selection

The router is selected during `Pipelex.setup()` by resolving the hub's `PIPE_ROUTER` slot. A boot-orchestrator plugin (e.g. our closed Temporal plugin) claims that slot in its `register()` when `plugins.boot_orchestrator` names it; otherwise the default in-process router is used:

```python
# A boot-orchestrator plugin swaps in its distributed router when
# plugins.boot_orchestrator == its own name; otherwise this default runs.
effective_pipe_router = self._resolve_hub_slot(
    slot=HubSlot.PIPE_ROUTER,
    default=lambda: PipeRouter(observer=multi_observer),   # Direct, in-process
)
```

### How PipeRouter Works

`PipeRouter` implements `PipeRouterProtocol` with a minimal `_run_pipe_job()`:

```python
async def _run_pipe_job(self, pipe_job: PipeJob) -> PipeOutput:
    return await pipe_job.pipe.run_pipe(
        job_metadata=pipe_job.job_metadata,
        working_memory=pipe_job.get_working_memory(),
        output_name=pipe_job.output_name,
        pipe_run_params=pipe_job.pipe_run_params,
        library_crate=pipe_job.library_crate,
    )
```

The router does not route by pipe type — it delegates to the pipe itself. Controllers handle their own orchestration internally.

### The batch-branch hook

`PipeRouterProtocol` carries one more entry point beside `run`:

```python
async def run_batch_branch(self, pipe_job: PipeJob) -> PipeOutput:
    return await self.run(pipe_job)
```

`PipeBatch` calls it instead of `run` for each per-item fan-out dispatch. That is the only place in the pipe tree where the *dispatch* carries semantics its `PipeJob` cannot express: a branch job holds the branch pipe and the item's memory, which is byte-for-byte the shape of any other dispatch. The hook is how a router learns "this one is a fan-out branch".

The default body IS the behavior for in-process routers — a branch is just a run, so `PipeRouter` deliberately does not override it and nothing about direct execution changed. A **distributed** router may override it to give each branch its own isolation (own retry, own history partition) while every other dispatch it receives runs inline. Because the default delegates to `run` and not to `_run_pipe_job`, batch branches still pass through the observer hooks.

---

## Pipe Controllers

Controllers are pipes that orchestrate the execution of other pipes. They resolve child pipes at runtime via `get_required_pipe()` from the library.

| Controller | Behavior |
|------------|----------|
| **PipeSequence** | Executes `sequential_sub_pipes` one after another. Each step receives working memory with outputs from previous steps. |
| **PipeBatch** | Iterates over a `ListContent` input. For each item, loads `branch_pipe` and executes it with a deep copy of working memory. Items run concurrently via asyncio. |
| **PipeCondition** | Evaluates a Jinja2 `expression` against working memory, maps the result via `outcome_map` to a pipe code, and executes that pipe. |
| **PipeParallel** | Loads multiple child pipes and executes them concurrently, each with its own working memory copy. |

All controllers follow the same pattern:

1. Call `get_required_pipe(child_pipe_code)` to resolve the child pipe from the library
2. Route through `get_pipe_router().run(PipeJob(...))` — the hub auto-selects the right router. `PipeBatch` is the one exception: its per-item branch dispatches go through [`run_batch_branch`](#the-batch-branch-hook) instead, so a distributed router can isolate them.
3. Aggregate results into working memory or output

### Auto-Switching Router

The hub (`get_pipe_router()`) returns the router for whichever orchestrator the process is booted under:

- **Direct execution**: the in-process `PipeRouter` — child pipes run in the same process.
- **Distributed execution**: the booted host-runtime orchestrator's router, which has claimed the hub's `PIPE_ROUTER` slot. That router auto-detects whether it is dispatching from the **submitter** (start a unit of durable work) or from **inside a running one** (continue within it, or spawn a nested unit).

How much of a controller tree a distributed backend spreads across separate durable units is that backend's call, not core's — core's contract is only that every dispatch reaches the router, and that batch branches are *labelled* as such via `run_batch_branch`. See the backend's own documentation for the topology it chooses.

!!! note "Library Dependency"
    Controllers depend on the library being loaded in the current process. `get_required_pipe()` queries the library scoped to the current run via `ContextVar`, which must have been populated by loading a `LibraryCrate`. In distributed execution, each worker-side job loads the crate from the `PipeJob` into a per-run `Library` instance before resolving child pipes. (Temporal-specific detail: the backend disables the Temporal sandbox via `--no-sandbox` because library loading is a side effect incompatible with replay semantics.) See [Runtime Bridge & Transport](./runtime-bridge-and-transport.md) for how the crate and working memory cross the boundary.

---

## Distributed Execution

In distributed execution, the PipeJob is serialized and sent to a host-runtime worker for execution. The concrete dispatch shape below is illustrated with Pipelex's [Temporal backend](https://pipelex.com/products#temporal); the boundary mechanics (LibraryCrate, deferred hydration, per-call isolation) are backend-neutral and documented in [Runtime Bridge & Transport](./runtime-bridge-and-transport.md).

### Flow

```mermaid
sequenceDiagram
    participant S as Submitter (API/CLI)
    participant T as Temporal Server
    participant W as Worker

    Note over S: pipeline_run_setup()
    S->>S: Load library, generate dynamic classes
    S->>S: Resolve pipe, build LibraryCrate
    S->>S: Create PipeJob (crate attached)

    Note over S: TemporalPipeRouter.run() (submitter side)
    S->>S: transport prep (closed)<br/>(WM → working_memory_raw)
    S->>T: Submit run workflow(PipeJob)
    T->>W: Dispatch workflow

    Note over W: run workflow
    W->>W: Create per-workflow ClassRegistry
    W->>W: Load LibraryCrate (register classes)
    W->>W: Hydrate working_memory_raw → WM
    W->>W: pipe.run_pipe()

    alt Concrete pipe
        W->>W: Execute via Activity
    else Dispatch the backend chooses to isolate
        W->>T: Child workflow (crate propagates)
        T->>W: Child result
    end

    W->>W: Dehydrate PipeOutput<br/>(WM → working_memory_raw)
    W->>T: Return dehydrated PipeOutput
    T->>S: Workflow result
    S->>S: Hydrate PipeOutput<br/>(working_memory_raw → WM)
```

### Key components (the backend's role)

A host-runtime orchestrator supplies the pieces that turn this generic shape into real workflows. The contract each implements is open core; the concrete classes are backend-specific. In Pipelex's Temporal backend:

- a **router** implementing `PipeRouterProtocol` (`TemporalPipeRouter`) that dispatches via Temporal instead of running in-process;
- a **worker-side job** (a Temporal workflow) that receives a deserialized `PipeJob`, loads the crate, hydrates working memory, and calls `pipe.run_pipe()` — exactly like the direct router;
- a **data converter** that uses Kajson to (de)serialize Pydantic models with subclass types preserved — critical because `PipeJob.pipe` is a `PipeAbstract` subclass and `WorkingMemory` holds `Stuff` objects with concept-specific content classes;
- a **worker entry point** that calls `Pipelex.make(boot_orchestrator="temporal")` and starts the worker with its registered workflows and activities (shipped as the plugin's own `worker` console command).

The concrete modules live in our closed Temporal plugin and are documented with it. The open contracts they satisfy — `PipeRouterProtocol`, the boundary DTOs, serialization — are listed in the [Orchestrator SPI](./orchestrator-plugins.md#the-orchestrator-spi).

### Content generation activities

Concrete pipe operators (PipeLLM, PipeCompose, PipeExtract, PipeImgGen) dispatch their actual leaf work as host-runtime activities, invoked from inside the worker-side job via the runtime's in-workflow content generator — no intermediate child workflow per content-generation call. A backend can route those activities to different task queues per activity (and, where applicable, per handle); in the Temporal backend that routing is configured under the plugin's own `[worker_config.activity_queues]` config (in the plugin's `temporal.toml`, not core config), falling back to the job's own task queue when unset. The concrete activity set and queue-routing reference live in our Temporal plugin's own docs; see also [Content Generation Across Boundaries](./distributed-content-generation.md) for the backend-neutral serialization mechanism.

!!! note "Future Improvement: Per-Pipe Routing"
    Router selection is currently global and binary: either all pipes go through the distributed backend, or none do. A simple `PipeCompose` (microseconds of Jinja2 rendering) gets the same dispatch overhead as a `PipeLLM` (minutes of API call time). Per-pipe or per-type routing decisions could improve efficiency.

---

## Crossing the boundary: crate, hydration, isolation

Distributed execution introduces three mechanisms that don't exist in direct mode, because the worker is a separate process that shares none of the submitter's library, class registry, or working-memory state. They are backend-neutral; for the full walkthrough see [Runtime Bridge & Transport](./runtime-bridge-and-transport.md). In short:

- **LibraryCrate propagation** — the submitter attaches a serializable snapshot of all pipes and concepts to the `PipeJob`; every worker-side job loads it (idempotently, by fingerprint) so `get_required_pipe()` resolves at every level.
- **Deferred hydration** — `WorkingMemory` crosses as a raw JSON dict (`working_memory_raw`) on both `PipeJob` inputs and `PipeOutput` return values, avoiding deserialization failures when a receiving process hasn't registered the bundle's dynamic concept classes. The transport-prep helpers are not part of open core; the raw field and the hydration helper are.
- **Per-call isolation** — each job gets its own `ClassRegistry` (pre-seeded from the global registry) and its own `Library`, scoped via `ContextVar`, so concurrent jobs with conflicting concept names (e.g., two bundles that both define `Result` with different fields) don't cross-contaminate.

!!! warning "Known backend limitation: StuffArtefact in dry-run"
    On a host runtime whose data converter serializes working memory for internal templating activities (Temporal's `act_jinja2_gen_text`, dispatched by PipeCondition and PipeCompose), dry-run mode can fail: working memory then holds `StuffArtefact` debug objects that are not JSON-serializable. PipeSequence and PipeParallel are unaffected — they dispatch child pipes without internal templating activities. This is a backend-specific concern; see our Temporal plugin's own docs for details and fix direction.

---

## Architecture Diagram

```mermaid
flowchart TB
    subgraph Setup["Pipeline Setup (API / CLI Process)"]
        S1["pipeline_run_setup()"]
        S2["Load library"]
        S3["Generate dynamic classes"]
        S4["Resolve pipe"]
        S5["Build LibraryCrate"]
        S6["Create PipeJob (with crate)"]
        S1 --> S2 --> S3 --> S4 --> S5 --> S6
    end

    S6 --> Decision{boot_orchestrator set?}

    subgraph Direct["Direct Execution"]
        D1["PipeRouter._run_pipe_job()"]
        D2["pipe.run_pipe()"]
        D3["Controller: get_required_pipe()"]
        D4["Recursive child execution"]
        D1 --> D2
        D2 --> D3
        D3 --> D4
    end

    subgraph Distributed["Distributed Execution (Temporal example)"]
        T1["Submitter router: transport prep (closed)"]
        T2["Kajson serialize PipeJob + crate"]
        T3["Host runtime (Temporal Server)"]
        T4["Worker: Kajson deserialize"]
        T5["Worker job: load crate, hydrate WM"]
        T6["pipe.run_pipe()"]
        T1 --> T2 --> T3 --> T4 --> T5 --> T6
    end

    Decision -- No --> D1
    Decision -- Yes --> T1

    D4 --> Result["PipeOutput"]
    T6 --> Result
```

---

## File Reference

| Component | File |
|-----------|------|
| PipeJob model | `pipelex/pipe_run/pipe_job.py` |
| Pipeline setup | `pipelex/pipeline/pipeline_run_setup.py` |
| PipeRouter (direct) | `pipelex/pipe_run/pipe_router.py` |
| PipeRouterProtocol | `pipelex/pipe_run/pipe_router_protocol.py` |
| Distributed router / worker job / data converter / worker CLI | backend-specific — in the host-runtime plugin (e.g. our Temporal plugin), not open core |
| Library manager | `pipelex/libraries/library_manager.py` |
| ConceptFactory | `pipelex/core/concepts/concept_factory.py` |
| StructureGenerator | `pipelex/core/concepts/structure_generation/generator.py` |
| LibraryCrate model | `pipelex/libraries/library_crate.py` |
| LibraryCrate factory | `pipelex/libraries/library_crate_factory.py` |
| Deferred hydration utility (open core) | `pipelex/runtime_bridge/primitives/hydration.py` |
| Interpreter hub (get_required_pipe) | `pipelex/interpreter_hub.py` |
| Runtime hub (get_class_registry) | `pipelex/runtime_hub.py` |
| PipeSequence | `pipelex/pipe_controllers/sequence/pipe_sequence.py` |
| PipeCondition | `pipelex/pipe_controllers/condition/pipe_condition.py` |
| PipeBatch | `pipelex/pipe_controllers/batch/pipe_batch.py` |
| PipeParallel | `pipelex/pipe_controllers/parallel/pipe_parallel.py` |
| Router selection | `pipelex/pipelex.py` (in `Pipelex.setup()`) |


## Inference Backend Plugins

# Inference Backend Plugins

Every model call in Pipelex — an LLM completion, an image generation, a document extraction, a web search — is served by an **inference worker**. Which worker handles a given model is decided entirely by data: a model's `sdk` field selects a backend, and a **backend plugin** is what teaches Pipelex how to build the worker for that `sdk`.

Core names no backend by import or by string. The built-in drivers (OpenAI, Gateway, Anthropic, Mistral, Bedrock, Google, FAL, HuggingFace, Docling, …) are plugins too — they ride the exact same seam an out-of-tree plugin would. This page documents that seam, the **Inference SPI** a plugin compiles against, and how to write one.

---

## The seam in one view

```
boot (Pipelex.setup)
  └─ build_registrar(config, builtin_plugins=BUILTIN_PLUGINS, …)   # pure, import-light
       ├─ for each built-in plugin (both layers' halves, composed)
       └─ for each installed entry point in the requested groups
            #  "pipelex.plugins.kernel" alone on a kernel-only boot; both groups on a full one
            └─ plugin.register(registrar)        # side-effect-free
                 └─ registrar.add_inference_backend(family=…, sdk=…, make_worker=…)
  └─ InferenceBackendRegistry(registrar.inference_backends)   # stored on the hub

run time (e.g. LLMWorkerFactory.make_llm_worker)
  └─ model_handle = ModelHandle.make_for_inference_model(...)
  └─ make_worker = get_inference_backend_registry().lookup(family=LLM, sdk=model_handle.sdk)
  └─ worker = make_worker(inference_model=…, backend=…, sdk_clients=…, reporting_delegate=…)
```

The worker factories (`LLMWorkerFactory`, `ImgGenWorkerFactory`, `ExtractWorkerFactory`, `SearchWorkerFactory`) hold **no** `match` over SDK strings. They build a `ModelHandle`, resolve the `InferenceBackend` config, look up the backend's `make_worker` by `(family, sdk)`, and call it. A lookup miss raises a friendly `InferenceBackendNotFoundError` ("… Is its plugin installed and enabled?").

---

## The contract: `PipelexPlugin`

A plugin is any object satisfying the `@runtime_checkable` `PipelexPlugin` protocol:

```python
class PipelexPlugin(Protocol):
    name: str            # unique, lowercase identifier (e.g. "openai")
    targets_api: int     # must equal PLUGIN_API_VERSION
    def register(self, registrar: PluginRegistrar) -> None: ...
```

`register` is the **only** method core calls, and it is **side-effect-free**: it may call the registrar's menu methods and nothing else — no hub access, no I/O, no client/SDK construction. This is what makes `build_registrar` safe to run more than once (it runs at boot, and again whenever the `pipelex plugins list` diagnostic command re-discovers plugins to print what each contributed).

`targets_api` is checked against `PLUGIN_API_VERSION`. A mismatch fails loud with `PluginApiVersionMismatchError` — a single coarse integer gate, not semver-range matching.

---

## Registering an inference backend

A backend plugin's `register` calls one menu method per `(family, sdk)` it serves:

```python
registrar.add_inference_backend(
    family=InferenceFamily.LLM,         # LLM | IMG_GEN | EXTRACT | SEARCH
    sdk="acme",                          # the model's `sdk` string
    make_worker=_make_acme_worker,       # a MakeWorkerFn (a plain callable)
)
```

A registry key is `(family, sdk)`. The same `sdk` string may appear in two families (e.g. `google` serves both `LLM` and `IMG_GEN`); they are distinct keys. A duplicate `(family, sdk)` fails loud with `DuplicateInferenceBackendError` naming **both** contributing plugins.

One plugin may register across several families from a single `register` — the built-in `gateway` plugin serves all four, `mistral` serves `LLM` + `EXTRACT`, `linkup` serves `EXTRACT` + `SEARCH`. This is the cross-family-vendor coordination point: one plugin, many backends.

---

## `MakeWorkerFn` — the locked call shape

A backend is a typed **callable**, not a one-method object. Its signature is the contract:

```python
def _make_acme_worker(
    *,
    inference_model: InferenceModelSpec,
    backend: InferenceBackend,
    sdk_clients: SdkClientRegistry,
    reporting_delegate: ReportingProtocol | None,
) -> InferenceWorkerAbstract:
    ...
```

The factory always passes all four keyword arguments. A stateless backend simply ignores the ones it doesn't need (`# noqa: ARG001` on the unused parameter — see the built-in `pypdfium2` and `azure_rest` plugins).

Two invariants shape what goes *inside* the closure:

- **Import-light.** The plugin module must import no backend SDK at module load. Do the SDK import **inside** the closure (`# noqa: PLC0415`), so merely discovering the plugin never pulls a heavy/optional dependency. Booting Pipelex with the built-ins registered imports none of `anthropic`, `mistralai`, `google.genai`, `boto3`, `fal_client`, `huggingface_hub`, `docling`, `linkup`, … — enforced by a subprocess import-blocker guard.
- **Fail at use, not at boot.** Guard an optional dependency with `require_sdk(...)` *inside* the closure. A missing extra then raises `MissingDependencyError` (naming the package and the `pipelex[<extra>]` install hint) only when the backend is actually used.

Client memoization goes through `sdk_clients.get_or_create(handle=…, build=lambda: …)` — the registry caches one SDK client per `ModelHandle`, so repeated worker construction reuses the connection.

---

## Listing models — an optional capability

Alongside its worker factory, a backend plugin **may** register a model lister — the callable behind `pipelex show models <backend>`. It is optional: a backend that cannot enumerate its models simply never calls `add_model_lister`, and the listing command reports that SDK as unsupported-for-listing. The contract grows by one *optional* method, so progressive disclosure is preserved.

```python
registrar.add_model_lister(sdk="acme", lister=_list_acme_models)
```

A `ListModelsFn` mirrors `MakeWorkerFn` — import-light to reference, lazy inside. It is always `async` (the listing loop awaits it) and is keyed by `sdk` alone (listing is per-SDK, not per-`(family, sdk)`):

```python
async def _list_acme_models(
    *,
    sdk: str,
    backend_name: str,
    backend: InferenceBackend,
    flat: bool,
    any_listed: bool,
) -> None:
    from my_pkg.acme_list import list_acme_models  # noqa: PLC0415

    await list_acme_models(sdk=sdk, backend_name=backend_name, backend=backend, flat=flat, any_listed=any_listed)
```

The same import-light / fail-at-use rules apply: a missing optional extra must raise `MissingDependencyError` only when the backend is actually listed — use the `require_sdk(...)` helper, or an equivalent `find_spec` guard inside the function the lister delegates to (the built-in listers do the latter, reusing the guard already in their `list_*_models` functions). If a registered lister's client variant cannot enumerate models at runtime (e.g. a Bedrock-backed Anthropic client), raise `ModelListingUnsupportedError(sdk=…)`: the loop treats that as the same soft "unsupported for listing" outcome as a missing lister — never a hard failure. A duplicate `sdk` lister fails loud with `DuplicateModelListerError` naming both plugins.

---

## Authoring a backend plugin (minimal example)

A complete LLM backend plugin for a hypothetical `acme` SDK:

```python
# acme_plugin.py
from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract
from pipelex.cogt.model_backends.backend import InferenceBackend
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.plugins.contract import PLUGIN_API_VERSION
from pipelex.plugins.inference_backend_registry import InferenceFamily, require_sdk
from pipelex.plugins.model_handle import ModelHandle
from pipelex.plugins.registrar import PluginRegistrar
from pipelex.plugins.sdk_client_registry import SdkClientRegistry
from pipelex.reporting.reporting_protocol import ReportingProtocol

_ACME_MISSING_MSG = "The acme SDK is required in order to use Acme models."


def _make_acme_worker(
    *,
    inference_model: InferenceModelSpec,
    backend: InferenceBackend,
    sdk_clients: SdkClientRegistry,
    reporting_delegate: ReportingProtocol | None,
) -> InferenceWorkerAbstract:
    require_sdk(spec="acme", extra="acme", msg=_ACME_MISSING_MSG)

    from acme import AcmeClient  # noqa: PLC0415

    from my_pkg.acme_llm_worker import AcmeLLMWorker  # noqa: PLC0415

    model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)
    sdk_instance = sdk_clients.get_or_create(
        handle=model_handle,
        build=lambda: AcmeClient(api_key=backend.api_key),
    )
    return AcmeLLMWorker(
        sdk_instance=sdk_instance,
        inference_model=inference_model,
        reporting_delegate=reporting_delegate,
    )


class AcmePlugin:
    name = "acme"
    targets_api = PLUGIN_API_VERSION

    def register(self, registrar: PluginRegistrar) -> None:
        registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="acme", make_worker=_make_acme_worker)
```

### Shipping it as an out-of-tree plugin

Declare an entry point in the `pipelex.plugins.kernel` group; discovery finds it automatically once the distribution is installed (no central enable-list — *presence* is the source of truth):

```toml
# pyproject.toml of your plugin package
[project.entry-points."pipelex.plugins.kernel"]
acme = "my_pkg.acme_plugin:AcmePlugin"
```

**The group declares your layer, and the choice is not cosmetic.** An inference backend is kernel-layer — it constructs a worker, never a `Pipe` — so it belongs in `pipelex.plugins.kernel`, and a kernel-only boot reads that group and finds it. Publish the same plugin under `pipelex.plugins.interpreter` and a kernel-only boot never even *imports* your module, so your backend is simply absent. The reverse mistake fails loudly instead: a `pipelex.plugins.kernel` plugin whose `register` reaches an interpreter-layer capability — `add_orchestrator`, `add_bundle_validator`, `add_pipe_func_executor`, or the three hub slots that hand back a `Pipe`-aware object (`claim_pipe_router`, `claim_pipe_run`, `claim_pipe_func_executor`) — raises `PluginLayerViolationError` naming the capability and the group to move to. The other three slots are kernel-tier and a kernel-group plugin may claim them freely: `claim_content_generator`, `claim_task_manager`, `claim_isolated_execution_probe`. The restriction is one-directional — an interpreter-group plugin may contribute kernel-tier capabilities too.

The single pre-split `pipelex.plugins` group is retired. It is not ignored: a plugin still published there fails startup with `RetiredPluginEntryPointGroupError`, because the alternative — being silently undiscovered — is the expensive failure to diagnose.

The entry point may resolve to a plugin instance or a zero-argument factory. A broken entry point is isolated and reported as `BrokenPluginError`, never a silent skip.

### Disabling a discovered plugin

`build_registrar` skips (and logs) any plugin whose `name` appears in the core-owned `plugins.disabled` denylist:

```toml
# .pipelex/pipelex.toml
[plugins]
disabled = ["acme"]
```

For external entry points the denylist is matched against the **entry-point name** *before* the plugin is loaded, so a broken or dependency-missing installed plugin can still be disabled to recover startup (it would otherwise raise `BrokenPluginError` at load). A working external plugin that sets a `name` different from its entry-point name is denylistable by **either** name.

Disabling a core-unconditional plugin (`openai`) is a configuration error, not a no-op — it raises `CoreUnconditionalPluginDisabledError`. There is intentionally **no** allowlist: a plugin's presence (built-in or installed entry point) is what enables it.

Use `pipelex plugins list` to see every discovered plugin, the entry-point group it was found under, what each contributed, and its denylist state. The **Group** column is the first thing to read when a plugin is missing: a built-in shows `—`, and an external plugin that resolved to the wrong layer shows it there.

---

## The Inference SPI

What an out-of-tree backend plugin imports *is* the contract. The published surface:

| Symbol | Module | Role |
|--------|--------|------|
| `PipelexPlugin`, `PLUGIN_API_VERSION` | `pipelex.plugins.contract` | the plugin protocol + version gate |
| `PluginRegistrar` | `pipelex.plugins.registrar` | the accumulator `register` writes into |
| `InferenceFamily`, `MakeWorkerFn`, `require_sdk` | `pipelex.plugins.inference_backend_registry` | family enum, callable type, dependency guard |
| `ListModelsFn` | `pipelex.plugins.model_lister_registry` | the optional model-listing callable type |
| `ModelListingUnsupportedError` | `pipelex.cogt.exceptions` | soft signal a lister raises when its client variant cannot list |
| `ModelHandle` | `pipelex.plugins.model_handle` | the backend selector derived from a model spec |
| `SdkClientRegistry` | `pipelex.plugins.sdk_client_registry` | per-handle client memoization |
| `InferenceModelSpec` | `pipelex.cogt.model_backends.model_spec` | the resolved model record |
| `InferenceBackend` | `pipelex.cogt.model_backends.backend` | the backend config record (api key, extras) |
| `InferenceWorkerAbstract` and the `{LLM,ImgGen,Extract,Search}WorkerAbstract` subclasses | `pipelex.cogt.…` | the worker contracts a plugin returns |
| `MissingDependencyError` | `pipelex.exceptions` | raised by `require_sdk` |

The SPI is a documented, versioned **module/symbol list** gated by `PLUGIN_API_VERSION` — not an `__init__.py` re-export shim (the repo bans re-exports; import by full path). Anything a plugin needs to import outside this surface is a design gap to resolve, not an accident to live with.

---

## Fail-loud guarantees

| Condition | Error |
|-----------|-------|
| `targets_api` ≠ `PLUGIN_API_VERSION` | `PluginApiVersionMismatchError` |
| published under the retired `pipelex.plugins` group | `RetiredPluginEntryPointGroupError` (names the plugins and the group each should move to) |
| a `pipelex.plugins.kernel` plugin registers an interpreter-layer capability | `PluginLayerViolationError` (names the capability and the group to move to) |
| duplicate `(family, sdk)` | `DuplicateInferenceBackendError` (names both plugins) |
| duplicate `sdk` model lister | `DuplicateModelListerError` (names both plugins) |
| `name` in `plugins.disabled` but core-unconditional | `CoreUnconditionalPluginDisabledError` |
| entry point raises while loading/registering | `BrokenPluginError` |
| lookup for an unregistered `(family, sdk)` | `InferenceBackendNotFoundError` ("… Is its plugin installed and enabled?") |
| optional SDK missing at use | `MissingDependencyError` (package + `pipelex[<extra>]` hint) |

---

## Related

- [Orchestrator Plugins](orchestrator-plugins.md) — the other per-call seam, riding the same discovery/denylist machinery
- [Storage Provider Plugins](storage-provider-plugins.md) — the config-selected-singleton seam (storage backend by `storage_config.method`)
- [Secrets Provider Plugins](secrets-provider-plugins.md) — the config-selected-singleton seam (secrets backend by `secrets_config.method`)
- [Pipe Routing & Execution](pipe-routing-and-execution.md) — where worker construction sits in the run path
- [Error Model](error-model.md) — how these errors render and dereference
- [Architecture Overview](architecture-overview.md)


## Orchestrator Plugins

# Orchestrator Plugins

A pipe a host runtime invokes through the runtime bridge runs along **two orthogonal axes**:

- **`orchestration_mode`** — *which* orchestrator runs the pipe. An **open string token**, not a closed enum: core owns only `"direct"` (in-process); every other token is contributed by the plugin that owns its orchestrator — [`"temporal"`](https://pipelex.com/products#temporal) (durable, on a Temporal worker fleet) by our Temporal plugin, [`"mistral-workflows"`](https://pipelex.com/products#mistral-workflows) (decomposed into Mistral Workflows primitives) by our Mistral Workflows plugin. Neither is built into the open-source `pipelex` core: both ship as external, closed-source host-runtime backends, distributed privately rather than on PyPI as part of Pipelex's [workflow-orchestration offer](https://pipelex.com/products#durable-execution).
- **`delivery`** — *whether the caller waits*. A **closed** core `DeliveryMode` enum (`BLOCKING` / `FIRE_AND_FORGET`) on the wire input, set by the endpoint, never received from a caller. On the SPI it is expressed as *which method* the endpoint calls — `execute` (blocking) or `start` (fire-and-forget) — so each return type is truthful on its own; `supports_fire_and_forget` advertises whether an orchestrator can do genuine async.

An **orchestrator** is what knows how to run a pipe under one token. Core names no orchestrator by import or by string. The bridge resolves the orchestrator for the requested token from a registry (keyed by the token `str`) and calls its `execute` — `"direct"` is contributed by a core plugin, `"temporal"` by our Temporal plugin, `"mistral-workflows"` by our external Mistral Workflows plugin. A lookup miss raises a generic `MissingOrchestratorError` that names no orchestrator. This page documents that seam, the **Orchestrator SPI** a host-runtime plugin compiles against, and how the Temporal plugin is wired.

---

## The seam in one view

```
<host-side bridge entry>(input_payload)       # not open core
  → build the PipeJob (boundary decode + library scope + trace_context)
  → orchestrator = get_orchestrator_registry().get_optional(mode=orchestration_mode)
  → if orchestrator is None: raise MissingOrchestratorError(mode)   # generic, names no orchestrator
  → return await orchestrator.execute(pipe_job=..., delivery_assignment=...)   # or orchestrator.start(...) for FIRE_AND_FORGET
```

The registry is built once at boot from whatever the discovered plugins contributed (`build_registrar` → `OrchestratorRegistry` on the hub). There is no `match orchestration_mode:` anywhere in the bridge — the token set is open, so validation is the registry lookup itself; adding a mode's behavior means registering an orchestrator for its token, nothing in core changes.

The *port* — `OrchestratorProtocol` and its registry — is kernel-layer, and the registry is set on the `RuntimeHub`. An orchestrator *adapter* is not: constructing one names the method interpreter, so the built-in `direct` one lives in `pipelex/interpreter_plugins/` rather than beside the inference plugins. An out-of-tree plugin makes the same split explicit instead of by directory: it declares its layer through the entry-point group it publishes under, and an orchestrator plugin therefore belongs in `pipelex.plugins.interpreter` ([below](#shipping-it-as-an-out-of-tree-plugin)). See [Hub layering](../contribute/hub-layering.md#where-the-built-in-plugins-split).

!!! note "Open seam, closed host-side entry"
    Pipelex's own host-side entry — the bridge entry point and `PipeJob` builder, plus the dispatch primitives that decode the boundary payload, scope the per-call library, and deliver results — is **not part of open core**: it backs a Pipelex [commercial capability](https://pipelex.com/products#durable-execution). What stays **open** is the seam you extend to run pipes on an orchestrator of your own: the orchestrator registry, `OrchestratorProtocol`, and the `pipelex.runtime_bridge` modules listed in [The Orchestrator SPI](#the-orchestrator-spi) below.

---

## The orchestrator contract

An orchestrator satisfies `OrchestratorProtocol` (`pipelex/plugins/orchestrator_registry.py`):

```python
class OrchestratorProtocol(Protocol):
    supports_fire_and_forget: bool

    async def execute(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeRunOutput: ...

    async def start(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeDispatchAck: ...
```

The delivery axis is split into two methods so each return type is truthful on its own. `execute` is the BLOCKING arm: it awaits completion and returns the completed-run `PipelexPipeRunOutput` — which always carries a main stuff (`main_stuff_name` is required, no "not finished yet" escape hatch). `start` is the FIRE_AND_FORGET arm: it genuinely enqueues the job and returns a `PipelexPipeDispatchAck` (`pipeline_run_id` + `workflow_id`, nothing more — nothing has run yet). `supports_fire_and_forget` is the capability a runner reads *before* dispatch — `/start` rejects honestly (4xx) when the resolved mode cannot do genuine async, instead of silently running blocking and acking; an orchestrator that cannot (like core's DIRECT) implements `start` by raising, unreachable behind that gate.

A plugin contributes one per token it serves by calling the registrar menu in its `register`, passing the token as a raw string (no enum, no cast):

```python
registrar.add_orchestrator(mode="temporal", orchestrator=TemporalOrchestrator())
```

Constructing the orchestrator instance must be **import-light** — it must not import the host-runtime SDK (`temporalio`, …) at module scope or in its `__init__`. The heavy import happens lazily inside `execute` (and a friendly `MissingOrchestratorError` is raised there if the mode's extra is absent), so discovering and registering the plugin never pulls the SDK. This is what keeps boot import-light even on a process that will never use the mode.

### A missing orchestrator is a generic, plugin-decoupled error

A token with no registered orchestrator (its plugin is not installed) raises `MissingOrchestratorError(mode=...)` (`pipelex/runtime_bridge/exceptions.py`). The message names the token but **no orchestrator** — *"No orchestrator is registered for orchestration mode '{mode}'; is its plugin installed?"* — so core stays fully decoupled from its plugins (it never spells out the Temporal or Mistral Workflows backend). The one special case is the core `"direct"` token: its orchestrator is always present, so a miss there reports a boot/discovery fault. The message survives STRICT error disclosure.

---

## Bundle validators: the `/validate` counterpart

`/validate` rides the same open-token seam as `execute`. A **bundle validator** produces a validation verdict for one orchestration mode, the way an orchestrator runs a pipe for one mode: the core `direct` plugin contributes the in-process validator, and an external orchestrator plugin contributes a worker-dispatched validator under its own token — which core never names. A host runtime resolves the requested mode and dispatches through the `BundleValidatorRegistry` on the hub instead of branching on a backend.

A validator satisfies `BundleValidatorProtocol` (`pipelex/plugins/bundle_validator_registry.py`):

```python
class BundleValidatorProtocol(Protocol):
    async def validate_bundles(
        self,
        *,
        mthds_contents: list[str],
        mthds_sources: list[str] | None,
        allow_signatures: bool,
        library_dirs: "Sequence[Path] | None",
    ) -> BundleValidationVerdict: ...
```

The `library_dirs` annotation is a quoted forward reference because `Sequence` and `Path` are imported under `TYPE_CHECKING` in `bundle_validator_registry.py`; a plugin that imports them at module scope can use the unquoted form.

Two contract points distinguish it from `OrchestratorProtocol.execute`:

- **No fire-and-forget arm.** Validation is inherently blocking, so — unlike orchestration's `execute`/`start` split — there is no `start` counterpart and no `supports_fire_and_forget` capability flag.
- **Verdict-as-value, not raise.** `validate_bundles` *returns* the verdict — `BundleValidationVerdict` is the union of the valid arm (a `ValidationReport`) and the invalid arm (an `ErrorReport` carrying `validation_errors`) — and raises only for a no-verdict infra fault, which a host runtime maps to a 5xx. This is the same valid/invalid pair the API maps onto its 200-always `/validate` wire, so the verdict contract is backend-independent.

The seam is deliberately typed at the MTHDS-protocol level (`ValidationReport` from `mthds.protocol`), not the concrete `PipelexValidationReport` envelope: the concrete report's module reaches the hub, so naming it from this hub-reachable seam would close an import cycle, and the seam is generic across orchestrators (language-standard altitude), so it speaks the protocol report — the Pipelex-runtime envelope is the concrete `ValidationReport` subtype the validators actually produce, and the API recovers that precise type at its edge. `library_dirs` is host context the in-process arm needs to load the method library; a worker-dispatched arm ignores it — its worker loads its own library.

A plugin contributes one per token, right next to its orchestrator:

```python
registrar.add_orchestrator(mode=DIRECT_ORCHESTRATION_MODE, orchestrator=DirectOrchestrator())
registrar.add_bundle_validator(mode=DIRECT_ORCHESTRATION_MODE, validator=DirectBundleValidator())
```

The failure modes mirror the orchestrator seam exactly: a duplicate token fails loud at discovery (`DuplicateBundleValidatorError`, naming both plugins), and a lookup miss raises `MissingBundleValidatorError` — the same generic, plugin-decoupled message shape as `MissingOrchestratorError` (*"is its plugin installed?"*), with the same `"direct"` special case reporting a boot/discovery fault.

---

## HTTP error mappers: rendering an orchestrator's transport faults

An orchestrator's *runtime* (a Temporal client, a Mistral workflow runner) raises SDK-specific transport faults — a server unreachable, a workflow timeout — that a host runtime serving HTTP must turn into a proper error response, not a catch-all 500. But core names no web framework and the SDK lives only in the plugin, so the host (`pipelex-api`) cannot itself know how to classify `temporalio.TemporalError`.

The plugin bridges that gap by contributing a **framework-agnostic mapper** — a function from one exception type to a structured `ErrorReport` (`pipelex/base_exceptions.py`):

```python
registrar.add_http_error_mapper(
    exc_type_provider=lambda: TemporalError,      # SDK imported only when a host resolves the mappers, never at register
    to_error_report=lambda exc: ErrorReport(...), # classified transient / RUNTIME
)
```

The exc type is supplied as a **provider thunk**, not the bare class, on purpose: naming `temporalio.TemporalError` requires importing `temporalio` (the whole SDK), so a bare `exc_type=` would force that import at `register` — breaking the import-light invariant for a plugin that hard-depends on a heavy SDK. The provider defers the import to read time (a host runtime's app construction), where the plugin — and therefore its SDK — is by definition installed.

The contract is deliberately split so no layer overreaches:

- **The plugin** owns *classification* — which exception, transient or not, which error domain. It stays import-light: `register` only records the provider + closure; the SDK import happens when the provider runs at read time (and the `to_error_report` closure when the mapper is first invoked), never at registration.
- **Core** owns *transport* — `registrar.get_http_error_mappers()` runs every provider, builds the `{exc_type: mapper}` dict, and is fail-loud on a duplicate *resolved* exception type (naming both plugins). `ErrorReport` is a core type, so the seam carries **no** web-framework import.
- **The host runtime** owns *presentation* — at app construction it iterates the resolved mappers and wraps each into one framework error handler (FastAPI, …) that runs the mapper, then renders the `ErrorReport` through its own RFC 7807 + `DisclosureMode` path. FastAPI / Starlette stays only in the host; core and the plugin import neither.

This is what lets the public `pipelex-api` base be orchestrator-agnostic and still render a Temporal (or Mistral) transport fault correctly: install the flavor's plugin and its mapper rides in; install none and there is simply nothing to wrap. The capability is optional, contributing one method to the plugin contract.

---

## Boot-orchestrator plugins: claiming the runtime

Some orchestrators don't just serve a per-call mode — they reconfigure the whole process to run *as* that runtime (a Temporal worker). Such a plugin **claims process-global hub slots**, but only when the core-owned boot gate names it. `plugins.boot_orchestrator == self.name` means "boot this process as a Temporal-default runtime", not "the Temporal plugin is on". The gate is a backend-agnostic name-match — core names no orchestrator, and `register` reads no config file (the rich orchestrator config self-loads inside the thunks):

Not every orchestrator goes this far. A per-call-only plugin — our Mistral Workflows plugin is the minimal example — contributes just its `"mistral-workflows"` orchestrator and claims **no** hub slots: its router is installed per workflow invocation by the workflow body itself, so there is no process-global boot slot to claim and it never participates in the boot gate below. The boot-orchestrator machinery in this section applies only to a plugin that boots the process as its runtime (today, our Temporal plugin).

```python
if registrar.config.plugins.boot_orchestrator == self.name:
    registrar.claim_content_generator(_make_temporal_content_generator)   # a thunk, not an instance
    registrar.claim_task_manager(_setup_temporal_task_manager)
    registrar.claim_pipe_router(_make_temporal_pipe_router)
    registrar.claim_pipe_run(_make_temporal_pipe_run)
    registrar.claim_isolated_execution_probe(_temporal_isolated_execution_probe)
    registrar.add_teardown(_teardown_temporal)
```

Each `claim_*` takes a **thunk** (a zero-arg factory), never a constructed instance. The thunk runs only at the boot apply-point, so `register` itself imports no `temporalio` — even on a worker. This is the deferred-thunk rule that keeps the import-light invariant intact at boot.

### The isolated-execution-probe slot

The first four slots swap in the runtime's implementations of core execution services. The fifth, `claim_isolated_execution_probe`, is different in kind: its thunk resolves not a service but an **ambient predicate** (`Callable[[], bool]`) reporting whether the current call runs inside an isolated sub-execution — a Temporal activity — whose side-effecting emissions must not be written into the parent run's replay-deterministic buffer. `ReportingManager` consults it (`hub.is_in_isolated_execution()`) to route an activity-side usage emission to the per-process log instead of the workflow's registered buffer. It exists as a hub slot for the same reason the others do: only a boot-orchestrator plugin whose runtime has a replay/activity split knows how to answer, and core names no such runtime. Unclaimed (any in-process boot), the hub's core default answers "never isolated" — the in-process orchestrator has no such split.

Because the gate is a name-match, `plugins.boot_orchestrator` must name a plugin that actually registered. After discovery, `Pipelex.setup` rejects a `boot_orchestrator` that no registered plugin carries — a typo or a missing plugin (e.g. `--orchestrator temporal` without the Temporal plugin installed) raises `UnknownBootOrchestratorError` instead of silently running in-process: nothing would claim the hub slots, so the process would otherwise fall through to the core defaults and execute on the wrong runtime. The check matches against **plugin names** (the same namespace the gate uses), not the `orchestration_mode` registry — a plugin's name and the token(s) it serves are separate namespaces that *may* differ, even where a shipped plugin keeps them identical (our Temporal plugin is named `temporal` and serves `temporal`). The error names no specific plugin, keeping core decoupled.

### Injection precedence

At each ordered hub slot, `Pipelex.setup` resolves in this precedence:

1. an explicit `setup()` parameter (test/host injection) — always wins;
2. a plugin slot-claim thunk;
3. the core default.

A slot claim must never silently override an explicit injection. Only the content generator and pipe router take an explicit `setup()` parameter; the other slots skip step 1 — the task manager (no core default either — unclaimed means no task manager wiring at all), the isolated-execution probe (whose core default is the always-False predicate above), and the pipe run (plugin thunk or core default, nothing else). Teardown runs the plugin-registered teardown callbacks **LIFO**, before core teardown, so a worker's in-flight runtime resources release first.

---

## Operational commands ship as console scripts

The plugin seam does **not** contribute commands to the host `pipelex` CLI. An operational command — a worker daemon, a one-time namespace bootstrap — is a daemon/utility, not a way a pipe *runs*, so a plugin that needs one ships its own `[project.scripts]` console script, which pip materializes into a standalone executable. Nothing is harvested onto `pipelex` at import time, so a broken or colliding plugin can never brick `pipelex --help` / `doctor` / `init`.

Our Temporal plugin follows this rule: its `worker` and `setup-namespace` commands ship as that plugin's own console script, declared in the plugin distribution's `pyproject.toml`:

```toml
[project.scripts]
my-runtime = "my_package.my_cli:app"
```

Because the script is declared by the plugin distribution itself, it travels with that dist — nothing in core to harvest or move.

---

## The Orchestrator SPI

What an out-of-tree orchestrator imports *is* a contract. The SPI is a documented, versioned set of modules and symbols (gated by `PLUGIN_API_VERSION` in `pipelex/plugins/contract.py`) — not an `__init__.py` re-export shim. It is sized to what a real orchestrator (our Mistral Workflows plugin) actually imports, not guessed. Anything an orchestrator needs that is *outside* this surface is a design bug — promote it into the SPI or remove the need.

| Area | Modules / symbols |
|---|---|
| Boundary serialization + boot | `pipelex.runtime_bridge.serialization` (`serialize_pipe_output`, `serialize_completed_output`, `PIPE_DISPATCH_ERRORS`), `pipelex.runtime_bridge.payloads` (`PipelexPipeRunInput`, `PipelexPipeRunOutput`, `PipelexPipeDispatchAck`), `pipelex.runtime_bridge.bootstrap` (`ensure_pipelex_booted`) |
| Mode + delivery + errors | `pipelex.runtime_bridge.orchestration_mode` (`OrchestrationMode`, `DIRECT_ORCHESTRATION_MODE`), `pipelex.runtime_bridge.delivery_mode` (`DeliveryMode`), `pipelex.runtime_bridge.exceptions` (`MissingOrchestratorError`, `PipelexBridgeDispatchError`) |
| Working-memory hydration | `pipelex.runtime_bridge.primitives.hydration` (re-hydrate `working_memory_raw` → typed `WorkingMemory`; stayed open because it is host-agnostic — used by core delivery and the open `pipelex-api` runner, and re-used across the boundary by the commercial plugins) |
| Plugin contract | `pipelex.plugins.contract` (`PipelexPlugin`, `PLUGIN_API_VERSION`), `pipelex.plugins.registrar` (`PluginRegistrar` menu: `add_orchestrator`, `add_bundle_validator`, `add_http_error_mapper`, `claim_*`, `add_teardown`; read accessor: `get_http_error_mappers`), `pipelex.plugins.orchestrator_registry` (`OrchestratorProtocol`), `pipelex.plugins.bundle_validator_registry` (`BundleValidatorProtocol`, `BundleValidationVerdict`) |
| Execution protocols | `PipeRouterProtocol` (incl. the [`run_batch_branch`](./pipe-routing-and-execution.md#the-batch-branch-hook) fan-out hook — concrete default, override it only to isolate batch branches), `PipeRunProtocol`, `ContentGeneratorProtocol`, the task-manager protocol |
| Payload / core types | `PipeJob`, `PipeOutput`, `DeliveryAssignment`, `WorkingMemory` (+ factory, `dump_for_transport`), `JobMetadata`, `LibraryCrate` |
| Library + hub scoping | `set_current_library` / `get_current_library`, `scoped_pipe_router`, `get_class_registry` (per-call library hydration via `library_crate_dump`) |
| Tracing / graph hooks | `trace_events`, `graph_tracer_manager`, `tracing_assembly` (per-step trace/usage events across the boundary) |

!!! warning "Not in the open SPI"
    The host-side bridge entry point, the cross-process dispatch primitives (boundary delivery, pipe classification, submitter-side hydration, trace flush, scoped-library, and the submit-arg envelope), and the transport-prep helpers were all extracted **out of open core**. Pipelex's own host-runtime embedding is a commercial capability — our Temporal and Mistral Workflows plugins build on that closed side; a third-party out-of-tree orchestrator compiles against the **open** rows above only.

---

## Worked example: our Temporal plugin

Our Temporal plugin — external, closed-source, shipped as its own distribution — is the reference orchestrator plugin. Its `register`:

- **always** (regardless of the boot gate): contributes a single `TemporalOrchestrator` registered once under the `"temporal"` token (import-light; `temporalio` is pulled lazily inside `execute`/`start`), the matching `TemporalBundleValidator` under the same token (the worker-dispatched `/validate` arm), and an HTTP error mapper classifying Temporal transport faults. The orchestrator advertises `supports_fire_and_forget = True`; its `execute` awaits completion and reports `make_workflow_id(...)`, its `start` enqueues the workflow and returns a `PipelexPipeDispatchAck` carrying the workflow id;
- **only when `plugins.boot_orchestrator == "temporal"`**: claims the content-generator / task-manager / pipe-router / pipe-run / isolated-execution-probe hub slots with thunks and registers the teardown callback — booting this process as a Temporal-default runtime.

The orchestrator itself keeps the `WorkflowExecutionError` catch and the `make_workflow_id` recompute in the blocking `execute` arm. It serializes its `PipeOutput` through `pipelex.runtime_bridge.serialization`, shared with the core DIRECT orchestrator so the boundary shape cannot drift.

The Temporal plugin is **external** — it ships as its own distribution and is discovered through a `pipelex.plugins.interpreter` entry point in that dist's `pyproject.toml`, not through `BUILTIN_PLUGINS`. Core's built-ins hold only the always-shipped inference and `direct` plugins and explicitly exclude Temporal — split by layer across `pipelex/providers/builtins.py` (the kernel adapters) and `pipelex/interpreter_plugins/builtins.py` (the interpreter-touching `direct` and `pipe_func` plugins, which is also where the two halves are composed); installing the plugin is all it takes to make the `"temporal"` orchestrator available — zero config, no core import of `temporalio`. Its operational `worker` / `setup-namespace` commands ship as that dist's own standalone console script, so they travel with it.

---

## Authoring an out-of-tree orchestrator plugin

A third-party host-runtime plugin is a distribution that:

1. defines a plugin class (`name`, `targets_api`, `register`) whose `register` calls `add_orchestrator(mode=..., orchestrator=...)` for the mode(s) it serves — import-light;
2. compiles its orchestrator against the Orchestrator SPI above (and nothing outside it);
3. advertises itself under the `pipelex.plugins.interpreter` entry-point group — an orchestrator constructs `Pipe`-aware objects, so it is interpreter-layer, and the group is how a plugin declares that. Publishing an orchestrator under `pipelex.plugins.kernel` instead raises `PluginLayerViolationError` at register time. See [Inference Backend Plugins](inference-backend-plugins.md#shipping-it-as-an-out-of-tree-plugin) for what each group means; the pre-split `pipelex.plugins` group is retired and now fails startup:

```toml
[project.entry-points."pipelex.plugins.interpreter"]
my_runtime = "my_package.my_plugin:MyRuntimePlugin"
```

An interpreter-group plugin is *not* restricted to interpreter-layer capabilities — the cross-check runs one way only. Our own Temporal plugin contributes an orchestrator and an HTTP-error mapper (kernel-tier) from the same `register`, which is why the rule is asymmetric.

Installing the distribution makes the mode available; uninstalling removes it. No core change, no central registration list. A discovered plugin can be quarantined without uninstalling via the `plugins.disabled` denylist (see [Inference Backend Plugins](inference-backend-plugins.md) for the shared discovery/denylist machinery).


## Storage Provider Plugins

# Storage Provider Plugins

Every asset Pipelex persists — a generated image, an uploaded document, a payload blob — is written through a single **storage provider** set on the hub at boot. Which provider that is comes entirely from data: one config field, `storage_config.method`, names a backend, and a **storage plugin** is what teaches Pipelex how to build the provider for that method.

Core names no storage backend by import or by string. The built-in providers (`local`, `in_memory`, `s3`, `gcp`) are a plugin too — the always-on `StoragePlugin` — riding the exact same seam an out-of-tree `pipelex-storage-<backend>` package would. This page documents that seam, the factory contract a plugin registers, and how to write one.

---

## A different shape from the inference/orchestrator seams

The [inference-backend](inference-backend-plugins.md) and [orchestrator](orchestrator-plugins.md) registries are selected **per call** — a model's `sdk`, a request's `orchestration_mode`. Storage is not: it is a **process-global singleton selected by its own config key**, independent of the orchestrator. So it rides a net-new mechanism the track calls a **keyed registry + config-selected singleton**:

> Plugins register N provider factories into a registry keyed by an open `method` token. At boot, core reads `storage_config.method`, looks that token up in the registry, and calls the factory to produce the one provider set on the hub.

This is deliberately **not** a hub slot (those are orchestrator-coupled, claimed only when `boot_orchestrator == plugin.name`). Storage selection has nothing to do with the orchestrator, so it gets its own registry and its own config key. The same mechanism also backs the [secrets provider](secrets-provider-plugins.md) seam (`secrets_config.method`).

---

## The seam in one view

```
boot (Pipelex.setup)
  └─ build_registrar(config, builtin_plugins=BUILTIN_PLUGINS, …)   # pure, import-light
       ├─ for each built-in plugin                   (StoragePlugin is one)
       └─ for each installed entry point in the requested groups
            #  "pipelex.plugins.kernel" alone on a kernel-only boot; both groups on a full one
            └─ plugin.register(registrar)            # side-effect-free
                 └─ registrar.add_storage_provider(method=…, factory=…)
  └─ StorageProviderRegistry(registrar.storage_providers)    # stored on the hub
  └─ storage_provider = registry.get_required(method=storage_config.method)(storage_config)
  └─ set_storage_provider(storage_provider)          # the one provider every consumer reads
```

There is no `match storage_config.method:` anywhere in boot — the token set is open, so validation *is* the registry lookup. Adding a backend means registering a factory for its token; nothing in core changes. Every downstream consumer (input normalizer, content generator, PDF/image utilities, delivery executor) keeps calling `get_storage_provider()` and is unaffected by which method was selected.

The provider is resolved **after** the secrets provider is on the hub, so a factory may perform a hub secrets read at its apply-point (the `gcp` factory does — see below).

### Injection precedence

`Pipelex.setup` resolves the storage provider in this order:

1. an explicit `setup(storage_provider=...)` parameter (test/host injection) — always wins;
2. the config-selected registry factory (`get_required(method=storage_config.method)(config)`).

There is no separate core default: the built-in `StoragePlugin` *is* the default supplier of every method, so an ordinary boot always resolves through step 2.

---

## The contract: `StorageProviderFactoryFn`

A storage backend is a typed **callable** — whole storage config in, provider out (`pipelex/plugins/storage_provider_registry.py`):

```python
StorageProviderFactoryFn = Callable[[StorageProviderConfig], StorageProviderAbstract]
```

Passing the whole `StorageProviderConfig` (not a pre-resolved sub-config) is deliberate: it lets a factory read whatever it needs — including a hub secrets read — at the boot apply-point, never at registration.

A plugin contributes one factory per method it serves by calling the registrar menu in its `register`, passing the token as a raw string:

```python
registrar.add_storage_provider(method="azure", factory=_make_azure_storage_provider)
```

`register` is **side-effect-free** — it may call the registrar menu and nothing else (no hub access, no I/O, no SDK import). That is what keeps `build_registrar` safe to run more than once (at boot, and again for the `pipelex plugins list` diagnostic).

### Import-light registration, deferred dependency

The factory is a plain callable stored at registration and **invoked only at the boot apply-point**. Two invariants follow, and they are what let an SDK-backed provider ship without weighing down every boot:

- **Import-light.** The plugin module must import no backend SDK at module load. Registering the built-in `s3`/`gcp` factories pulls in neither `aioboto3` nor `google-cloud-storage` — the SDK guard lives *inside* the provider's I/O methods (`_get_session` / `_get_bucket`), not its `__init__`. Selecting `s3` therefore *constructs* an `S3StorageProvider` even with the extra absent; the guard fires only on the first actual load/store.
- **Fail at use, not at boot.** An optional dependency raises `MissingDependencyError` (naming the package and the `pipelex[<extra>]` install hint) when the backend is *used*, not when it is registered or selected.

---

## The built-in `StoragePlugin`

`pipelex/providers/storage/storage_plugin.py` is the reference storage plugin. It is **core-unconditional** — storage is required infra, so it joins the kernel layer's `KERNEL_CORE_UNCONDITIONAL_PLUGIN_NAMES` — composed into `CORE_UNCONDITIONAL_PLUGIN_NAMES` at the boot entrypoint — and cannot be disabled into a boot with no storage:

```python
class StoragePlugin:
    name = "storage"
    targets_api = PLUGIN_API_VERSION

    def register(self, registrar: PluginRegistrar) -> None:
        registrar.add_storage_provider(method=StorageMethod.LOCAL, factory=_make_local_storage_provider)
        registrar.add_storage_provider(method=StorageMethod.IN_MEMORY, factory=_make_in_memory_storage_provider)
        registrar.add_storage_provider(method=StorageMethod.S3, factory=_make_s3_storage_provider)
        registrar.add_storage_provider(method=StorageMethod.GCP, factory=_make_gcp_storage_provider)
```

Each factory is a module-level closure that None-checks its sub-config, calls `lazy_validate()`, and constructs the provider. The built-in tokens are the `StorageMethod` enum values (`"local"`, `"in_memory"`, `"s3"`, `"gcp"`); because a `StrEnum` key compares and hashes equal to its plain-`str` form, boot's plain-`str` `config.method` resolves them.

The `gcp` factory is the one that reads a secret from the hub — legal because the factory runs at the boot apply-point, after the secrets provider is set, never inside `register`:

```python
def _make_gcp_storage_provider(config: StorageProviderConfig) -> StorageProviderAbstract:
    ...
    return GcpStorageProvider(
        bucket_name=config.gcp.bucket_name,
        project_id=config.gcp.project_id,
        credentials_file_path=get_secrets_provider().get_required_secret(secret_id="GCP_CREDENTIALS_FILE_PATH"),
        signed_urls_lifespan=config.gcp.signed_urls_lifespan,
    )
```

---

## Selecting a method by config

`storage_config.method` is an **open `str` token** (Decision D1), not a closed enum. The built-ins use the `StorageMethod` values; an external `pipelex-storage-<backend>` plugin registers its own (e.g. `"azure"`). A config naming an external method **parses fine** — the per-method sub-config validator requires a matching sub-config only for the four built-in tokens and lets any other token through:

```toml
# .pipelex/pipelex.toml
[pipelex.storage_config]
method = "azure"          # an out-of-tree provider — no built-in sub-config required
```

Whether that token names an *installed* provider is validated at **registry lookup**, not at parse: an unknown method surfaces as `UnknownStorageMethodError` at boot, which is the right layer — it lists the registered methods so the fix is obvious.

!!! note "External-provider config surface is a scoped follow-up"
    `StorageProviderConfig` has fixed, typed per-method sub-models, so an out-of-tree `azure` provider has nowhere to read *its* structured config yet. The seam for built-in methods lands first; a generic passthrough sub-config for external providers is a captured follow-up (Decision D3), not built speculatively. Until it lands, an external provider reads its own config from the environment or its own file. **One live gap:** `StorageProviderConfig.uri_format` (read by `GeneratedContentFactory._build_storage_key` on every content store) is only defined for the four built-in methods and raises `StorageConfigError` for any other token — so an external provider selects and boots cleanly but a *generated-content store* through it fails until D3 gives external methods a `uri_format`. External providers are therefore usable today for their own direct storage API, not yet as the backing store for generated content.

---

## Fail-loud guarantees

| Condition | Error |
|-----------|-------|
| `storage_config.method` names no registered provider | `UnknownStorageMethodError` (lists the registered methods) |
| published under the retired `pipelex.plugins` group | `RetiredPluginEntryPointGroupError` (names the plugins and the group each should move to) |
| two plugins register the same `method` | `DuplicateStorageProviderError` (names both plugins) |
| `name` (`"storage"`) in `plugins.disabled` | `CoreUnconditionalPluginDisabledError` |
| entry point raises while loading/registering | `BrokenPluginError` |
| optional SDK missing at use | `MissingDependencyError` (package + `pipelex[<extra>]` hint) |

The duplicate detection is the same fail-loud `_add` helper the inference/orchestrator menus use — a conflict names both contributing plugins, so a collision between two installed storage packages is unambiguous.

---

## Authoring an out-of-tree storage plugin

A third-party storage plugin is a distribution that:

1. implements a `StorageProviderAbstract` subclass — the three hooks `_load_with_metadata`, `_store`, and `public_url` — deferring any SDK import into those methods (import-light);
2. defines a plugin class (`name`, `targets_api`, `register`) whose `register` calls `add_storage_provider(method="<token>", factory=...)` — and nothing else;
3. advertises itself under the `pipelex.plugins.kernel` entry-point group — a storage provider is kernel-layer, and the group is how a plugin declares that (see [Inference Backend Plugins](inference-backend-plugins.md#shipping-it-as-an-out-of-tree-plugin) for what each group means and what happens if you pick the wrong one; the pre-split `pipelex.plugins` group is retired and now fails startup):

```toml
# pyproject.toml of your plugin package
[project.entry-points."pipelex.plugins.kernel"]
azure_storage = "pipelex_storage_azure.plugin:AzureStoragePlugin"
```

Installing the distribution makes the method selectable (`storage_config.method = "azure"`); uninstalling removes it. No core change, no central registration list — *presence* is the source of truth. A discovered plugin can be quarantined without uninstalling via the `plugins.disabled` denylist (matched against the entry-point name *before* load, so a broken install can still be disabled to recover startup — see [Inference Backend Plugins](inference-backend-plugins.md) for the shared discovery/denylist machinery).

Use `pipelex plugins list` to see every discovered plugin, the entry-point group it was found under, what each contributed, and its denylist state. The **Group** column is the first thing to read when a plugin is missing: a built-in shows `—`, and an external plugin that resolved to the wrong layer shows it there.

---

## Related

- [Secrets Provider Plugins](secrets-provider-plugins.md) — the sibling config-selected-singleton seam riding the same mechanism; the `gcp` factory above reads its provider
- [Inference Backend Plugins](inference-backend-plugins.md) — the per-call sibling seam and the shared discovery/denylist machinery
- [Orchestrator Plugins](orchestrator-plugins.md) — the other per-call seam, plus the hub-slot mechanism storage deliberately avoids
- [Error Model](error-model.md) — how these errors render and dereference
- [Architecture Overview](architecture-overview.md)


## Secrets Provider Plugins

# Secrets Provider Plugins

Every credential Pipelex reads — an API key, a service-account file path, a signing secret — is resolved through a single **secrets provider** set on the hub at boot. Which provider that is comes entirely from data: one config field, `secrets_config.method`, names a backend, and a **secrets plugin** is what teaches Pipelex how to build the provider for that method.

Core names no secrets backend by import or by string. The built-in `env` provider (reads secrets from environment variables) is a plugin too — the always-on `SecretsPlugin` — riding the exact same seam an out-of-tree `pipelex-secrets-<backend>` package (Vault, AWS Secrets Manager, GCP Secret Manager, …) would. This page documents that seam, the factory contract a plugin registers, and how to write one.

This is the second application of the mechanism the [storage provider](storage-provider-plugins.md) seam introduced; secrets reuses it wholesale, so the two pages describe the same shape with the nouns swapped.

---

## A different shape from the inference/orchestrator seams

The [inference-backend](inference-backend-plugins.md) and [orchestrator](orchestrator-plugins.md) registries are selected **per call** — a model's `sdk`, a request's `orchestration_mode`. Secrets is not: it is a **process-global singleton selected by its own config key**, independent of the orchestrator. So it rides the same **keyed registry + config-selected singleton** mechanism storage uses:

> Plugins register N provider factories into a registry keyed by an open `method` token. At boot, core reads `secrets_config.method`, looks that token up in the registry, and calls the factory to produce the one provider set on the hub.

This is deliberately **not** a hub slot (those are orchestrator-coupled, claimed only when `boot_orchestrator == plugin.name`). Secrets selection has nothing to do with the orchestrator, so it gets its own registry and its own config key — the sibling of `storage_config.method`.

---

## The seam in one view

```
boot (Pipelex.setup)
  └─ build_registrar(config, builtin_plugins=BUILTIN_PLUGINS, …)   # pure, import-light
       ├─ for each built-in plugin                   (SecretsPlugin is one)
       └─ for each installed entry point in the requested groups
            #  "pipelex.plugins.kernel" alone on a kernel-only boot; both groups on a full one
            └─ plugin.register(registrar)            # side-effect-free
                 └─ registrar.add_secrets_provider(method=…, factory=…)
  └─ SecretsProviderRegistry(registrar.secrets_providers)    # stored on the hub
  └─ secrets_provider = registry.get_required(method=secrets_config.method)(secrets_config)
  └─ set_secrets_provider(secrets_provider)          # the one provider every consumer reads
```

There is no `match secrets_config.method:` anywhere in boot — the token set is open, so validation *is* the registry lookup. Adding a backend means registering a factory for its token; nothing in core changes. Every downstream consumer (`get_secret(...)`, telemetry credentials, the storage `gcp` arm, search/extract workers) keeps calling `get_secrets_provider()` and is unaffected by which method was selected.

### Where in boot secrets is resolved

Secrets is resolved **early** — right after plugin discovery, after the gateway/terms precondition gate but before the telemetry factory (its first in-process consumer) and well before storage selection. Two consequences that this ordering guarantees:

- **Telemetry** receives the config-selected secrets provider when it is constructed.
- **Storage's `gcp` factory** reads the config-selected secrets provider from the hub at *its* apply-point (it resolves `GCP_CREDENTIALS_FILE_PATH`) — secrets is already on the hub by then. This ordering is pinned by an integration test that boots a non-`env` secrets method alongside `gcp` storage and asserts the gcp arm read the external provider.

### Injection precedence

`Pipelex.setup` resolves the secrets provider in this order:

1. an explicit `setup(secrets_provider=...)` parameter (test/host injection) — always wins;
2. the config-selected registry factory (`get_required(method=secrets_config.method)(config)`).

There is no separate core default: the built-in `SecretsPlugin` *is* the default supplier of the `env` method, so an ordinary boot always resolves through step 2.

---

## The contract: `SecretsProviderFactoryFn`

A secrets backend is a typed **callable** — whole secrets config in, provider out (`pipelex/plugins/secrets_provider_registry.py`):

```python
SecretsProviderFactoryFn = Callable[[SecretsProviderConfig], SecretsProviderAbstract]
```

Passing the whole `SecretsProviderConfig` (not a pre-resolved sub-config) is deliberate: it lets a factory read whatever it needs — an SDK-backed provider (Vault, AWS Secrets Manager) reads its own settings — at the boot apply-point, never at registration. The built-in `env` factory ignores the config entirely.

A plugin contributes one factory per method it serves by calling the registrar menu in its `register`, passing the token as a raw string:

```python
registrar.add_secrets_provider(method="vault", factory=_make_vault_secrets_provider)
```

`register` is **side-effect-free** — it may call the registrar menu and nothing else (no hub access, no I/O, no SDK import). That is what keeps `build_registrar` safe to run more than once (at boot, and again for the `pipelex plugins list` diagnostic).

### Import-light registration, deferred dependency

The factory is a plain callable stored at registration and **invoked only at the boot apply-point**. Two invariants follow, and they are what let an SDK-backed provider ship without weighing down every boot:

- **Import-light.** The plugin module must import no backend SDK at module load. The built-in `SecretsPlugin` needs none (the `env` provider reads environment variables); an out-of-tree `pipelex-secrets-vault` plugin must keep its `hvac`/`boto3`/etc. import *inside* the provider's methods (or its factory), never at module top-level, so discovery stays import-light even when the extra is installed.
- **Fail at use, not at boot.** An optional dependency raises `MissingDependencyError` (naming the package and the `pipelex[<extra>]` install hint) when the backend is *used*, not when it is registered or selected.

---

## The built-in `SecretsPlugin`

`pipelex/providers/secrets/secrets_plugin.py` is the reference secrets plugin. It is **core-unconditional** — a secrets provider is required infra, so it joins the kernel layer's `KERNEL_CORE_UNCONDITIONAL_PLUGIN_NAMES` — composed into `CORE_UNCONDITIONAL_PLUGIN_NAMES` at the boot entrypoint — and cannot be disabled into a boot with no secrets provider:

```python
class SecretsPlugin:
    name = "secrets"
    targets_api = PLUGIN_API_VERSION

    def register(self, registrar: PluginRegistrar) -> None:
        registrar.add_secrets_provider(method="env", factory=_make_env_secrets_provider)
```

The `env` factory ignores its config and returns a bare `EnvSecretsProvider` — the `config` parameter exists only to conform to `SecretsProviderFactoryFn`; an SDK-backed external provider is where reading the config earns its keep:

```python
def _make_env_secrets_provider(config: SecretsProviderConfig) -> SecretsProviderAbstract:
    return EnvSecretsProvider()
```

`env` is a single-impl backend: `EnvSecretsProvider` resolves `get_required_secret` / `get_optional_secret` from environment variables, and its versioned reads (`*_specific_version`) raise `NotImplementedError` because environment variables have no version history. A full external provider (Vault, AWS Secrets Manager) implements those versioned reads.

---

## Selecting a method by config

`secrets_config.method` is an **open `str` token** (Decision S1), not a closed enum. The built-in uses `"env"`; an external `pipelex-secrets-<backend>` plugin registers its own (e.g. `"vault"`). A config naming an external method **parses fine** — the token is stored verbatim and its installability is validated later, at registry lookup:

```toml
# .pipelex/pipelex.toml
[pipelex.secrets_config]
method = "vault"          # an out-of-tree provider — selected iff its plugin is installed
```

Whether that token names an *installed* provider is validated at **registry lookup**, not at parse: an unknown method surfaces as `UnknownSecretsMethodError` at boot, which is the right layer — it lists the registered methods so the fix is obvious.

!!! note "External-provider config surface is a scoped follow-up"
    `SecretsProviderConfig` today carries only `method` — the built-in `env` backend needs no per-method sub-config. An out-of-tree `vault` provider has nowhere to read *its* structured config yet; a generic passthrough for external providers is a captured follow-up (Decision S4), not built speculatively. Until it lands, an external provider reads its own config from the environment or its own file.

---

## Fail-loud guarantees

| Condition | Error |
|-----------|-------|
| `secrets_config.method` names no registered provider | `UnknownSecretsMethodError` (lists the registered methods) |
| published under the retired `pipelex.plugins` group | `RetiredPluginEntryPointGroupError` (names the plugins and the group each should move to) |
| two plugins register the same `method` | `DuplicateSecretsProviderError` (names both plugins) |
| `name` (`"secrets"`) in `plugins.disabled` | `CoreUnconditionalPluginDisabledError` |
| entry point raises while loading/registering | `BrokenPluginError` |
| optional SDK missing at use | `MissingDependencyError` (package + `pipelex[<extra>]` hint) |

The duplicate detection is the same fail-loud `_add` helper the inference/orchestrator/storage menus use — a conflict names both contributing plugins, so a collision between two installed secrets packages is unambiguous.

---

## Authoring an out-of-tree secrets plugin

A third-party secrets plugin is a distribution that:

1. implements a `SecretsProviderAbstract` subclass — `get_required_secret` / `get_optional_secret`, the versioned `*_specific_version` reads, and `set_secret_as_env_var` — deferring any SDK import into those methods (import-light);
2. defines a plugin class (`name`, `targets_api`, `register`) whose `register` calls `add_secrets_provider(method="<token>", factory=...)` — and nothing else;
3. advertises itself under the `pipelex.plugins.kernel` entry-point group — a secrets provider is kernel-layer, and the group is how a plugin declares that (see [Inference Backend Plugins](inference-backend-plugins.md#shipping-it-as-an-out-of-tree-plugin) for what each group means and what happens if you pick the wrong one; the pre-split `pipelex.plugins` group is retired and now fails startup):

```toml
# pyproject.toml of your plugin package
[project.entry-points."pipelex.plugins.kernel"]
vault_secrets = "pipelex_secrets_vault.plugin:VaultSecretsPlugin"
```

Installing the distribution makes the method selectable (`secrets_config.method = "vault"`); uninstalling removes it. No core change, no central registration list — *presence* is the source of truth. A discovered plugin can be quarantined without uninstalling via the `plugins.disabled` denylist (matched against the entry-point name *before* load, so a broken install can still be disabled to recover startup — see [Inference Backend Plugins](inference-backend-plugins.md) for the shared discovery/denylist machinery).

Use `pipelex plugins list` to see every discovered plugin, the entry-point group it was found under, what each contributed, and its denylist state. The **Group** column is the first thing to read when a plugin is missing: a built-in shows `—`, and an external plugin that resolved to the wrong layer shows it there.

---

## Related

- [Storage Provider Plugins](storage-provider-plugins.md) — the sibling config-selected-singleton seam this one mirrors; its `gcp` arm reads the secrets provider at boot
- [Secrets Provider Injection](../advanced/secrets-provider-injection.md) — supplying a custom secrets provider from your own vault or secrets manager
- [Inference Backend Plugins](inference-backend-plugins.md) — the per-call sibling seam and the shared discovery/denylist machinery
- [Orchestrator Plugins](orchestrator-plugins.md) — the other per-call seam, plus the hub-slot mechanism these config-selected seams deliberately avoid
- [Error Model](error-model.md) — how these errors render and dereference
- [Architecture Overview](architecture-overview.md)


## Error Model

# Error Model

In Pipelex, an error is **data**, not a control-flow accident. Every failure is classified once — at the layer that knows the most about it — and that classification travels intact to every consumer: the human reading a Rich panel, the agent parsing JSON, a distributed worker's retry engine, and the HTTP adapter picking a status code.

This page covers the contract that makes that possible: the `ErrorReport` schema, the classification enums, how inference workers classify SDK exceptions, how classification survives every wrapping layer, and how it survives serialization across a distributed worker boundary.

---

## Design Principle

Three rules hold across the codebase, and everything else builds on them.

**Single-rooted hierarchy.** Every custom exception inherits from `PipelexError` (`pipelex/base_exceptions.py`). There is one root, so one `to_error_report()` contract covers the whole tree.

**Classify at the source, never lose it.** The layer that catches a third-party exception knows the most about it. It classifies there. Every layer above is a *wrapper* — it adds context (pipe code, stack) but inherits the classification rather than re-deriving or discarding it.

**No broad catches in business logic.** `except Exception` is allowed only at CLI entry points and async task roots. Ruff rule `BLE001` enforces this — an unexpected exception crashes loudly instead of being silently swallowed.

!!! info "Why classify, instead of just propagating the exception?"
    A raw `openai.RateLimitError` tells a Python `except` clause what to catch, but it does not tell a distributed worker's retry engine whether to retry, the HTTP adapter which status to emit, or an agent whether the failure is the user's fault. Classification turns an exception into a decision input that every consumer can act on uniformly.

---

## The Layer Model

An error rises through a series of layers. Each layer has exactly one job.

| Layer | Role | What it does with errors |
|-------|------|--------------------------|
| **5 — CLI entry points** | `pipelex` / `pipelex-agent` commands | Catch, format for human (Rich) / agent (JSON·MD) / HTTP |
| **4 — CLI factories** | `cli_factory.py`, `agent_cli_factory.py` | Catch setup errors, route to handlers |
| **3 — Pipeline runner** | `PipelexMTHDSProtocol.execute()` | Catch + wrap as `PipelineExecutionError` |
| **2 — Pipe router / operators** | `PipeRouter`, pipe operators | Catch + wrap with pipe context (`pipe_code`, `pipe_stack`) |
| **1 — Workers / SDK calls** | `pipelex/providers/*/` | **Catch the SDK exception → classify → raise `CogtError`** |
| **0 — Third-party SDKs** | OpenAI, Anthropic, Google, … | Raise raw, untyped provider exceptions |

Classification happens once, at **Layer 1**. Layers 2–5 are wrappers: they attach context as they catch and re-raise, but the `error_category`, `error_domain`, `model`, and `provider` set at Layer 1 reach Layer 5 unchanged (see [Cause-Chain Enrichment](#cause-chain-enrichment)).

---

## ErrorReport — the Serialization Schema

`ErrorReport` (`pipelex/base_exceptions.py`) is the single source of truth for error serialization. It is a frozen Pydantic model with `extra="forbid"`.

| Field | Type | Meaning |
|-------|------|---------|
| `error_type` | `str` | The exception class name |
| `message` | `str` | Human-readable message |
| `title` | `str` | Stable human-readable summary — the RFC 7807 `title` |
| `type_uri` | `str` | Per-class documentation URI — the RFC 7807 `type` |
| `error_category` | `str \| None` | `InferenceErrorCategory` value (inference errors only) |
| `error_domain` | `str \| None` | `ErrorDomain` value — `input` / `config` / `runtime` |
| `retryable` | `bool \| None` | Whether a retry could succeed |
| `user_action` | `UserAction \| None` | Typed advice — `kind` + free-form `detail` |
| `model` | `str \| None` | Model handle, when the failure is attributable to one |
| `provider` | `str \| None` | Backend name, when attributable |
| `provider_metadata` | `ProviderErrorMetadata \| None` | SDK metadata — status code, request id, `retry_after` |
| `validation_errors` | `list[ValidationErrorItem] \| None` | Structured per-error diagnostics on a bundle-validation failure (`ValidateBundleError` only) |

`PipelexError.to_error_report()` is the entry point. `to_dict()` serializes, dropping `None` fields; `from_dict()` is its strict inverse.

### The identity triple, and why renaming an error class is a wire break

`error_type`, `title` and `type_uri` are the three identity fields on every report. `title` and `type_uri` are *presentation*, and each has a declaration hatch — set `_declared_title` or `_declared_type_uri` directly in a subclass body and that value is used verbatim instead of the auto-derived one (inheritance is deliberately bypassed via `cls.__dict__`, so a parent's curated title never captures its subclasses).

`error_type` has no such hatch: it is `type(self).__name__`, the Python class name with no indirection. That makes it the **machine contract** — consumers outside this repo `switch` on that string. Renaming an error class therefore breaks them *silently*: their build stays green and the branch simply stops matching, falling through to a generic error path.

The guard against that is a committed snapshot of the full `(error_type, title, type_uri)` set at `tests/data/errors/error_identity.txt`, regenerated with `make generate-error-identity` (alias `make gei`) and gated by `tests/unit/pipelex/errors/test_error_identity_snapshot.py`. A rename cannot land without producing a reviewable one-line-pair diff on that file at the moment it is made — which is also the moment to plan the matching consumer updates.

### `validation_errors` — structured bundle-validation diagnostics

A bundle-validation failure (`ValidateBundleError`) aggregates per-error data across stages and projects it onto `validation_errors` as a list of typed `ValidationErrorItem`s, so the structured error report an HTTP API surfaces carries machine-mappable diagnostics (not just a single `detail` string). Each item's `category` is one of the **closed** `ValidationErrorCategory` set:

- `blueprint_validation` — interpreter / blueprint-validation faults. A blueprint-stage `PipeValidationError` raised *inside* a pydantic model validator (e.g. the PipeBatch `input_item_name` == `input_list_name` collision, or the SubPipe `batch_over` == `batch_as` collision — both `batch_item_name_collision`) is wrapped by pydantic as a `value_error`; the blueprint categorizer unwraps it (`ctx["error"]`) so the item keeps its structured `error_type` and `pipe_code` / `domain_code` locators instead of degrading to the no-`error_type` residual. The item stays in `blueprint_validation` (not `pipe_validation`) because the fault genuinely surfaced at the parse boundary, before any pipe was instantiated — only the `error_type` is recovered, not the stage. This category also serves as the **last-resort residual**: a parse-level failure (a TOML-syntax error, an empty blueprint, a bundle-elaborator failure) is raised with only a message and no categorized data, so when *nothing else* produced an item the builder projects that message as one `blueprint_validation` item (no `error_type`, no `source` — the bundle could not become a blueprint at all).
- `pipe_factory` — pipe-factory failures (e.g. a missing concept).
- `pipe_validation` — pipe/concept validation (missing input variable, type mismatch).
- `dry_run` — the **residual** dry-run failure (`DryRunError` / `PipeRunError`) with no structured locator. It is projected as one message-only item **only when no categorized error has data**. It is graph-level, so it typically carries **no `source`**.

Together the two residuals make the **structured-info invariant total**: every invalid verdict carries a non-empty `validation_errors[]`, never a bare message. The builder tries the channels in order — categorized data, then the `dry_run` residual (the more specific channel), then the `blueprint_validation` fallback — and emits exactly one residual only when no earlier channel produced an item.

Besides `category` and `message`, each item carries whatever identity fields its stage produced — `error_type`, `pipe_code`, `concept_code`, `domain_code`, `field_path`, `field_name`, `variable_names`, `missing_concept_code`, `declared_concepts`, and a `source` (the declaring file path, or the per-content source the in-memory load path was given) that hands a consumer the owning file for cross-file diagnostic placement. When the error has a deterministic remedy, the item also carries a [`suggested_fix`](#suggested_fix-structured-deterministic-fixes).

**Signatures are never an error.** An unimplemented `PipeSignature` reached during validation is a *runnability fact*, not a validation failure: the validator no longer raises on it. The assembled library's outstanding signatures ride the validation report's `pending_signatures`, and `is_runnable = not pending_signatures`. `allow_signatures` is a sweep-mechanics flag only (whether signature pipes are mock-run and listed in `validated_pipes`) — it does not change the verdict, so strict ≡ lenient in the report body. The "is this a failure?" decision moves to the consumer: the CLI exits non-zero on `not is_runnable` unless `--allow-signatures`; the HTTP caller reads `is_runnable`. (The **execute/run** path is different: running a stub still raises `PipeSignatureNotExecutableError`.)

**Host-wiring guards are programmer errors, not content verdicts.** `validate_bundle`'s "provide exactly one of `mthds_contents` / `mthds_file_path`" guard and `resolve_crate_from_contents`'s `mthds_sources`-length-mismatch guard raise `PipelexUnexpectedError` (→ 500, redacted under STRICT), not `ValidateBundleError` — a caller wiring bug must not be reported as if the submitted bundle were invalid. The empty-`mthds_contents` guard stays caller-facing (it can legitimately reflect an end user submitting no bundles).

`ValidationErrorItem` and the builder are the single source of truth across surfaces: `build_validation_error_items()` (`pipelex/pipeline/validation_errors.py`) is called by both `ValidateBundleError.to_error_report()` (the API path) and the agent CLI's `extract_validation_errors()` (the CLI JSON envelope), so the two structured shapes cannot drift. The item lives in `pipelex/base_exceptions.py` alongside `ErrorReport` — not next to the source error-data models — because `ErrorReport` references it as a typed field and the root exceptions module must not import the `pipelex.core` error modules.

`validation_errors` is one of the fields kept under STRICT disclosure (it is in `_STRICT_KEPT_FIELDS`): the items describe the caller's *own* submitted bundle, not server internals, so redacting them would gut the hosted path's diagnostics.

```python
report = exc.to_error_report()
report.to_dict()         # {"error_type": "LLMCompletionError", "message": "...", ...}
ErrorReport.from_dict(d) # strict inverse — raises ValidationError on a malformed dict
report.http_status       # 422 / 429 / 500 — for HTTP adapters
```

!!! warning "`ErrorReport` is `extra="forbid"`"
    `from_dict()` rejects unknown keys, so it is the strict inverse of `to_dict()`. A report dict that crosses a serialization boundary and fails validation on the way back is an internal contract bug — the writer and the reader share the schema within one deploy. A cross-boundary recovery helper that rebuilds a report (e.g. a distributed-worker bridge) is expected to catch that `ValidationError` and synthesize a fallback report so failure-webhook delivery stays intact while keeping the contract bug visible; any other caller of `from_dict()` should treat the validation failure as a bug to fix.

### `suggested_fix` — structured deterministic fixes

When a validation error has a deterministic remedy, its `ValidationErrorItem` carries a `suggested_fix` — a `SuggestedFix` (`pipelex/suggested_fix.py`, deliberately stdlib+pydantic-only so `pipelex.base_exceptions` can import it without a cycle; naming is brand-neutral, fixes are a language-level concept):

- `fix_code` — the kebab-case rule id (e.g. `match-sequence-output`). The planner's `KNOWN_FIX_CODES` set is the validation set for user-facing rule filters (`--select` / `--ignore`); an unknown code is rejected loudly, never lenient-ignored, because a typo'd filter selects *behavior*.
- `description` — human-readable statement of the change.
- `safety` — `safe` fixes may be auto-applied; `unsafe` ones require explicit opt-in.
- `source` — the file the ops target, when known (multi-file libraries). An applier must only apply ops to the file they target.
- `ops[]` — the fix itself, as **semantic TOML patch ops** addressed by table path (`FixOpKind`: `set_key`, `ensure_table`, `delete_key`, `delete_table`, `rename_table_key`; each op's `table_path` follows the same conventions as the items' `field_path`). The ops are the machine contract; any rendered diff or `💡 Suggested fix:` line is presentation.

The **fix planner** (`pipelex/pipeline/fixes/planner.py`) translates enriched typed error data into `SuggestedFix` payloads — pure functions keyed strictly on `error_type` + structured fields, never on message strings. Each rule fires only when its enrichment is present (set only at the raise sites that know the correct value), so the same error type raised elsewhere without enrichment is structurally suppressed. The planner runs inside `build_validation_error_items()`, so every consumer of the validation report — CLI, API, MCP — sees fixes with zero extra plumbing.

Applying fixes is the runtime's job too: the **applier** (`pipelex/pipeline/fixes/applier.py`) mutates a tomlkit DOM in place per op (guarded — an op whose target table is absent is skipped and reported, never raised) and then reflows the whole file to canonical MTHDS style, and the **convergence loop** (`pipelex/pipeline/fixes/fix_loop.py`) runs validate → apply SAFE fixes → re-validate to a fixed point, reporting non-convergence loudly. The user-facing surface is [`pipelex fix bundle`](../tools/cli/fix.md).

On the hosted API the same payload rides the wire verbatim as `validation_errors[].suggested_fix`; how it appears in HTTP error responses is documented on the API side, in `pipelex-api`'s `docs/error-responses.md` → "Suggested fixes".

---

## Classification Enums

Two `StrEnum`s drive every downstream decision.

### InferenceErrorCategory

Defined in `pipelex/cogt/exceptions.py`. Drives retry decisions — `is_retryable` is `True` only for `TRANSIENT`.

| Category | Meaning | Retryable | Typical cause |
|----------|---------|-----------|---------------|
| `TRANSIENT` | A brief, self-correcting failure | ✅ | Rate limit, 5xx, connection blip |
| `CONFIGURATION` | The setup is wrong | ❌ | Bad API key, missing backend |
| `CONTENT` | The input or prompt is wrong | ❌ | Content-policy violation, bad prompt |
| `CAPACITY` | Account quota / billing exhausted | ❌ | `insufficient_quota`, HTTP 402 |
| `AMBIGUOUS` | Outcome unknown — may have committed | ❌ | Connection dropped mid-request |
| `UNKNOWN` | Could not classify | ❌ | Unrecognized inner exception |

```python
class InferenceErrorCategory(StrEnum):
    TRANSIENT = "transient"
    # ... CONFIGURATION, CONTENT, CAPACITY, AMBIGUOUS ...
    UNKNOWN = "unknown"

    @property
    def is_retryable(self) -> bool:
        match self:
            case InferenceErrorCategory.TRANSIENT:
                return True
            case _:  # all other categories
                return False
```

!!! info "`AMBIGUOUS` vs `UNKNOWN`"
    `AMBIGUOUS` means the *error type is known* but the operation may or may not have committed — a blind retry is unsafe for a non-idempotent call. `UNKNOWN` means classification itself failed. Both are non-retryable, for different reasons.

### ErrorDomain

Defined in `pipelex/base_exceptions.py`. Set as a class-level attribute on the exception, drives HTTP status.

| Domain | Meaning | HTTP status | Who can fix it |
|--------|---------|-------------|----------------|
| `INPUT` | Caller sent something it can fix | **422** | The caller |
| `CONFIG` | Environment / configuration change needed | **500** | The operator |
| `RUNTIME` | A failure during execution | **500** | Depends on the cause |

`error_domain_to_http_status()` is the pure mapping table. `ErrorReport.http_status` layers one rule on top: a provider 429 (`provider_metadata.status_code == 429`) takes precedence over the domain, so the API can emit a `Retry-After` header.

```python
class PipelexConfigError(PipelexError):
    error_domain = ErrorDomain.CONFIG     # class-level — every instance carries it
```

---

## Worker Classification

Layer 0 → Layer 1. Every inference worker under `pipelex/providers/*/` catches its SDK's typed exceptions and re-raises a categorized `CogtError`.

### The Uniform Shape — Extract / Classify / Render

Every inference worker's SDK-exception handler collapses to a three-step pipeline: **Extract** turns the SDK exception into a provider-blind `ProviderErrorMetadata`, **Classify** maps that metadata to a category + user-action, and **Render** picks the `CogtError` subclass to raise.

```python
except (APIError, APIConnectionError, APITimeoutError) as exc:
    metadata = extract_openai_metadata(exc)
    classification = classify_inference_error(metadata)
    raise render_inference_error(
        metadata=metadata,
        classification=classification,
        family=InferenceErrorFamily.LLM,
        model_desc=self.inference_model.desc,
        model_handle=self.inference_model.name,
    ) from exc
```

The three steps live in three modules. Only the per-provider Extract functions stay plugin-local; Classify and Render are single shared functions.

| Module | Step | What it owns |
|--------|------|--------------|
| `pipelex/cogt/inference/error_classification.py` | Extract | `ProviderErrorMetadata`, `SDKErrorEnvelope`, `UserAction`, `UserActionKind`, the 12 `extract_*_metadata` functions, plus pure discriminators (`is_quota_exhaustion`, `is_content_policy_violation`, `is_network_error`) exposed as `@property` on the metadata |
| `pipelex/cogt/inference/error_classify.py` | Classify | `classify_inference_error()` — provider-blind mapping from `ProviderErrorMetadata` → `ClassificationResult(category, user_action_kind, is_model_not_found)` |
| `pipelex/cogt/inference/error_render.py` | Render | `render_inference_error()` — picks the `CogtError` subclass from `InferenceErrorFamily` plus `is_model_not_found` (e.g. `LLMModelNotFoundError` vs `LLMCompletionError`) |

Provider-specific nuance is normalized away in Extract (e.g. Google's `code` becomes `status_code`; AWS Bedrock error codes are mapped to HTTP statuses), so Classify has no provider branching. HTTP status drives classification; status-less errors dispatch on the SDK exception type name. The `tests/unit/pipelex/cogt/inference/test_provider_classification_parity.py` meta-test walks every `ProviderName` against the extract-fn registry so adding a new provider without wiring it fails fast.

### ProviderErrorMetadata and UserAction

Every raised inference error carries structured SDK metadata and typed advice.

```python
class ProviderErrorMetadata(BaseModel):
    provider: str
    sdk_exception_type: str
    status_code: int | None = None
    request_id: str | None = None
    retry_after_seconds: float | None = None
    provider_error_code: str | None = None
    body: Any | None = Field(default=None, exclude=True)   # may carry secrets
```

!!! warning "`body` is excluded from serialization"
    The raw provider response `body` can carry account ids, billing details, or credential fragments. It is held in-process but `exclude`d from every serialized form — CLI JSON, agent output, and any serialized worker payload.

`UserAction` pairs a discrete `UserActionKind` (`WAIT_AND_RETRY`, `CHECK_BILLING`, `CHECK_CREDENTIALS`, `CHANGE_INPUT`, `CHANGE_MODEL`, `CONTACT_SUPPORT`, `UNKNOWN`) with a free-form `detail` string — so the CLI can render consistent guidance while keeping provider-specific text.

### The `instructor` Unwrap

On structured-generation paths, `instructor` wraps the real SDK exception in an `InstructorRetryException`. `extract_underlying_sdk_exception()` recovers it, so it routes through the same per-provider categorization as the plain-text path. A genuinely unrecognized inner exception (e.g. a `pydantic.ValidationError` from a schema mismatch) lands in `UNKNOWN` rather than being mis-labelled as a `CONTENT`-policy violation.

### Model and Provider Attribution

Inference-failure leaf errors (`LLMCompletionError`, `ImgGenGenerationError`, …) are raised deep inside a plugin and do not know which model handle invoked them. Each worker family fills that in at its public-method chokepoint:

```python
def fill_model_and_provider(self, model_handle: str | None, *, backend_name: str | None) -> None:
    """Fill model_handle / backend_name from the worker, only when still unset."""
```

---

## Cause-Chain Enrichment

A wrapper exception — `PipeRunError` → `PipeRouterError` → `PipelineExecutionError` — carries no `error_category` of its own. `to_error_report()` enriches the report from the `__cause__` chain, so the inference classification survives every wrapping layer.

```python
def _enrich_error_report_from_cause(self, report: ErrorReport) -> ErrorReport:
    cause = self.__cause__
    if not isinstance(cause, PipelexError):
        return report
    cause_report = cause.to_error_report()
    return ErrorReport(
        error_type=report.error_type,                                  # keep own identity
        message=report.message,
        error_category=report.error_category or cause_report.error_category,
        error_domain=report.error_domain or cause_report.error_domain,
        # ... retryable, user_action, model, provider, provider_metadata ...
    )
```

A wrapper keeps its own `error_type` and `message` but inherits every classification field it does not set itself.

!!! warning "Overrides must call the enrichment helper"
    A `to_error_report()` override on a subclass **must** end with `self._enrich_error_report_from_cause(report)`. Otherwise that subclass becomes a black hole that drops the cause's classification. A cyclic-`__cause__` guard ensures a malformed chain can never turn error reporting into a `RecursionError`.

---

## Crossing a Distributed Worker Boundary

The error model is built to survive serialization. Because `ErrorReport` round-trips through `to_dict()` / `from_dict()`, a failure that happens on a remote worker can reach the submitting process with its full classification intact — not just a message string.

The runtime itself stays transport-agnostic: the machinery that carries an error across a worker boundary ships in the **host-runtime plugin** for each distributed backend, not in core. A backend plugin is responsible for three things.

**Packing.** Convert a `PipelexError` into the transport's failure type and stash `to_error_report().to_dict()` in its details payload, so worker and submitter code keep the full classification rather than a bare message. The same step derives the transport's retry decision from `InferenceErrorCategory.is_retryable`.

**Recovering.** On the submitter side, walk the returned failure, pull the packed dict, and rebuild the `ErrorReport`. Recovery is **total**: when no report dict is found — a non-Pipelex exception, a worker crash, a timeout — the plugin synthesizes a fallback report so the recovery path always has structured classification to surface.

**A fail-safe floor.** Ensure a domain error that escapes the conversion path fails the unit of work *terminally* rather than hanging. In a durable-execution system the default for an unconverted exception might be to retry forever, so "convert all the errors we know about" is not enough — the floor must hold for the errors, and the code paths, that nobody enumerated.

**Net effect:** a pipe failing on a remote worker reaches the CLI and HTTP adapters with the *same* `error_category` / `retryable` / `model` / `provider` / `user_action` as the identical failure run locally — and a failure that escapes conversion fails loud and bounded instead of hanging.

See [Runtime Bridge & Transport](./runtime-bridge-and-transport.md) for the boundary these converters span; the per-backend converters themselves live in the host-runtime plugins.

---

## Interfaces

### CLI

The agent CLI (`pipelex-agent`) emits a structured error to **stderr**, markdown by default and JSON with `--error-format json`. When `--error-format` is omitted it **inherits the value of `--format`** (the success-output flag) — so `--format json` still flips both as it did before the split. Both exit with code 1.

| Command | Error output |
|---------|--------------|
| `run`, `validate`, `init`, `models`, `check-model`, `doctor` | Markdown (default) or JSON via `--error-format` (or via `--format`, which `--error-format` inherits) |
| `inputs`, `concept`, `pipe`, `accept-gateway-terms` | JSON only |
| `fmt`, `lint` | Native `plxt` output (subprocess passthrough); falls back to JSON only when the `plxt` binary itself is missing |

The human CLI (`pipelex`) renders a Rich error panel — red banner, structured fields, the `user_action` tip, doc/Discord links — through the shared `display_error_panel()` helper in `pipelex/cli/error_handlers.py`.

#### Validate exit-code policy (0 / 1 / 2)

The `validate` surface — both the bare `pipelex validate {bundle,method,pipe}` group and the agent CLI's `pipelex-agent validate` — exits with **three** codes that mirror the hosted `/validate` 200-verdict-vs-non-2xx-no-verdict split:

| Exit | Class | Condition |
|------|-------|-----------|
| `0` | valid | `is_valid` — including valid-but-not-runnable **with** `--allow-signatures` |
| `1` | negative verdict | a produced "no": an invalid bundle (`ValidateBundleError`), or valid-but-not-runnable **without** `--allow-signatures` (a strict signature breach) |
| `2` | no verdict | the CLI could not produce a verdict — bad args, an unresolvable target (no `.mthds` in a directory, a missing file, an unknown/ambiguous pipe code), or a setup/internal error during validate |

**The verdict lives in the structured `is_valid` field, not the exit code.** The exit code is a convenience signal for naive shell/CI/Makefile use (`set -e`, `cmd && next`, `if cmd; then`); machine consumers (hooks, the Codex hook, runners) MUST read `is_valid` (and `error_domain`) from the JSON for their block/warn decisions rather than branching on the exit code. Decoupling the verdict from the exit code is what keeps any future exit-code change non-breaking. The 1-vs-2 split is also additive for flat consumers: both stay non-zero, so anything that only tests zero-vs-non-zero is unaffected.

Implementation: the agent CLI threads `exit_code` through `agent_error(...)` (`agent_output.py`, default 1); the validate commands pass `exit_code=2` at every no-verdict site and keep the default 1 on the `ValidateBundleError` arm and the signature gate. The bare CLI sets the code directly via `typer.Exit(...)` in `cli/commands/validate/*` and via the `exit_code` parameter on `handle_model_choice_error` / `handle_model_availability_error` in `cli/error_handlers.py`. Shared boot handlers (`make_pipelex_for_cli`'s gateway/inference/telemetry/model-deck-preset paths) stay exit 1 — they are shared across `run`/`build`/`validate` and out of the validate-policy scope.

### API

`pipelex` is a library — there is no API server in the package. Downstream HTTP repos consume the `ErrorReport`:

- `error_domain_to_http_status(error_domain)` — pure domain → status table.
- `ErrorReport.http_status` — full property, layering the provider-429 passthrough on top.

A downstream FastAPI exception handler calls `ErrorReport.http_status` and is a trivial adapter — it must not redefine the mapping.

### Inputs and Outputs

**Inputs.** `to_error_report()` takes a live `PipelexError`. `ErrorReport.from_dict()` takes a `to_dict()` payload — strictly, raising `ValidationError` on drift. (A distributed-worker bridge adds a cross-boundary recovery helper that walks a returned failure's `__cause__` chain and rebuilds the report; it lives in the host-runtime plugin, not core.)

**Outputs.** `to_error_report()` returns an `ErrorReport`; `to_dict()` returns a `None`-free `dict`. Side effects: telemetry events emitted on pipeline failure at Layer 3; the agent CLI writes to stderr and raises `typer.Exit(...)` — code 1 by default, or the validate surface's 0/1/2 policy (see [Validate exit-code policy](#validate-exit-code-policy-0-1-2)).

---

## Architecture

```mermaid
flowchart TB
    SDK["Layer 0 — SDK exception<br/>(openai.RateLimitError)"]
    W["Layer 1 — Worker classifies<br/>is_quota_exhaustion_*() → CogtError<br/>+ InferenceErrorCategory + ProviderErrorMetadata"]
    WRAP["Layers 2-3 — Wrappers<br/>PipeRouterError → PipelineExecutionError<br/>(attach pipe context)"]
    REPORT["ErrorReport<br/>via to_error_report() + cause-chain enrichment"]

    SDK -->|"raise ... from exc"| W
    W -->|"raise ... from exc"| WRAP
    WRAP --> REPORT

    REPORT --> RICH["Human CLI<br/>Rich panel"]
    REPORT --> AGENT["Agent CLI<br/>JSON / Markdown"]
    REPORT --> HTTP["HTTP adapters<br/>.http_status"]

    W -.->|"pack on worker"| TEMP["Distributed worker bridge (plugin)<br/>report packed into transport details"]
    TEMP -.->|"recover on submitter"| REPORT

    classDef src fill:#fff3e0,stroke:#e65100,color:#000
    classDef cls fill:#e8eaf6,stroke:#3949ab,color:#000
    classDef out fill:#e8f5e9,stroke:#2e7d32,color:#000
    class SDK src
    class W,WRAP,REPORT,TEMP cls
    class RICH,AGENT,HTTP out
```

---

## Implementation

### Class Hierarchy

`PipelexError` is the single root. `CogtError` is the inference branch — it overrides `to_error_report()` to add `error_category`, `retryable`, `user_action`, `provider_metadata`, and reads `model_handle` / `backend_name` from the instance.

```
Exception
└── PipelexError                  base_exceptions.py — error_domain, user_action, to_error_report()
    ├── PipelexConfigError         → error_domain = CONFIG
    ├── PipelexSetupError          → error_domain = CONFIG
    ├── CogtError                  cogt/exceptions.py — error_category, provider_metadata
    │   ├── LLMCompletionError      ← per-instance category from the worker
    │   ├── ImgGenGenerationError   ← per-instance category
    │   ├── ModelNotFoundError      ← sibling family raised on provider HTTP 404
    │   │   ├── LLMModelNotFoundError / ImgGenModelNotFoundError
    │   │   └── ExtractModelNotFoundError / SearchModelNotFoundError
    │   └── ... (see worker classification) ...
    ├── PipelineExecutionError      pipeline/exceptions.py — error_domain = RUNTIME
    └── ... (one exceptions.py per package) ...
```

### Factory-time vs Runtime

| When | What carries metadata | How |
|------|----------------------|-----|
| **Class definition** | `error_domain`, `error_category` defaults, `user_action` defaults | Class-level attributes — one source of truth per exception type |
| **Raise time** | Per-instance `error_category`, `user_action`, `provider_metadata` | Constructor args — set by the worker that classified the failure |
| **Report time** | `model`, `provider`, cause-chain fields | `fill_model_and_provider()` at the worker chokepoint; `_enrich_error_report_from_cause()` on `to_error_report()` |

The "outcome" exceptions (`LLMCompletionError`, `ImgGenGenerationError`, `ExtractJobFailureError`, `SearchJobFailureError`) intentionally carry **no** class-level `error_category` — their category is genuinely per-instance, decided by the worker.

---

## Reference

### Quick-Ref

```python
# Produce a report from any PipelexError
report = exc.to_error_report()          # enriched from the __cause__ chain
payload = report.to_dict()              # None-free dict for serialization

# Consume a report
report.http_status                      # 422 / 429 / 500
report.user_action_detail()             # free-form advice text, or None
report.error_category                   # "transient" / "capacity" / ...

# Round-trip across a boundary
ErrorReport.from_dict(payload)           # strict inverse of to_dict()

# Retry decision
InferenceErrorCategory.TRANSIENT.is_retryable   # True — only TRANSIENT
```

### File → Purpose

| File | Purpose |
|------|---------|
| `pipelex/base_exceptions.py` | `PipelexError`, `ErrorReport`, `ErrorDomain`, `ValidationErrorItem`, `error_domain_to_http_status()` |
| `pipelex/pipeline/validation_errors.py` | `build_validation_error_items()` — shared CLI/API structured bundle-validation builder |
| `pipelex/cogt/exceptions.py` | `CogtError`, `InferenceErrorCategory` |
| `pipelex/cogt/inference/error_classification.py` | Extract — `ProviderErrorMetadata`, `SDKErrorEnvelope`, `UserAction`, `UserActionKind`, per-provider `extract_*_metadata` functions, pure discriminators |
| `pipelex/cogt/inference/error_classify.py` | Classify — `classify_inference_error()`, `ClassificationResult` |
| `pipelex/cogt/inference/error_render.py` | Render — `render_inference_error()`, `InferenceErrorFamily` |
| `pipelex/cogt/inference/provider_name.py` | `ProviderName` enum keying the extract-fn registry |
| `pipelex/providers/*/` | Per-provider inference workers — Layer 0 → 1 classification |
| `pipelex/pipeline/exceptions.py` | `PipelineExecutionError`, `PipeExecutionError` |
| `pipelex/cli/error_handlers.py` | Human CLI Rich panels — `display_error_panel()` |
| `pipelex/cli/agent_cli/commands/agent_output.py` | Agent CLI JSON / markdown delivery |

### Behavior Summary

| Scenario | Behavior |
|----------|----------|
| Rate limit hit | `TRANSIENT` → retryable; transport retry honors `Retry-After` |
| Quota / billing exhausted | `CAPACITY` → non-retryable; `UserAction(CHECK_BILLING)` |
| Bad API key | `CONFIGURATION` → non-retryable; `error_domain = CONFIG` → HTTP 500 |
| Model or deployment not found (provider HTTP 404) | Raises a dedicated `*ModelNotFoundError` sibling (`LLMModelNotFoundError`, `ImgGenModelNotFoundError`, `ExtractModelNotFoundError`, `SearchModelNotFoundError`); operator re-raises `PipeOperatorModelAvailabilityError` |
| Content-policy violation | `CONTENT` → non-retryable; `UserAction(CHANGE_INPUT)` |
| LLM returns schema-mismatched JSON | `instructor` re-asks; if exhausted → `UNKNOWN` |
| Connection dropped mid-request | `AMBIGUOUS` → non-retryable (outcome unknown) |
| Wrapper exception (no own category) | Inherits cause's classification via enrichment |
| Failure on a distributed worker | `ErrorReport` recovered from the transport's serialized details — same classification as local |
| Worker exception with no `ErrorReport` | Synthesized fallback report — `error_domain = RUNTIME` |

---

## Next Steps

- [Pipe Routing & Execution](./pipe-routing-and-execution.md) — the layer model errors rise through
- [Runtime Bridge & Transport](./runtime-bridge-and-transport.md) — the process boundary the error bridge spans (the per-backend error converters live in the host-runtime plugins)
- [Cogt Configuration](../configuration/config-technical/cogt-config.md) — `transport_max_retries` and the Tier 1 retry policy
- [Agent CLI](../tools/cli/agent-cli.md) — the JSON / markdown error contract


## Runtime Bridge & Transport

# Runtime Bridge & Transport

This page is for contributors working on Pipelex internals. For the capability overview, see the user-facing [Distributed Execution](../distributed-execution/index.md) page instead.

Pipelex runs a pipe either **direct** (everything in one Python process) or **distributed** on a host runtime (a separate worker process, potentially on another machine). Both paths reach pipe execution through the same framework-agnostic **runtime bridge**. This page documents that bridge backend-neutrally: the boundary data types, how a library snapshot and working memory cross a process boundary, and how concurrent runs stay isolated. The concrete realizations — Temporal-backed durable execution and the Mistral Workflows integration — are commercial Pipelex platform capabilities; their backend-specific topology lives in their own plugin repos.

!!! info "A commercial capability rides on this seam"
    Pipelex's own host-runtime embedding — running a pipe as durable, distributed workflows — is part of Pipelex's [workflow-orchestration offer](https://pipelex.com/products#durable-execution). This page documents the **open seam** and the **open boundary types**; the cross-process plumbing that drives them is not part of open core (see [Where open core ends](#where-open-core-ends) below). For the orchestrator extension point a third-party backend compiles against, see [Orchestrator Plugins](./orchestrator-plugins.md).

---

## Direct vs distributed, one bridge

| Mode | Where pipes run | How the PipeJob travels |
|------|-----------------|--------------------------|
| **Direct** | All pipes in the same process. Library, class registry, and pipe resolution are shared in-memory. | Stays in-process; no serialization. |
| **Distributed** | The PipeJob is serialized and sent to a worker — a separate process. | Crosses the boundary as structured JSON, carrying a serializable library snapshot. |

Both modes share the same pipe definitions, library loading, and controller logic. The only difference is *where* a pipe runs and how its `PipeJob` gets there. Distributing execution across separate processes introduces problems that don't exist in-process — those problems, and their solutions, are this page's subject. For the routing layer that sits above the bridge (how a controller dispatches its child pipes in each mode), see [Pipe Routing & Execution](./pipe-routing-and-execution.md).

---

## The boundary data types

Three open-core Pydantic models are everything that crosses a process boundary.

### PipeJob — the unit of execution

`PipeJob` (`pipelex/pipe_run/pipe_job.py`) carries everything needed to run a single pipe:

| Field | Purpose |
|-------|---------|
| `pipe` | The resolved pipe object (concrete operator or controller). |
| `working_memory` | Runtime data store — typed `Stuff` objects keyed by variable name. |
| `working_memory_raw` | Raw JSON dict of working memory for deferred hydration (cross-process only). |
| `pipe_run_params` | Execution config: run mode (LIVE/DRY), output multiplicity, pipe stack for cycle detection. |
| `job_metadata` | Pipeline run ID, user ID, tracing context. |
| `library_crate` | Serializable library snapshot for distributed execution. |

### PipeOutput — the result that comes back

`PipeOutput` (`pipelex/core/pipes/pipe_output.py`) carries the run result. Like `PipeJob`, it has both a typed `working_memory` and a `working_memory_raw` dict — the raw field is how a result travels back across a boundary without requiring class resolution at every hop (see [Deferred hydration](#deferred-hydration)).

### LibraryCrate — a serializable library snapshot

A `LibraryCrate` (`pipelex/libraries/library_crate.py`) is a flat, serializable snapshot of the library — the pipes and concepts a worker needs to run the job:

```python
class LibraryCrate(BaseModel):
    concepts: dict[str, ConceptBlueprint | str]  # concept ref -> blueprint or description
    pipes: dict[str, PipeBlueprintUnion]          # pipe ref -> blueprint
    domains: dict[str, DomainBlueprint]           # domain code -> domain metadata
    source_map: dict[str, str]                    # ref -> source file (error traceability)
    fingerprint: str                              # SHA256 of serialized content
```

Domain is encoded in the dictionary keys (e.g., `scoring.WeightedScore`, `scoring.compute_score`), not in a structural container. `source_map` lets error messages trace back to origin files. The `fingerprint` enables idempotent loading — a worker that already loaded a crate with the same fingerprint skips the reload. The crate is a plain Pydantic model, so a host runtime serializes it as structured JSON in the job input — fully inspectable, not opaque bytes.

---

## The three cross-process challenges

Sending a pipe run to a separate worker process introduces three challenges that single-process execution never faces:

1. **Pipe resolution** — Controllers call `get_required_pipe()` to resolve child pipes. The worker's library must contain those pipes. → solved by **LibraryCrate propagation**.
2. **Dynamic class deserialization** — `.mthds` bundles generate Python classes at runtime (e.g., `RawText` inheriting from `TextContent`). A receiving process must have those classes registered before it can deserialize `WorkingMemory`. → solved by **deferred hydration**.
3. **Concurrent isolation** — One worker may run many jobs at once. Two jobs that each define a concept named `Result` with different structures must not share a class registry or library. → solved by **per-call scoping**.

---

## LibraryCrate propagation

The submitter builds a `LibraryCrate` from its loaded library (`pipelex/libraries/library_crate_factory.py`) and attaches it to the `PipeJob`. On the worker, the job's entry point loads the crate into a scoped library, registering the bundle's dynamic concept classes, so `get_required_pipe()` resolves every pipe at every level of nesting. Because a distributed run can fan out into nested jobs (a controller's child pipes), every nested job carries the same crate; loading is idempotent via the fingerprint, so re-receiving a crate already loaded on a worker is a no-op.

---

## Deferred hydration

### The chicken-and-egg problem

A host runtime's data converter deserializes a payload (a job input, a return value) before any bundle code has run — but reconstructing a typed `WorkingMemory` needs the bundle's dynamic concept classes registered, and those only get registered when the `LibraryCrate` is loaded *inside* the job. Two deserialization gaps follow:

1. **Input** — the `PipeJob` input contains `WorkingMemory` whose dynamic classes don't exist yet on the worker.
2. **Output** — a nested job's `PipeOutput` return value contains `WorkingMemory` with dynamic classes, and the parent's data converter typically runs outside the job's scope, so it can't reach the per-job class registry.

### The solution: carry working memory raw

Both `PipeJob` and `PipeOutput` carry a `working_memory_raw` field — a plain JSON dict — alongside the typed `working_memory`. The raw dict needs no class resolution to cross a boundary, which decouples transport from class registration.

- **Input dehydration** — before dispatch, the working memory is moved into `working_memory_raw` (a plain dict produced by `WorkingMemory.dump_for_transport()`, which stays in open core at `pipelex/core/memory/working_memory.py`). After the worker loads the crate, the job's entry point hydrates the raw dict back to a typed `WorkingMemory`, inside the job's scope.
- **Output dehydration** — before a nested job returns, its working memory is dehydrated to `working_memory_raw`. A **parent job** that consumes the nested output (a controller combining branch results — PipeParallel, PipeBatch) rehydrates it on receive, inside its **own already-loaded scope**: the parent's library carries the same crate (checked by fingerprint via `is_crate_loaded`), so hydration rebinds the nested output to the parent's own dynamic classes — the identity the parent's models reference. **Delivery** workers never load the crate — they render from the raw dict, only best-effort hydrating the main stuff via globally registered classes (falling back to a generic dict render when the class isn't registered), so they stay crate-free. The **top-level submitter** rehydrates too, opening a fresh scoped library when it hasn't loaded the crate itself.

The worker-side hydration helper (`pipelex/runtime_bridge/primitives/hydration.py`) is open core: it iterates the raw dict, uses `StuffContentFactory` to rebuild typed content from the registered classes, and preserves aliases. It also guards against cross-exec class-identity mismatches (`_validate_as_known_class` round-trips items through `model_dump()`): the dynamic classes a worker re-execs from the crate are new Python identities sharing a name with whatever a data converter eagerly rebuilt, and a naive `model_validate` would reject the older instance.

### Boundary-by-boundary (backend-neutral)

| Boundary | Direction | What crosses | Hydrate on receive? |
|----------|-----------|--------------|----------------------|
| Submitter → top worker job | input | `PipeJob` (`working_memory_raw` + `library_crate`) | Yes — worker loads crate, then hydrates in scope |
| Parent job → nested job | input | Same `PipeJob` shape (crate re-loaded idempotently) | Yes — in the nested job's own scope |
| Nested job → parent | output | `PipeOutput` (`working_memory_raw`) | Yes — in the parent's own scope, whose library already carries the crate (fingerprint match), so branch instances rebind to the parent's class identities |
| Parent job → delivery | result arg | `PipeOutput` (`working_memory_raw`, no crate) | **No** — delivery renders from raw, best-effort local hydration of the main stuff using only globally registered classes |
| Top worker job → submitter | output | `PipeOutput` (`working_memory_raw`) | Yes — the submitter rehydrates, opening a fresh scoped library if a crate is available |

Because delivery renders from the raw dict (`pipelex/pipe_run/delivery_executor.py` branches on which field is populated), a delivery worker never needs the crate loaded — which is the precondition for true distributed execution. Dynamic concepts lose typed rendering on the delivery worker (a generic field-walking fallback still produces readable JSON/HTML), but the worker stays crate-free.

!!! note "Where the transport prep lives"
    Moving working memory into `working_memory_raw` for a job, dehydrating a result before return, and the submitter-side rehydration are cross-process plumbing, and are **not** part of open core. The *raw field itself*, `dump_for_transport`, and the worker-side hydration helper stay **open core**, because they are host-agnostic and are also exercised by the open `pipelex-api` runner.

---

## Per-call scoping

A single worker may process many concurrent jobs. If two jobs define a concept named `Result` with different fields, they must not share a `ClassRegistry` or `Library` — otherwise the second registration overwrites the first and silently corrupts data. Each job therefore creates its own scoped state:

1. **ClassRegistry** — a fresh `ClassRegistry` pre-seeded from the global registry (which holds the base classes from `PIPELEXPATH`). The crate's dynamic classes register here, never in the global registry.
2. **Library** — a fresh `Library` (`library_manager.open_fresh_library()`) with that `ClassRegistry` attached, set as current via a `ContextVar` keyed by the run id. Run-id keying (not job/workflow id) is deliberate: a reused workflow id across retries/resets could otherwise let a closed predecessor's cleanup tear down a live successor's library on the same worker.
3. **Lookup chain** — `hub.get_class_registry()` reads the `ContextVar`, returns the scoped library's registry, and falls back to the global registry when no library is set (e.g. a data converter running outside the job scope — which is exactly why output uses deferred hydration).
4. **Cleanup** — `library_manager.teardown(library_id)` drops the library and its registry; the `ContextVar` is reset. No manual GC needed.

The class-registry accessor and its library scoping (`pipelex/runtime_hub.py`, `pipelex/system/registries/class_registry_access.py`, `pipelex/interpreter_hub.py`, `pipelex/libraries/library.py`, `pipelex/libraries/library_manager.py`) is open core — the same machinery serves direct execution, the open runner, and any host-runtime plugin.

---

## Where open core ends

The boundary types and the host-agnostic helpers are **open** (MIT, in the `pipelex` distribution):

| Concern | Open core (`pipelex`) |
|---|---|
| Boundary DTOs | `PipeJob`, `PipeOutput`, `LibraryCrate`, `runtime_bridge/payloads.py` |
| Serialization | `runtime_bridge/serialization.py` |
| Mode / delivery model | `orchestration_mode`, `DeliveryMode`, the orchestrator registry / SPI |
| Working-memory raw + dump | `working_memory_raw`, `WorkingMemory.dump_for_transport` |
| Worker-side hydration | `runtime_bridge/primitives/hydration.py` |

The cross-process *plumbing* that drives a pipe run across the boundary — the host-side bridge entry point, the transport-prep step, and the dispatch primitives (submitter rehydration, delivery, trace flush, scoped-library, pipe classification, and the submit-arg envelope) — is **not open core**. Pipelex's own host-runtime embedding is a commercial capability.

A third-party orchestrator compiles against the **open** surface only (the [Orchestrator SPI](./orchestrator-plugins.md#the-orchestrator-spi)); Pipelex's own host-runtime plugins (Temporal, Mistral Workflows) additionally build on the closed side.

---

## Next steps

- [Orchestrator Plugins](./orchestrator-plugins.md) — the orchestration-mode seam, the Orchestrator SPI, and the open/closed boundary in contract terms.
- [Pipe Routing & Execution](./pipe-routing-and-execution.md) — how PipeJobs are created and routed in direct and distributed modes.
- [Content Generation Across Boundaries](./distributed-content-generation.md) — how dynamic Pydantic models and large binary payloads cross a worker boundary.
- [Distributed Execution](../distributed-execution/index.md) — the user-facing capability overview, and the [workflow-orchestration offer](https://pipelex.com/products#durable-execution) (Temporal-backed durable execution, the Mistral Workflows integration).


## Content Generation Across Boundaries

# Content Generation Across Boundaries

This page is for contributors working on Pipelex internals. For the capability overview, see the user-facing [Distributed Execution](../distributed-execution/index.md) page instead.

When content generation (LLM structured output, image generation, PDF extraction, web search) runs on a separate worker process, two serialization problems appear that single-process execution never faces. This page covers them backend-neutrally — the mechanisms live in open core and are the same regardless of which host runtime carries the payload. The concrete activity dispatch is a [commercial platform capability](https://pipelex.com/products#durable-execution); each backend (Temporal, Mistral Workflows) realizes it in its own plugin.

!!! note "Every inference leaf goes through the same seam"
    All inference operators dispatch their leaf call through the swappable `ContentGenerator` abstraction (`pipelex/cogt/content_generation/content_generator.py`): direct inline, or — on a host runtime — wrapped as an activity by the runtime's in-workflow content generator. The backend choice is independent of the run mode: under `run_mode=DRY` the chosen backend still dispatches and the leaf mocks inside it. Wrapping the leaf as a host-runtime activity is what makes it replay-safe (the result is recorded in the run's history) and lets a leaf failure cross the worker boundary as a classified error, instead of re-executing on every replay and hanging the submitter with an unclassified fault.

---

## The two serialization challenges

| Challenge | Trigger | Root cause | Solution |
|-----------|---------|------------|----------|
| **Dynamic class propagation** | LLM structured output (`PipeLLM` with object concept) | The submitter creates a Pydantic model at runtime from `.mthds` concept definitions; the worker process has no such class | Embed the JSON schema in the assignment, reconstruct the class on the worker, carry the source code through the transport's payload metadata |
| **Large payload management** | Image generation (`PipeImgGen`), PDF extraction (`PipeExtract`) | Binary data (base64 images, rendered pages) exceeds a host runtime's payload limit (Temporal's is ~2 MB) | Activities store binary data to external storage, return lightweight URI references |

---

## Dynamic class propagation

### Schema embedding

When a pipe produces structured output, the caller captures the Pydantic model's JSON schema into the assignment — a serializable argument that carries everything the worker needs:

```python
class ObjectAssignment(BaseModel):
    object_class_name: str
    object_class_schema: dict[str, Any]
    llm_assignment_for_object: LLMAssignment

    @staticmethod
    def make_for_class(
        object_class: type[BaseModel],
        llm_assignment: LLMAssignment,
    ) -> "ObjectAssignment":
        return ObjectAssignment(
            object_class_name=object_class.__name__,
            object_class_schema=object_class.model_json_schema(),
            llm_assignment_for_object=llm_assignment,
        )
```

The schema is a plain `dict` — fully serializable, inspectable on the wire, and self-contained. No class reference crosses the process boundary.

### Schema-to-model reconstruction

Reconstruction is the *boundary* answer, and only that. An in-process run never walks it: `ContentGenerator` passes the caller's live class down beside the assignment, and `resolve_object_class` (`pipelex/cogt/content_generation/object_class_resolution.py`) uses it as-is. That matters because the rebuild is lossy — custom validators, `json_schema_extra` hints, and the output structure's own description do not survive the round trip — so in-process the provider would otherwise be constrained by a weaker contract than the author wrote, for no reason. On a worker there is no choice: the class genuinely cannot travel, so the rebuild below is the right answer there.

On the worker, `SchemaToModelFactory.make_from_json_schema(schema, *, class_name)` (`pipelex/cogt/content_generation/schema_to_model_factory.py`) rebuilds a live Pydantic class from the embedded schema:

```python
source_code = cls._generate_source_from_schema(schema)        # datamodel-code-generator
reconstructed_class = cls._exec_and_extract_class(source_code, class_name=class_name)  # exec()
reconstructed_class.__kajson_class_source__ = source_code  # attached for downstream use
```

The reconstruction pipeline:

1. **Generate** — `datamodel-code-generator` converts the JSON schema into Python source code defining a `BaseModel` subclass.
2. **Execute** — `exec()` compiles the source in an isolated namespace and extracts the named class.
3. **Cache** — A thread-safe SHA-256 hash cache is guarded by a single lock held across the entire check-then-generate sequence, not double-check locking: the lock is acquired once, the cache is checked, and — on a miss — generation and exec happen before the lock is released. This kills same-schema thundering herds (concurrent first-misses each paying a full codegen+exec round) at the cost of serializing cache hits on other schemas behind any in-flight miss, an acceptable trade-off given real workloads use a small bounded set of distinct schemas cached for the process lifetime.
4. **Tag** — The generated source code is attached as `__kajson_class_source__` on the class, enabling Kajson to carry it through the transport.

!!! warning "Class name normalization"
    `datamodel-code-generator` normalizes schema titles to PascalCase (e.g., `dynamic_concept_test__Greeting` becomes `DynamicConceptTestGreeting`). The code tries both the original and normalized names. If neither matches, it raises `ValueError` listing the available classes.

### Carrying the class source across the boundary

The reconstructed class's source rides in the transport's **payload metadata**, not in the payload data itself — keeping the JSON payload clean and the reconstruction transparent. A host runtime's data converter injects it on serialize and re-attaches it on deserialize:

**Serialize:**

```python
class_source = getattr(type(value), "__kajson_class_source__", None)
if class_source is not None:
    metadata["kajson_class_source"] = class_source.encode()
```

**Deserialize:**

```python
source_bytes = payload.metadata.get("kajson_class_source")
class_source_code = source_bytes.decode() if source_bytes else None
pydantic_gizmo = kajson.loads(
    data,
    class_registry=get_class_registry(),
    class_source_code=class_source_code,
)
# Re-attach source so it survives further hops
if class_source_code is not None and isinstance(pydantic_gizmo, BaseModel):
    type(pydantic_gizmo).__kajson_class_source__ = class_source_code
```

Re-attaching after `kajson.loads()` ensures the source survives if the object crosses another boundary (e.g. a worker returning a result to a parent that forwards it). The `__kajson_class_source__` attachment is open core; the converter that injects it into payload metadata is the host-runtime plugin's (Pipelex's Temporal backend realizes it as `BaseModelPayloadConverter`).

### The type bridge

The class reconstructed on the worker is a structural match to the original, but a different Python class in memory — so `isinstance()` checks would fail. The caller bridges the gap with a `model_validate` round-trip, in `revalidate_leaf_object` (`pipelex/cogt/content_generation/object_revalidation.py`):

```python
if isinstance(raw_obj, object_class):
    return raw_obj
return object_class.model_validate(raw_obj.model_dump(mode=dump_mode, serialize_as_any=True, by_alias=True))
```

`model_dump(serialize_as_any=True)` produces a plain dict from the reconstructed object, and `model_validate()` rebuilds it as a proper instance of the original class, restoring type safety for downstream code.

`by_alias=True` is what keys that dict correctly. `model_json_schema()` emits **by alias**, so the schema's property names are what the original class accepts on the way back in — but the reconstructed class does not always name its fields that way. `datamodel-code-generator` renames any property that cannot be a field name (a python keyword, or a name shadowing a `BaseModel` attribute such as `json`, `copy`, `schema`, `construct`) and records the original as an alias. Dumping by field name would emit `construct_` for a field the original calls `construct`. Where nothing was renamed, the alias is the field name, so this is a no-op.

The short-circuit is what makes this a *boundary* conversion rather than a step on every path. In-process the leaf already worked from the caller's class, so the object is already one — note `isinstance`, not `type(...) is`: a provider library typically returns a *subclass* of the response model, not the class itself. Converting it anyway would run the caller's validators a second time on data they already normalized, which corrupts the output of any validator that transforms rather than merely rejects. A reconstructed class is never a subclass of the original, so the boundary path falls through to the conversion as before.

The helper is public so that both implementations of `ContentGeneratorProtocol` can share it — the in-process generator, which uses it today, and a host runtime's workflow arm, which still carries its own copy and adopts the shared one on its next release. The point of the single home is that the conversion, the short-circuit and the typed dry-run fidelity error stop being two copies that only agree until someone edits one. `dump_mode` is the one thing the two boundaries differ on: an object that crossed a payload boundary needs `mode="json"`, because some field types only round-trip cleanly through their json form.

### Structured web search reuses the same mechanism

Structured web search (`PipeSearch` with a non-text output concept) faces the identical dynamic-class problem and splits the same way, with one twist: the boundary arm must keep the *result* serializable too.

- **At the boundary**, `SearchObjectAssignment` mirrors `ObjectAssignment` — it ships `output_class_name` + `output_class_schema` alongside the `SearchAssignment`, the activity reconstructs a throwaway class to drive the provider call, and `search_gen_structured` returns the **raw result dict**. The submitter re-validates that dict against the original output class — a pure, deterministic step that keeps the dynamic class on the submitter side and never ships it across the boundary.
- **In-process**, `search_gen_structured_object` takes the caller's class, hands it to the provider unchanged, and returns an instance of it. The validation happens once, at the leaf, next to the provider. Returning the instance rather than a dump of it is what keeps it once: the search leaf has no object to short-circuit on if its result has already become a dict.

The sourced-answer path (`make_search_sourced_answer`) has no dynamic class at all: it returns a `SearchResultContent`, a native serializable model.

---

## Large payload management

### The payload size constraint

A host runtime's payload has a practical size limit (Temporal's is around 2 MB). A single generated image can exceed it in base64; PDF extraction produces multiple page images. Sending raw binary through the runtime would fail conversion or degrade the run history.

### The store-then-reference pattern

Activities handle the full binary lifecycle internally — generate, store, return a lightweight reference. The storing is open-core: `GeneratedContentFactory` (`pipelex/cogt/content_generation/generated_content_factory.py`) uploads binary data via the configured `StorageProviderAbstract` (S3, local filesystem, …) and returns `ImageContent` or `PageContent` — Pydantic models carrying URIs, MIME types, and metadata, but never raw bytes. A host runtime's activity simply wraps that helper:

```python
async def generate_and_store_images(img_gen_assignment):
    """Large binary (base64/bytes) is stored inside the activity and never
    crosses the runtime boundary — only URLs come back."""
    factory = GeneratedContentFactory(storage_provider=get_storage_provider())
    return await img_gen_image_list_and_store(
        img_gen_assignment=img_gen_assignment,
        generated_content_factory=factory,
    )
```

!!! info "What crosses the boundary"
    `ImageContent` carries `url` (storage URI), `public_url`, `mime_type`, paired `width`/`height`, and `caption` — but never raw bytes. The `url` can be an S3 URI, HTTP URL, or local file path depending on storage configuration.

What gets stored vs. what crosses the boundary, by content type:

| Content type | What gets stored | What crosses the boundary |
|-------------|-----------------|----------------------------|
| Generated image | Base64/bytes → storage via `GeneratedContentFactory` | `list[ImageContent]` with URIs |
| Extracted pages | Extracted page images → storage | `list[PageContent]` with URI references |
| Rendered page views | PDF page renders → storage | `list[ImageContent]` with URIs |
| LLM text | Nothing (text is small) | Plain `str` |
| LLM object | Nothing (JSON is small) | `BaseModel` + `__kajson_class_source__` in metadata |
| Search sourced answer | Nothing (answer + source refs are small) | `SearchResultContent` (answer + `DocumentContent` sources) |
| Search structured | Nothing (JSON is small) | Raw `dict`, re-validated against the output class on the submitter |

Each host-runtime plugin dispatches these through its own activities and queue routing; for the Temporal realization (the `act_*` activity set, per-activity task queues), see our Temporal plugin's own docs.

---

## Kajson class resolution

When `kajson.loads()` receives both a `class_registry` and `class_source_code`, the resolution follows a priority chain:

```python
if class_source_code is not None:
    source_registry = _build_registry_from_source(class_source_code)
    if class_registry is not None:
        # Source-derived classes fill gaps; explicit registry takes priority
        for name, cls in source_registry.root.items():
            if not class_registry.has_class(name):
                class_registry.register_class(cls, name=name, ...)
    else:
        class_registry = source_registry
```

Resolution order:

1. **Explicit `class_registry`** — the per-call scoped registry (see [Runtime Bridge & Transport](./runtime-bridge-and-transport.md#per-call-scoping))
2. **Source-derived classes** — from `class_source_code`, filling gaps in the explicit registry
3. **`sys.modules`** — already-imported classes
4. **Dynamic import** — last resort, importing the module path from `__module__` metadata

!!! note "exec() safety"
    `_build_registry_from_source()` uses `exec()` internally. The source code originates from `datamodel-code-generator` running against the same JSON schema the submitter used — it is never user-supplied arbitrary code.

---

## File Reference

| Component | File |
|-----------|------|
| Assignment models (schema embedding) | `pipelex/cogt/content_generation/assignment_models.py` |
| Schema-to-model reconstruction | `pipelex/cogt/content_generation/schema_to_model_factory.py` |
| Content generator (type bridge) | `pipelex/cogt/content_generation/content_generator.py` |
| LLM generation functions | `pipelex/cogt/content_generation/llm_generate.py` |
| Search generation functions | `pipelex/cogt/content_generation/search_generate.py` |
| Generated content factory (storage) | `pipelex/cogt/content_generation/generated_content_factory.py` |
| Kajson serialization | `kajson` (external PyPI package) |
| ImageContent model | `pipelex/core/stuffs/image_content.py` |
| PageContent model | `pipelex/core/stuffs/page_content.py` |

The host-runtime data converter and the `act_*` activities that carry these payloads are backend-specific and live in their plugin repos, not in open core.

---

## Next Steps

- [Runtime Bridge & Transport](./runtime-bridge-and-transport.md) — boundary DTOs, LibraryCrate propagation, deferred hydration, per-call scoping
- [Pipe Routing & Execution](./pipe-routing-and-execution.md) — how PipeJobs travel through the system
- [Architecture Overview](./architecture-overview.md) — the two-layer design and how components fit together


# Project

## Contributing

--8<-- "CONTRIBUTING.md"


## Configuration Internals

# Configuration Internals (defaults and overrides)

This page is for **contributors**: it explains where Pipelex defaults live, how configuration is merged at runtime, and where to change the behavior in code.

For the user-facing configuration guide, see [Configuration Overview](../configuration/index.md).

## Mental model (for contributors)

Pipelex configuration is merged in layers:

- **Shipped defaults**: maintained by the Pipelex project and used as the baseline by the installed package.
- **Global config** (`~/.pipelex/`): machine-wide settings for a developer, applied to every project on the machine.
- **Project config** (`{project_root}/.pipelex/`): per-project settings, edited by teams using Pipelex.
- **Override files** at each level (`pipelex_local.toml`, `pipelex_{environment}.toml`, `pipelex_{run_mode}.toml`, `pipelex_override.toml`, `pipelex_temporary_override.toml`): optional, typically gitignored, used for personal or ephemeral tweaks.

The key idea: global personal preferences layer **under** project-specific settings — both are loaded, and project values win on collisions. Nothing in `~/.pipelex/` is shadowed by the mere existence of a project `.pipelex/`.

## Where defaults live (Pipelex repository)

- **Default values** (baseline): `pipelex/pipelex.toml`, shipped inside the package
- **User-facing templates** copied into `.pipelex/` by the init flow: `pipelex/kit/configs/…`

## Merge order (runtime)

The configuration loading/merging behavior is implemented in `pipelex/system/configuration/config_loader.py`. Files are deep-merged via `load_toml_from_path_and_merge_with_overrides`; the later a file appears in the load list, the higher its precedence per leaf key.

Load order:

1. **Package defaults** — `pipelex/pipelex.toml` shipped with the installed package.
2. **Global base** — `~/.pipelex/pipelex.toml`.
3. **Global override sequence** — from `~/.pipelex/`, in this order:
    1. `pipelex_local.toml`
    2. `pipelex_{environment}.toml`
    3. `pipelex_{run_mode}.toml` *(see unit-test special case below)*
    4. `pipelex_override.toml`
    5. `pipelex_temporary_override.toml`
4. **Project base** — `{project_root}/.pipelex/pipelex.toml`, if a project `.pipelex/` exists and is distinct from `~/.pipelex/`.
5. **Project override sequence** — same five files as step 3, read from `{project_root}/.pipelex/`, if the project dir is distinct from the global dir.
6. **Programmatic `extra_overrides`** passed into `ConfigLoader.load_config()` (if any).

!!! note "Unit-test special case"
    When `runtime_manager.is_unit_testing` is true, the `pipelex_{run_mode}.toml` entry is sourced exclusively from `./tests/pipelex_{run_mode}.toml` (e.g. `tests/pipelex_unit_test.toml`). Global and project run_mode files are not loaded, keeping test runs hermetic from machine-wide overrides.

## Where to change things in code

- **Config merge logic**: `pipelex/system/configuration/config_loader.py`
- **Override file sequences**: `ConfigLoader._override_files_for_dir` and `ConfigLoader._plugin_override_files_for_dir` in that same file

## Contributor guidelines for config changes

- If you change **defaults**, update `pipelex/pipelex.toml` (and consider whether templates in `pipelex/kit/configs/…` should also be updated).
- If you change **templates**, keep them user-focused and stable; avoid adding internal-only details.
- If you change **merge order or override semantics**, treat it as a potentially breaking change and document it (changelog + migration notes if needed).

---

## Test Profile Configuration

The test profile system controls which AI models are included in parametrized tests. This is separate from the main Pipelex configuration.

### Files

| File | Purpose | Tracked |
|------|---------|---------|
| `.pipelex-dev/test_profiles.toml` | Base profiles and model collections | Yes |
| `.pipelex-dev/test_profiles_override.toml` | Local customizations | No (gitignored) |

### How it works

1. Collections define reusable lists of models organized by provider
2. Profiles reference collections or specify models directly
3. The preprocessing command resolves references and generates fixture files

### Key code locations

- **Profile loading & merging**: `pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py`
- **Generated fixtures**: `tests/integration/pipelex/fixtures/_generated_model_sets.py`

### Common tasks

```bash
# Regenerate fixtures with default (dev) profile
make regenerate-test-models

# Use a specific profile
make regenerate-test-models TEST_PROFILE=full
```

For full documentation, see [Test Profile Configuration](../under-the-hood/test-profile-configuration.md).



## Keyword-Only Arguments

# Keyword-only arguments convention

This is the canonical convention for keyword-only function parameters across the `pipelex/` source tree (tests are out of scope). The goal is self-documenting call sites: when a function takes more than one meaningful argument, the caller must name them, so `do_thing(retries=3, timeout=30)` is forced over the unreadable `do_thing(3, 30)`. The convention is mechanically enforced by an AST guard (the `check-keyword-only` dev command, wired into both `make agent-check` and the `make check` family, and gated in CI). The `pipelex/` source tree is fully compliant, so the guard hard-blocks on **any** violation: new code must follow this convention or it fails the build. This document is the human-readable specification that guard implements.

## The rule

A function or method must place a bare `*` separator in its signature so that every parameter after the subject is keyword-only. The compliant shapes are:

```python
def f(*, opt1, opt2): ...            # fully keyword-only — always compliant, no registry entry needed
def f(subject, *, opt1, opt2): ...   # positional subject — compliant ONLY under a recorded subject grant
```

The first non-`self`/`cls` parameter — the subject — may stay positional-or-keyword **only when a subject grant for that def is recorded in `subject_grants.toml`** (see the next section). We deliberately do NOT use the positional-only `/` separator anywhere; a granted subject may be passed positionally or by keyword, at the caller's discretion.

A signature is a violation when any of these holds:

- **Missing bare `*`** — after dropping a leading `self`/`cls` and the subject, a positional-or-keyword parameter remains. Only the subject may ever be positional; everything after it must be named. So `def f(a, b)` and `def truncate(text, max_length=80)` are violations regardless of any grant.
- **Ungranted subject** — the def keeps a positional subject (`def render(node)`, `def f(spec, *, opt)`) but no grant for it exists in the registry.
- **Banned literal-typed subject** — the subject is annotated `bool`, `int`, or `float`, including their `Optional[X]` / `X | None` / `Union[X, None]` forms. This is a violation no matter what: a grant can never cover it, because call sites read as bare literals (`f(True)`, `retry(3)`) — exactly the opacity this convention exists to kill. `str` subjects remain grantable; they are the house style (`pipe_code`, `name`, `uri`, …). An unannotated subject is not provably literal and remains grantable.
- **Stale grant** — a registry entry whose def no longer exists, whose recorded `param` no longer matches the def's subject, or whose def no longer has a positional subject at all (demoted, renamed, moved, or newly carved out). Staleness is symmetric and hard-fails: a rename or file move forces a deliberate re-grant, not silent rot.

The registry carries one more rule that is not about any signature: its entries must stay in sorted key order (see [Staleness is symmetric](#staleness-is-symmetric)).

## Exception 1 — granted subject parameters

> **History note — this exception was narrowed, deliberately.** Exception 1 used to be a *generic* permission: any first non-`self`/`cls` parameter could stay positional, guard-approved, no questions asked. That judgment-shaped permission was systematically abused by coding agents, which left first arguments positional when they were not the subject or not obviously so — the check passed, so the prose was ignored. The permission is now an **explicitly granted, recorded exception**: judgment still exists, but it happens at grant time and in review of the registry diff, while CI stays 100% deterministic. Agents follow checks, not prose — so the check now carries the whole rule.

A def may keep its subject positional only under a **subject grant**: an entry in `subject_grants.toml` at the repo root, keyed by the def's stable identity (`<relative_path>::<qualified_name>`), recording the subject's parameter name and a one-sentence rationale:

```toml
version = 1

["pipelex/graph/render.py::render_node"]
param = "node"
rationale = "Verb–object: renders the node; single obvious operand."
```

The registry is machine-written and sorted by key, so diffs stay stable and merge conflicts resolve trivially — and that order is **checked**, not merely claimed here (`unsorted-grant`). An unsorted registry parses fine and grants exactly the same subjects, so a bulk rewrite that scrambles it — a package move, a sed over paths — would otherwise break the invariant silently. It is written **only** by the grant command; never edit it by hand except to delete a stale entry:

```bash
make subject-grant FUNC="pipelex/graph/render.py::render_node" RATIONALE="Verb–object: renders the node; single obvious operand."
# alias: make sgr — wraps: pipelex-dev subject-grant "<path>::<qualname>" --rationale "…"
```

The command validates the def (it must exist, be inspected by the guard, and have a positional non-literal-typed subject), records the `param` automatically, and rewrites the file sorted. It refuses literal-typed subjects, fully-keyword-only defs, carved-out defs, and empty rationales. Same-qualname defs (`@overload` stubs, conditional redefinitions) share one entry — the recorded `param` must match each of them, which forces overload stubs to align their subject name.

### What earns a grant — the rubric

A subject grant is warranted when EITHER:

- **Verb–object test:** the function name is a verb (phrase) and the param is its direct object — the call reads as a sentence (`render(node)`, `validate_bundle(bundle)`, `parse_concept_spec(spec_data)`) — AND it is the **single candidate** (if you hesitate between two params, neither is the subject) — AND typical call sites pass a **self-labelling expression**, never a bare literal (literal-typed subjects are banned outright anyway);
- OR the def must satisfy a **positional `Callable` protocol** (it is passed as a value to something that calls it positionally) and a grant keeps it compliant without reaching for the heavier `# kw-only: ignore`.

When in doubt → keyword-only. The grant is the exception tier; all-keyword is always compliant and often more readable. Rationales must be def-specific but may be terse for obvious keeps ("verb–object; single operand") — the value is that someone actually looked; copy-paste boilerplate across dozens of entries defeats the point.

The grant is a permission, not a requirement — making the subject keyword-only too is always allowed, needs no registry entry, and is frequently the better choice. The economics are intentional: `make fko` mechanically produces the all-keyword form, so the lazy path is the strict path; keeping a positional subject costs a deliberate command with an honest rationale, visible as a registry diff in every PR.

### Staleness is symmetric

The guard verifies the registry in both directions on every run: a positional subject without a grant fails, and a grant without a matching def fails (`dead-grant`), as does a grant whose recorded `param` no longer matches the def's subject (`grant-param-mismatch`). Renames, file moves, demotions to all-keyword, and newly added carve-outs (`@override`, framework decorators) all kill the grant — delete the entry or re-grant deliberately. A missing or malformed `subject_grants.toml` is an explicit check error, never treated as an empty registry (that would mass-flag every granted subject in the tree).

File order is checked alongside them (`unsorted-grant`). Order is read from the raw text, not from the parsed registry: TOML declares table order insignificant, so a `dict` is no evidence of what the file looks like. The text scan is cross-checked against the parsed keys, and a disagreement is an explicit check error rather than a quietly narrowed scan — an entry the scan cannot see would otherwise be exempt from the order rule without anyone knowing.

### Registry schema

A registry entry carries exactly two keys — `param` and `rationale`; anything else fails the check. When the registry was introduced, every pre-existing positional subject entered a transitional review grind (treated as new, not grandfathered); that transition is over — every grant now carries a genuine, def-specific review rationale. `pipelex-dev check-keyword-only --report` shows the full inventory and the per-package grant counts.

## Exception 2 — symmetric tuples (explicit allowlist only)

A small set of functions take a short, conventionally-ordered tuple of arguments where positional calling is the universally-understood form and naming would be pure noise: `clamp(value, low, high)`, `Point(x, y)`, `replace(text, old, new)`, `lerp(a, b, t)`, `set_env(key, value)`. These are exempt ENTIRELY — the whole function, including any trailing options — but ONLY via an explicit allowlist keyed by qualified name and file, curated in the guard's source (NOT in `subject_grants.toml` — the registry covers single subjects; the allowlist exempts whole functions, a different semantic). There is no pattern-based guessing: a function is symmetric-tuple-exempt if and only if it is listed in the allowlist. This keeps the carve-out auditable and prevents the exemption from quietly swallowing genuine subject-plus-options signatures, which are the main thing this convention wants to fix.

The curated allowlist is intentionally short. Each entry is a genuine ordered tuple under a well-known convention, not a subject followed by descriptive options.

- `pipelex.system.environment.set_env` — `set_env(key: str, value: str)`. Canonical key/value pair mirroring dict assignment; naming them would be pure noise.
- `pipelex.kit.single_file_agent_rules.unified_diff` — `unified_diff(before: str, after: str, path: str)`. `before`/`after` is the canonical ordered diff pair; `path` is the diff header. Reads as the universally-known diff convention.
- `pipelex.tools.misc.diff.diff_files` — `diff_files(path1, path2)`. Symmetric two-operand comparison; the `1`/`2` suffixes self-label ordering and there is no old/new semantics that keywords could clarify.
- `pipelex.tools.misc.diff.diff_dirs` — `diff_dirs(dir1, dir2)`. Same symmetric two-directory comparison; `dir1`/`dir2` is a self-labelling ordered pair.
- `pipelex.tools.typing.class_utils.are_classes_equivalent` — `are_classes_equivalent(class_1, class_2)`. Symmetric structural-equivalence check; order is irrelevant (equivalence is commutative) and the `1`/`2` suffixes self-label the operands. No trailing options.

The operating rule for two-operand functions — when only one of the operand-positional and the all-keyword form should win:

- **A clean symmetric/ordered pair with NO trailing options** may go on the allowlist and stay fully positional (the entries above). Order is either irrelevant (`are_classes_equivalent`) or universally understood (`diff_files`'s `1`/`2`).
- **A pair that also carries trailing options** is reshaped to fully keyword-only instead — the whole-function allowlist is all-or-nothing, so it cannot keep the pair positional while forcing the options keyword, and letting the options go positional is exactly the unreadable case this convention targets. Examples now keyword-only: `copy_file(*, source_path, target_path, overwrite=True)`, `mirror_dir(*, source_dir, target_dir, …)`, `sync_toml_values(*, source_path, target_path, dry_run=False)`, `has_diff_dirs(*, dir1, dir2, exclude_files=None, exclude_dirs=None)`, `is_multiplicity_compatible(*, source_multiplicity, target_multiplicity)`, the graph tracer's `add_edge(*, source_node_id, target_node_id, edge_kind, …)`.

This deliberately keeps the allowlist whole-function (no per-entry "leading positional count" machinery): the handful of pairs that would benefit from staying positional-with-keyword-options are rare enough that reshaping to all-keyword is the simpler, uniform answer.

Adding to the allowlist is a deliberate act: the entry must name a genuine ordered tuple under a recognized convention (key/value, x/y, min/max/clamp, before/after, src/dst, geometric coordinates, interpolation operands) with no trailing options, justified in the same conservative spirit as the entries above. When in doubt, leave it off and let the function be keyword-only — that is the safe default.

## Carve-outs (skipped entirely)

Carve-outs are different from exceptions: a carved-out def is never inspected at all, regardless of its parameter shape — it neither needs a subject grant nor can hold one alive. These are signatures the guard cannot or must not touch because some framework or the interpreter itself owns the calling convention.

### Dunder and operator methods

Any method whose name matches `^__[A-Za-z0-9_]+__$` (a leading and trailing double underscore around a real identifier body) is skipped. The interpreter and the data-model protocols invoke these positionally — `__init__`, `__eq__`, `__call__`, `__getitem__`, `__format__`, `__contains__`, `__enter__`, `__exit__`, and so on — and never by keyword, so forcing keyword-only would be both wrong and impossible to satisfy. The match is on the method NAME node only (not the source line), is a full-match (anchored), and requires a non-empty body between the underscore pairs so the degenerate `____` cannot match. This deliberately does NOT carve out name-mangled half-dunders (`__private` with no trailing `__`): those are ordinary methods we call ourselves, and they remain subject to the rule.

### Pydantic validators and serializers

A def decorated with any of these pydantic decorator names is skipped, because pydantic invokes the wrapped callable with a fixed positional protocol (`value`, `info`, `handler`, ...) that we do not control:

- `field_validator`
- `model_validator`
- `field_serializer`
- `model_serializer`
- `validator` (pydantic v1 legacy)
- `root_validator` (pydantic v1 legacy)

Only `field_validator`, `model_validator`, and `model_serializer` are actually used in the source today; the v1 legacy names and `field_serializer` are included defensively so the guard stays correct if they are introduced later. Decorators may be call-form (`@field_validator("name", mode="before")`) or bare (`@model_serializer`); the guard handles both by unwrapping `ast.Call.func` when present and matching on the unqualified decorator name. The `parse_model_reference` helper referenced via `Annotated[..., BeforeValidator(parse_model_reference)]` is NOT a decorated def — it is a plain single-parameter module function (whose lone subject needs a grant like any other), and needs no carve-out.

### Framework entrypoints

A def is skipped when it is registered with a framework whose calling convention is not ours to dictate. Matching is on the decorator's attribute suffix (the receiver name varies — `app`, `graph_app`, `show_app`, `kit_app`, `build_app`, ...), and on the pytest decorators by their attribute/name:

- Typer/click commands and callbacks — any decorator that is an attribute access ending in `.command` or `.callback`. The guard scans the WHOLE decorator stack for a framework match (a framework decorator may sit above a custom one such as `@convert_pipelex_errors`), not just the innermost or outermost decorator.
- pytest fixtures — `fixture`, matched on the bare decorator name so both `@pytest.fixture` and bare `@fixture` (`from pytest import fixture`) are covered (the only src-side fixtures live in the shipped `pipelex/test_extras/shared_pytest_plugins.py` plugin module).
- Jinja2 filters/tests/globals — `pass_context`, `pass_environment`, `pass_eval_context`, matched on the bare decorator name (covers both `@pass_context` from `from jinja2 import pass_context` and the attributed `@jinja2.pass_context`). The Jinja2 engine invokes a filter POSITIONALLY from template syntax — `{{ value | tag("name") }}` calls `tag(context, value, "name")` — so the wrapped callable's arguments cannot be made keyword-only. This is the same framework-entrypoint category as Typer/pytest. A multi-argument filter without one of these decorators (rare) is covered by the `# kw-only: ignore` escape hatch; single-argument filters (`escape_script_tag(value)`) have an ordinary lone subject and take a grant like any other def.

FastAPI is intentionally absent: there are no route handlers in `pipelex/` source — the FastAPI server lives in the separate `pipelex-api/` repo.

Known coverage gap (documented, not silently ignored): some Typer commands are registered by call-style `app.command(name=...)(some_fn)` against functions defined in separate modules that carry NO decorator, so a decorator-keyed guard cannot see them via the framework carve-out. These target functions consistently type their parameters as `Annotated[T, typer.Argument(...)]` / `Annotated[T, typer.Option(...)]`. The guard treats a def as a framework entrypoint when any of its parameters carry `Argument`/`Option` metadata in their `Annotated[...]` annotation — matched on the trailing callee name, so both the qualified `typer.Option(...)` form and the bare `Option(...)` form (`from typer import Option`) are recognized — which closes the gap without resorting to a broad path-based exclusion of CLI modules.

### Base / Protocol / ABC overrides

A def decorated with `@override` (from `typing_extensions` / `typing`) is skipped. See the next section for why this single decorator is a complete and reliable signal in this codebase.

## Override handling — decision

DECISION: skip any def carrying an `@override` decorator. We do NOT introduce a dedicated `@kw_exempt` decorator for overrides, and we do NOT attempt base-aware (import-resolving / type-inferring) detection.

Rationale. An overriding method cannot freely re-shape its signature — it must stay compatible with the base/Protocol it implements — so the correct place to apply the keyword-only convention (and to record the subject grant, when the base keeps a positional subject) is the base definition, after which implementations inherit the contract and pyright's `reportIncompatibleMethodOverride` keeps them in line. Skipping `@override` defs is consistent with fixing the base. Crucially, `@override` is a reliable AND complete signal here: `pyproject.toml` sets `reportImplicitOverride = true`, so any method overriding a nominal base MUST carry `@override` or `make agent-check` already fails before the keyword-only guard runs — the marker cannot silently rot. Protocol implementations are structural and not forced by that setting, but house style applies `@override` to them consistently too; the residual risk is a hypothetical future Protocol implementor that forgets `@override`, which is already a style deviation the team avoids and is covered by the inline escape hatch below. A `@kw_exempt` decorator would duplicate semantics `@override` already carries and add a second redundant annotation to dozens of override sites for no gain. Base-aware detection would require import resolution a pure-AST guard cannot do reliably, and the codebase has already solved the problem for us via `reportImplicitOverride`. The guard matches `@override` structurally: a decorator that is an `ast.Name` with `id == "override"` or an `ast.Attribute` with `attr == "override"`, covering both `override` and `typing_extensions.override` without resolving imports.

## Inline escape hatch

A `# kw-only: ignore` comment on the `def` line (or the `async def` line) suppresses exactly one violation for that function — it short-circuits everything, including the subject-grant rules and the literal-subject ban. This is the documented fallback for the rare legitimate case the carve-outs and allowlist do not cover: chiefly a def some framework invokes positionally without a recognizable decorator (a `typer.Option(callback=...)` target like `version_callback(value: bool)`, an `__import__` hook, an SDK error callback), or a Protocol implementor that genuinely lacks `@override`. The escape hatch is the exception, not the mechanism: a reviewer seeing `# kw-only: ignore` should expect a short justification nearby. The marker is matched against the comment text on the def line; it suppresses the single def it sits on, not a whole class or module.

## Worked examples

A subject plus options — the canonical shape, legal only with a grant:

```python
# subject_grants.toml carries: ["pipelex/builder/build.py::build_pipe"] param = "spec" rationale = "…"
def build_pipe(spec, *, dry_run, retries, validate): ...   # build_pipe(my_spec, dry_run=True, retries=3, validate=False)
```

A single subject and nothing else — needs a grant too (strict-all scope; lone-subject defs are not implicitly exempt):

```python
def render(node): ...   # render(my_node) — granted: verb–object, single operand
```

All parameters named, including the first — always compliant, no registry entry, and often the most readable choice:

```python
def copy_payload(*, source, target): ...   # copy_payload(source=a, target=b)
```

Even a single defaulted option must be named — this is the common case the convention fixes, not an exception to it:

```python
# before: truncate(text, 120) — what does 120 mean?
def truncate(text, max_length=80): ...
# after: truncate(text, max_length=120) — with a grant for `text`, or fully keyword-only with none
def truncate(text, *, max_length=80): ...
```

A literal-typed subject — banned outright, no grant possible; make it keyword-only:

```python
# before: doctor_cmd(True) — reads as a bare literal
def doctor_cmd(fix: bool): ...
# after: doctor_cmd(fix=True)
def doctor_cmd(*, fix: bool): ...
```

A symmetric tuple on the allowlist — exempt entirely, stays positional:

```python
def set_env(key: str, value: str) -> None: ...   # set_env("PIPELEX_ENV", "prod")
```

A directional pair with a trailing option — neither operand is "the object", and a whole-function allowlist exemption would let the option go positional too, so the whole signature becomes keyword-only:

```python
# before — copy_file(src, dst, True): what does True mean?
def copy_file(source_path, target_path, overwrite=True): ...
# after — copy_file(source_path=src, target_path=dst, overwrite=False); every argument is named
def copy_file(*, source_path, target_path, overwrite=True): ...
```

A clean symmetric pair with no trailing options is the opposite case — it goes on the Exception 2 allowlist and stays fully positional (`are_classes_equivalent(class_1, class_2)`).

An override — skipped because of `@override`; fix (and grant) the base signature instead:

```python
@override
def _store(self, data, *, key, content_type): ...   # base StorageProviderAbstract._store defines the contract and holds any grant
```

A dunder — skipped by name; the interpreter owns the calling convention:

```python
def __format__(self, format_spec: str) -> str: ...   # called positionally by format()/f-strings
```

A pydantic validator — skipped by decorator; pydantic owns the positional protocol:

```python
@field_validator("some_field", mode="before")
def normalize(cls, value, info): ...
```

## Enforcement

The entire `pipelex/` source tree is compliant, so the guard hard-blocks on **any** violation — there is no baseline and no tolerated debt. Each violation kind names its own remedy in the check output:

| Kind | Meaning | Remedy |
|---|---|---|
| `missing-star` | a non-subject positional-or-keyword param remains | place the bare `*` (or `make fko`) |
| `ungranted-subject` | positional subject with no registry entry | `make fko` to go all-keyword, or `make subject-grant` |
| `literal-subject` | `bool`/`int`/`float` subject (incl. Optional/union-with-None) | make it keyword-only — grants are impossible |
| `grant-param-mismatch` | the grant's recorded `param` no longer matches the def | re-grant, or fix the registry |
| `dead-grant` | the grant matches no def with a positional subject | delete the entry (or re-grant after a rename/move) |
| `unsorted-grant` | the entry sits out of key order in the registry file | re-record any grant (it rewrites the file sorted), or move the entry back by hand |

The guard runs at several gates:

- A Claude Code `PostToolUse` hook (`.claude/hooks/check-keyword-only.sh`, wired in `.claude/settings.json`) — the tightest loop. After every `Edit`/`Write`/`MultiEdit` of a `pipelex/**/*.py` file it checks just that file and blocks with the offending signatures if it regressed, so an agent learns at edit time rather than at the end of the session. It runs the stdlib-only core by file path (no Typer/hub import), so it costs a few tens of milliseconds — including one `tomllib` read of the registry. The hook is check-only — it never rewrites a file mid-edit. It cannot see registry-level violations (dead grants and file order are whole-file concerns, invisible from one edited source file); the full check owns those.
- `make agent-check` — the fast everyday gate. It runs the **auto-fixing** variant (`fix-keyword-only`) early, right after `fix-unused-imports` and before `ruff format`, so a mechanically-fixable violation is corrected in place instead of failing the build. The fixer is **non-gating** — it reports the violations it can't mechanically fix but does not abort — so `format`, `lint`, `pyright`, and `mypy` all still run (you get keyword-only *and* type feedback in a single pass, and the tree is never left half-mutated and unformatted). The **read-only** `check-keyword-only` then runs **last** and is what actually blocks on any remaining violation. ⚠ Note the fixer's reach: an **ungranted subject def is a violation**, so `make agent-check` will silently keyword-only it — record the grant BEFORE running checks if you want the subject to stay positional.
- `make check` — the heavy aggregate gate. It runs the **check-only** `check-keyword-only` (no rewriting), so CI-equivalent runs stay read-only.
- CI — a dedicated lint job (`lint-keyword-only` in `.github/workflows/lint-check.yml`) gated by the required `Lint (all)` status check, so no non-compliant signature can merge. This is the hard guarantee; the hook and `make` gates are local conveniences layered on top. The registry diff in the PR is where grant judgment gets reviewed.

When you add or change a signature, place the bare `*` after the subject and grant the subject (or make everything keyword-only — no grant needed). If the type checker is blind to how a function is called (a framework or the interpreter invokes it positionally — a Jinja2 filter, an `__import__` hook, an aiohttp route handler, a PostHog `on_error` callback), the guard cannot detect that statically; the carve-outs above cover the known cases, and a genuinely justified one-off uses the `# kw-only: ignore` escape hatch. `make agent-test` is the safety net for these dynamic call surfaces — pyright/mypy will pass a wrongly-keywordized callback that the suite then catches at runtime.

## Auto-fix

`pipelex-dev check-keyword-only --fix` (exposed as `make fix-keyword-only` / `fko`, and run automatically inside `make agent-check`) rewrites every *mechanically-fixable* violation by inserting a bare `*` as far left as possible — immediately after `self`/`cls` (and after any `/`) — so **every** non-`self`/`cls` parameter becomes keyword-only, not just the ones after the subject: `def f(a, b)` becomes `def f(*, a, b)`, `def m(self, a, b)` becomes `def m(self, *, a, b)`, and an ungranted `def render(node)` becomes `def render(*, node)`. (The all-keyword form is always compliant and needs no grant — the fixer takes it.) `missing-star`, `ungranted-subject`, and `literal-subject` violations are all fixable this way; the registry-level kinds (`grant-param-mismatch`, `dead-grant`, `unsorted-grant`) are never rewritten — the fixer edits signatures, and the registry has exactly one writer, the grant command. The raw insert lands on the def's original line(s); the `make fix-keyword-only` / `fko` target runs `ruff format` right after the rewrite, so the standalone path leaves a `ruff format`-clean tree (inside `agent-check` the later `format` step would normalize it anyway). The rewrite is idempotent and is re-parsed before being written — a rewrite that would not parse is discarded and the violation reported instead.

A few shapes the guard flags cannot be fixed by a single bare-`*` insert, and are reported for a manual fix rather than rewritten:

- a `*args` is present — the parameters before it cannot be made keyword-only by a bare `*`, and a bare `*` cannot coexist with `*args`;
- a keyword-only section already exists (a bare `*` is already in the signature, with a positional parameter still ahead of it) — the existing `*` must be moved by hand, since a second one is a syntax error. This includes the common `def f(subject, *, opt)` shape when the subject is ungranted or literal-typed;
- two or more positional-only parameters (before a `/`, excluding a leading `self`/`cls`) remain — a bare `*` cannot precede the `/`, so they stay positional and cannot reach compliance mechanically. (A *single* positional-only subject still gets the `*` after the `/`, e.g. `def f(a, /, b, c)` → `def f(a, /, *, b, c)` — but the positional-only subject itself then still needs a grant or the `/` removed by hand.)

The `--fix` path is **non-gating**: it mutates and reports, but never exits non-zero on the violations it couldn't fix. The read-only `check-keyword-only` (no flag) owns the gating — it runs last in `agent-check` and is the variant `make check` / CI use. So `make fko` fixes what it can and prints the rest without failing your shell; run `make check-keyword-only` (alias `cko`) when you want the pass/fail gate. This split is deliberate: a tree-mutating step shouldn't also be the hard gate, or an abort could leave files rewritten-but-unformatted and mask the type-check phase.

Auto-fix is not a substitute for review: the guard is blind to framework-positional callers (see the safety-net note above), so a signature it "fixes" might be one some framework invokes positionally. And a fixed `ungranted-subject` def might have *deserved* its positional subject — if so, revert the def and record the grant instead. `make agent-test` remains the safety net — verify after a bulk auto-fix.

Run the guard directly to see the full picture:

```bash
make check-keyword-only            # alias: make cko — one line on pass, the per-kind violation list on fail
make fix-keyword-only              # alias: make fko — auto-insert the bare `*`; reports any manual-fix cases
make subject-grant FUNC="pipelex/….py::qualname" RATIONALE="…"   # alias: make sgr — record a subject grant
.venv/bin/pipelex-dev check-keyword-only --report   # full inventory by package + per-package grant counts
.venv/bin/pipelex-dev check-keyword-only --fix      # auto-fix (what `make fko` runs)

# Lean single-file check (what the PostToolUse hook runs): stdlib only, invoked by file path so it
# skips the pipelex package import chain. Prints violations to stderr and exits 2, else exits 0.
.venv/bin/python pipelex/cli/dev_cli/commands/keyword_only_guard.py pipelex/some/edited_file.py
```


## Hub Layering

# Hub layering convention

Pipelex resolves its cross-cutting dependencies through a *hub*: a singleton container plus a set of module-level accessor functions, so a component can reach the config, a worker, or the pipe library without importing the module that owns it. There are **two** hubs, one per layer, and which one a symbol lives on is an architectural boundary, not a filing preference. This document is the canonical specification of that boundary.

The two layers are the textbook language-implementation split:

- **The kernel layer** is the machinery present at execution time whatever is loaded — config, console, secrets, storage, telemetry, the model deck and inference workers, the content generator, reporting, and the value data model. Its hub is `runtime_hub`.
- **The interpreter layer** reads a method and executes it — the libraries, the pipes, the router, the pipeline manager. Its hub is `interpreter_hub`.

Do not read `runtime_hub` as related to the `pipelex.runtime_bridge` package: they share a word, but the hub is the kernel layer's service container while `runtime_bridge` is a transport.

The layer used to be called the *runtime* layer, and two module names still carry that former name: `runtime_hub` (the kernel layer's hub) and `runtime_boot` (its composition root). They are pinned across repo boundaries, so they are renamed when the kernel layer moves into its own package rather than piecemeal. `runtime_bridge` means "runtime" in the orchestration-venue sense and keeps the word for good.

## The one rule

> `interpreter_hub` may import `runtime_hub`. **`runtime_hub` must never import `interpreter_hub`.**

That single arrow is the whole architecture. Everything below is a consequence of it.

Only the *forbidden* direction is load-bearing. The permitted one is currently unused: the one kernel-layer thing `interpreter_hub` needs is the class-registry scoping slot, which lives below *both* hubs (see [The class-registry exception](#the-class-registry-exception)). So today, importing either hub loads the other in neither direction — which is stronger than the rule requires, not a violation of it.

There is deliberately **no** `pipelex.hub`. It was deleted rather than kept as an alias for either layer, so a stale import fails loudly at import time instead of silently resolving to the wrong layer.

## The two layers

### `pipelex/runtime_hub.py` — process-scoped infrastructure

`RuntimeHub` holds what is configured once at boot and never varies per method:

| group | symbols |
| --- | --- |
| container | `RuntimeHub`, `get_runtime_hub`, `set_runtime_hub` |
| config | `get_required_config`, `get_optional_config` |
| console | `get_console` |
| secrets | `get_secrets_provider`, `get_secret` |
| system registries | `get_class_registry`, `get_func_registry` |
| storage | `get_storage_provider` |
| telemetry | `get_telemetry_manager`, `get_otel_tracer` |
| models | `get_models_manager`, `get_model_deck`, `get_sdk_client_manager` |
| inference | `get_inference_manager`, `get_llm_worker`, `get_img_gen_worker`, `get_extract_worker` |
| content generation | `get_content_generator`, `scoped_content_generator` |
| reporting | `get_report_delegate`, `is_in_isolated_execution` |
| run mode | `is_dry_run_forced`, `resolve_run_mode_for_boot` |
| tracing | `scoped_event_log`, `get_event_log_override` |
| plugin registries | `get_inference_backend_registry`, `get_model_lister_registry`, `get_orchestrator_registry`, `get_bundle_validator_registry`, `get_storage_provider_registry`, `get_secrets_provider_registry` |

### `pipelex/interpreter_hub.py` — library-scoped method machinery

`InterpreterHub` holds what belongs to the *loaded method*:

| group | symbols |
| --- | --- |
| container | `InterpreterHub`, `get_interpreter_hub`, `set_interpreter_hub` |
| library manager | `get_library_manager`, `get_library` |
| library lookups | `get_concept_library`, `get_required_concept`, `get_native_concept`, `get_required_domain`, `get_optional_domain`, `get_pipe_library`, `get_pipes`, `get_required_pipe`, `get_optional_pipe`, `get_required_entry_pipe`, `get_optional_entry_pipe`, `get_pipe_source` |
| current-library contextvar | `set_current_library`, `get_current_library`, `get_current_library_id_or_none`, `clear_current_library`, `scoped_current_library` |
| library dirs | `resolve_library_dirs`, `get_default_library_dirs` |
| pipe router | `get_pipe_router`, `set_pipe_router`, `teardown_current_pipe_router`, `scoped_pipe_router` |
| run | `get_pipe_run`, `get_pipeline_manager`, `get_pipeline` |
| pipe func | `get_pipe_func_executor`, `scoped_pipe_func_executor`, `get_pipe_func_executor_registry` |

Setters follow their getters onto the matching container.

### Which hub does a new symbol belong to?

Ask **what its lifecycle is**, not what package it happens to be typed by:

- Configured once at boot from config + plugin registration, identical for every method the process runs → `runtime_hub`.
- Tied to a loaded method's libraries, or varying with the current library → `interpreter_hub`.

Placement is decided by *kind*, not by who currently imports it. Several accessors have no in-tree importer (`get_bundle_validator_registry`, `get_orchestrator_registry`, `get_default_library_dirs`, …) — they are live plugin contract and are reached through the container. Do not relocate or delete one because a grep looks empty.

`get_pipe_func_executor_registry` is the one placement worth flagging: it is a plugin registry by kind (which would put it in the kernel layer) but its protocol lives in `pipe_operators/func/` (which puts it in the interpreter layer). It is **interpreter-layer**. See [Known inversions](#known-inversions) for the underlying wart.

Not everything a hub module exports is an accessor. `resolve_run_mode_for_boot` *applies* the keyless-boot forced-DRY flag rather than reading it, and it lives beside `is_dry_run_forced`, the accessor it consumes. The lifecycle question above still decides placement — the flag is set once at boot and identical for every method the process runs, so the rule that interprets it is kernel-layer by kind. What makes the *module* the right home rather than either caller is that there are now two: the pipe tier's `PipeRunParamsFactory.make_run_params` and the kernel tier's `PipelexKernel.make` both mint run params, and a boot-scoped invariant copied into each is one that drifts. When a rule over hub state acquires a second caller, move the rule to the hub — do not let the second caller reimplement it.

## Why the boundary exists

`runtime_hub` must not name anything from `libraries`, `pipe_operators`, `pipe_controllers`, `codegen`, `builder`, `interpreter_plugins`, `pipe_machinery`, `pipe_signature`, `mthds_parsing`, `pipeline` or `pipe_run` at module level. That list is now stated in whole top-level packages, which is the point of the modularity moves: it used to have to trail "…or the Pipe-touching modules of `core.pipes`", because some of what it forbids lived under a runtime-named package. Those module-level imports exist only to type the getters, but they are what made a single hub drag the entire method interpreter into every consumer that just wanted `get_console()`.

The property that matters is measurable: **importing the Pipelex kernel layer loads zero modules from any of the packages named above.** That is both the assertion the closure test pins and the outward-facing claim — the inference engine does not know the MTHDS language exists, so you can embed it without loading a line of the parser, the pipe machinery, or a single pipe kind. Verify it from the repo root on a synced venv:

```bash
.venv/bin/python - <<'PY'
import sys
from pathlib import Path

import pipelex.cogt.content_generation.content_generator  # noqa: F401

INTERPRETER = {"libraries", "pipe_operators", "pipe_controllers", "codegen", "builder", "interpreter_plugins", "pipe_machinery", "pipe_signature", "mthds_parsing", "pipeline", "pipe_run"}
loaded = {name: mod for name, mod in sys.modules.items() if name.startswith("pipelex.")}
interpreter = sorted(n for n in loaded if n.split(".")[1] in INTERPRETER)
sloc = 0
for mod in loaded.values():
    file = getattr(mod, "__file__", None)
    if file:
        sloc += sum(1 for line in Path(file).read_text(encoding="utf-8").splitlines() if line.strip() and not line.strip().startswith("#"))
print(f"pipelex modules: {len(loaded)} | sloc: {sloc} | interpreter modules: {len(interpreter)}")
print("offenders:", interpreter)
PY
```

`interpreter modules` must print **0**. When it does not, the `offenders` list names the modules that leaked, and the shortest import path to one of them is what to fix. Swap the imported module to measure any other entry point.

## Where core splits

`pipelex/core/` used to be two layers under one name, and the kernel-layer declaration had to work around it. It does not any more — `pipelex.core` is a single declared entry — but the split is worth recording, because *how* it was resolved is the general lesson.

For a long time `core/` held two different kinds of thing:

- **The data model — runtime.** `core.concepts`, `core.domains`, `core.stuffs`, `core.memory`, the input/output *specs* under `core.pipes.inputs` and `core.pipes.stuff_spec`, and `core.pipes.pipe_output`. These describe what a method's values *are*. Nothing in them needs a loaded method, and each measures **zero** modules from the packages the predicate names. `pipe_output` reads as Pipe machinery but is not: it names no `Pipe` — it holds the working memory, graph spec and usage a run *produced* — and it sits inside `runtime_hub`'s own closure (the chain under [The class-registry exception](#the-class-registry-exception) passes straight through it).
- **The Pipe machinery — interpreter.** `pipe_abstract`, `pipe_blueprint`, `pipe_factory`, `validation`, `template_guard_lint` and the two renderers. Every one of them names a `Pipe`, and a pipe is the interpreter's own object — they import `pipe_operators` / `pipe_controllers` / `libraries` **directly**, not through a hub. No amount of dependency injection makes those kernel-layer.

**The resolution was to move the second kind out, not to describe it more precisely.** All of it now lives in two top-level packages of its own: `pipelex.pipe_machinery` holds the Pipe machinery proper plus the pipe-kind registration manifest (`registry_models.PipeRegistryModels` — the fattest edge of the lot, a list of every pipe class and every pipe factory, so importing it loaded the whole interpreter), and `pipelex.mthds_parsing` holds the MTHDS parser and the bundle blueprint it produces, whose `pipelex_bundle_blueprint` is a discriminated union over every pipe blueprint. See [Registration surface](registration-surface.md) for what adding a pipe kind touches.

One model stayed behind on purpose, and it is the interesting case. `PipelexBundleBlueprintValidationErrorData` is raised by the parser but imported by `pipeline/` and `libraries/`, which carry it into every kernel-layer import closure. Filing it with the parser would have made the boundary unenforceable for `mthds_parsing` — the closure predicate would have needed a permanent exclusion for it. It lives in `core.exceptions` instead, beside the two sibling structured error-data models it shares `PipeValidationErrorType` with. **Moving the leaf is what buys the clean predicate**; excluding it would only have recorded the problem.

So the declaration names `pipelex.core` wholesale, which is a claim the measurement now supports: every `pipelex.core.*` module loads zero modules from the interpreter packages the predicate names, and none reaches `interpreter_hub`. It once listed core's data-model packages one by one, and that enumeration was the shape of the problem — a package-granular declaration is only honest when the packages match the layers.

The dividing line is worth stating as a rule of thumb: **if it names a `Pipe`, it belongs to the interpreter layer.**

### Injected providers, not ambient lookups

The data-model half used to reach for `get_concept_library()` / `get_native_concept()` / `get_required_concept()` — ambient lookups into a loaded method, which is exactly what made `core/` inseparable from the interpreter. It now takes what it needs as a parameter, through one read-side contract: `ConceptProviderAbstract` (`pipelex/core/concepts/concept_provider_abstract.py`) — resolve a ref, a code, or a native code into a `Concept`, answer compatibility questions about two of them, and resolve a concept's declared `structure_class_name` into the class itself.

**Class resolution is part of that contract, and it took a second pass to get there.** The hub split moved *concept* resolution onto the provider but left *class* resolution ambient one level down, inside `Concept` itself: the model reached `get_class_registry()` to answer "what is my structure class?" and to run the structural half of a compatibility check. A `Concept` is a subclass of the MTHDS-protocol wire model `ConceptAbstract` — pure serializable data — so that coupled a standard-owned wire model to a Pipelex process-global. `ConceptProviderAbstract.get_structure_class` decouples them: a `Concept` carries the class *name* as a plain protocol string, and turning a name into a type is the provider's business, never the model's.

Be precise about what that buys, because it is easy to overstate. The read is still **ambient** — `ConceptLibrary.get_structure_class` calls `get_class_registry()`, which selects a registry from the current async context rather than from the provider instance (a `ConceptLibrary` holds none; the per-library `ClassRegistry` lives on `Library`). What changed is that the whole tree now has *one* implementation of that read, behind one seam, instead of it happening inside a wire model. Scoping it to the provider's own library is consequently a one-place change — see `wip/inputs/provider-scoped-class-resolution.md` for why nothing needs it yet.

Two shapes fall out of that, and they are the ones to copy:

- **Compatibility is two tiers, composed by the thing that owns resolution.** `Concept.are_compatible_by_declaration` is pure over the model's fields (plus an injected `concept_resolver`): dynamic short-circuits, ref equality, declared-class-name equality, `refines` chains. `ConceptLibrary.is_compatible` asks that first and only resolves classes when it is inconclusive. A name that should have resolved and didn't now raises `ConceptStructureClassNotFoundError` instead of answering `False` — "unknown" and "incompatible" are different answers, and conflating them was a latent wrong-verdict bug.
- **Rendering takes the resolved class, not the name.** `Concept.render_concept_representation` receives a `structure_class`; `StuffSpec.render_stuff_spec` takes the `concept_provider`, resolves once, and passes the class down. Interpreter-layer callers (`output_renderer`, `input_renderer`, `pipe_io_contracts`, `builder/runner_code`) supply `get_concept_library()`.

The **write side** stays ambient on purpose: `concept_factory` and `structure_generation/generator.py` generate structure classes and register them at library-load time, which is genuinely the registry's business. That those two are the only modules under `pipelex/core/concepts/` touching the registry is pinned as a golden set by `tests/unit/pipelex/core/concepts/test_concept_registry_boundary.py` — nothing else can see it, since both the accessor and its users are kernel-layer.

It is **read-side only**. Managing a library — adding, removing, listing, setup/teardown — stays in the interpreter layer: `ConceptLibraryAbstract` *extends* it, keeping its management half in `libraries/`. Splitting read from write is what lets a core module state its dependency honestly ("I need something that can resolve concepts") without inheriting a library lifecycle it has no business touching.

**There is deliberately no pipe counterpart.** Nothing in core's kernel half resolves a pipe. The two places that follow a pipe reference found *inside* a pipe graph — a condition's mapped pipes, a sequence's last step — live in `pipe_machinery/rendering/`, which is interpreter-layer and calls `interpreter_hub.get_required_pipe` directly; a provider parameter there would buy no closure property. `get_required_pipe` therefore stays on `PipeLibraryAbstract`. Split a read half out when a kernel-layer caller actually needs one, not for symmetry with concepts.

Pipe resolution comes in two flavours on this hub, and picking the wrong one is a correctness bug rather than a style choice. `get_required_pipe` / `get_optional_pipe` resolve an **in-body** reference — one pipe naming another from inside a method — and are strict: the ref names its own domain or it names nothing. `get_required_entry_pipe` / `get_optional_entry_pipe` resolve a code a **human** supplied at an entry point (a CLI argument, an API request field), where a bare code is still matched across domains. A CLI command on the strict pair is needlessly strict; a controller on the entry pair reopens the cross-domain hole that makes `[exports]` unenforceable.

One caveat on "strict", so nobody relies on it absolutely: a **cross-package** `alias->bare_code` ref is still resolved by searching the aliased entries, because the qualification pass leaves `alias->…` refs alone — it cannot know a dependency's domain layout, so there would be no canonical spelling to migrate such a ref to. That search is alias-scoped and cannot reach a host pipe, so it does not reopen the hole; it is a deferral to the packaging design, not an exception to the rule.

The injection point is the interpreter layer. `pipe_factory`, `input_renderer` and `output_renderer` — already interpreter-bound, so nothing is lost — resolve the concept library themselves and pass it down; `pipeline/execution_seams.py` does the same for `WorkingMemoryFactory`. That is the one-way arrow the whole design rests on, applied inside `core/`: **the half that knows about a loaded method resolves the collaborator and hands it downward.**

When you add a core data-model function that needs a concept — or a concept's structure class — add a `concept_provider` parameter. Do not add a hub import, and do not reach for `get_class_registry()`: the guard will reject the first, and the second is the coupling this section exists to describe.

One module outside `core/` cannot import either hub at module level, and the constraint is a **static-analysis** cycle rather than a runtime one — worth knowing before you restate it. `pipe_machinery/pipe_abstract.py` reads `class_registry_access` directly to decorate graph-registry entries with a JSON Schema. It is *not* in `runtime_hub`'s kernel closure (`plugins.orchestrator_registry` and `pipe_run.pipe_job` reach it through `TYPE_CHECKING`-only edges, and a clean-interpreter probe shows all three absent) — but pyright's `reportImportCycles` counts those edges, so either hub import is a hard type-check failure, reported at `interpreter_hub.py` where no line-level ignore can reach it. Deferring the import does not dodge it. The below-both-hubs accessor is what that case is for; its docstring records the measurement.

## Where the built-in plugins split

Two different splits run through the built-in plugins, and they are easy to conflate. The first is **mechanism vs adapters**, and it does *not* cross a layer boundary: `pipelex/plugins/` is the plugin mechanism — the `PipelexPlugin` contract, the `PluginRegistrar`, and the capability registries the registrar accumulates into — while `pipelex/providers/` holds the built-in adapters that register through it, one directory per provider. Both packages are declared kernel-layer. What the split makes visible is the one-way dependency between them: adapters depend on the mechanism, and the mechanism knows nothing of its adapters. It also stops `pipelex/plugins/` from reading as "the vendors", which is what made the second split hard to see. Note the scope of that claim — it is about these two packages, not about the whole tree: `cogt` does name four providers directly, which [Known inversions](#known-inversions) records and justifies.

Most of those directories are vendor adapters, but not all of them, which is why the package is `providers/` and not `vendors/`: `providers/secrets/` registers the built-in `env` backend, which has no vendor at all, and `providers/storage/` registers local, in-memory, S3 and GCP behind one directory. In both cases only the registration shim lives here — the implementations are under `pipelex/tools/`.

The second split *is* the layer boundary, and the same rule of thumb divides it. Most built-ins adapt a kernel-layer port — an inference backend, a model lister, a storage or secrets provider — and those are the packages under `pipelex/providers/`. Two adapt interpreter-layer ports, because their job is to *construct* interpreter-layer objects: `DirectOrchestratorPlugin` registers a `DirectOrchestrator` and a `DirectBundleValidator`, and `PipeFuncPlugin` registers a concrete `DirectPipeFuncExecutor`. Those two live in **`pipelex/interpreter_plugins/`**.

External plugins are unaffected by the mechanism/adapter split: they import `pipelex.plugins.contract` and `pipelex.plugins.registrar`, both of which stayed put. They are, however, affected by the *layer* split — they are discovered through the two `pipelex.plugins.kernel` / `pipelex.plugins.interpreter` entry-point groups, which are group identifiers rather than module paths, and a plugin declares its layer by choosing one. (Careful: `pipelex.plugins` is also the name of the kernel-layer *package* holding the mechanism, and that is the sense every other mention of it on this page carries — the declared-package list, the transitive-breach worked example, the closure-test entry points. A package name and an entry-point group are unrelated things that happen to share a spelling. The single pre-split group of that name is retired and now fails startup.)

The registrar's slots are *ports* and they stay kernel-layer — `OrchestratorProtocol`, `BundleValidatorProtocol` and `PipeFuncExecutorFactoryFn` all live under `plugins/*_registry.py`. Only the *adapters* are interpreter-layer. That is ordinary ports-and-adapters, and it is what the layer rule's own remedy text prescribes: have the interpreter layer install it downward at boot. Where each derived registry is *installed* still follows the hub partition rather than the port's home: `RuntimeBoot.setup` puts the orchestrator, bundle-validator, backend, lister, storage and secrets registries on the `RuntimeHub`, and `Pipelex.setup` puts the `PipeFuncExecutorRegistry` on the `InterpreterHub` — the one placement flagged [above](#which-hub-does-a-new-symbol-belong-to). Since [the boot split](#where-the-boot-splits) the two installs sit in the two halves that match their hubs, so the partition is now visible in the code rather than only in this sentence.

Composition happens where welding is legal. `pipelex/providers/builtins.py` exports `KERNEL_BUILTIN_PLUGINS`; `pipelex/interpreter_plugins/builtins.py` imports that half — downward, which the interpreter layer may do — and exports the composed `BUILTIN_PLUGINS` and `CORE_UNCONDITIONAL_PLUGIN_NAMES`. Both are **parameters** of `build_registrar`, never module globals it imports, so `plugins/discovery.py` stays kernel-layer: `pipelex.py` (the interpreter boot) and the `pipelex plugins list` diagnostic pass the composed lists in, while `runtime_boot.py` defaults to the kernel half alone. Exactly one place still answers "what are the built-in plugins" — it just lives in the layer permitted to say it.

This is the seam external plugins already use. Our Temporal plugin contributes an orchestrator and our Daytona plugin a PipeFunc executor; both are interpreter-touching, both live outside this repo, and both arrive through the `pipelex.plugins.interpreter` entry-point group without welding anything. The in-tree built-ins used to be the only ones that welded — [Known inversions](#known-inversions) records what that cost.

One consequence is worth stating as a contract, and it is written into `PipelexPlugin`'s docstring: **a plugin belongs to exactly one layer.** A capability needing adapters in both is two plugins. Nothing enforces it mechanically; a straddling plugin would simply put the interpreter back into every kernel closure, and the guard's transitive rule would say so.

## Where the boot splits

The composition root is split the same way, and it is the same move: a kernel half in a declared kernel-layer home, and an interpreter half that imports it downward and adds what only a method-running process needs.

`pipelex/runtime_boot.py` holds `RuntimeBoot`. It stands up config, logging, the secrets provider, telemetry, the Kajson class registry with `CoreRegistryModels`, the template sets, the SDK client manager, the model deck, the plugin-derived runtime registries, storage, the content generator, the inference manager, the reporting delegate and the observers — and it loads **zero interpreter modules** doing it. `pipelex/pipelex.py` holds `Pipelex`, which subclasses it and appends the `InterpreterHub`, the composed plugin manifests, the `PipeFuncExecutorRegistry` and its executor, the `LibraryManager` and default library dirs, the `PipelineManager`, the `PipeRegistryModels` registration, the `PipeRouter` and the `PipeRun`.

Two top-level modules, not a `boot/` package: they mirror `runtime_hub.py` / `interpreter_hub.py`, which is the tree's strongest naming precedent for a layered pair. `Pipelex` does not move and is not renamed — it is the public entry point every consumer and sibling repo imports, and the layer boundary never required moving it.

**This is what finally gives `KERNEL_BUILTIN_PLUGINS` a caller.** The plugin split created that constant with no importer in the tree; `RuntimeBoot.setup` defaults to it, so a kernel-only boot discovers exactly the kernel half and the two interpreter-touching built-ins are simply absent. The consequence is worth stating plainly, because it is the reason nothing needed redesigning to make this work: `OrchestratorRegistry`, `BundleValidatorRegistry` and the PipeFunc executor modes are all interpreter-contributed, and therefore **empty on a kernel-only boot** — which is harmless only because nothing in the kernel half *resolves* out of them at boot. They are constructed, set on the hub, and looked up at run time by the interpreter. The two registries the kernel half does resolve out of, secrets and storage, are kernel-contributed and core-unconditional.

Inheritance rather than composition, deliberately: it preserves every attribute address (`self.models_manager`, `self.class_registry`, `self.telemetry_manager` and the rest read the same from both halves, so no consumer inside or outside this repo changes), and it reads as what it is — the interpreter boot *is* the runtime boot plus the interpreter constructions. Two consequences are load-bearing:

- **The singleton resolves by subclass.** `RuntimeBoot` carries `MetaSingleton` too, and the class-level accessors go through `MetaSingleton.get_subclass_instance`, so `RuntimeBoot.is_fully_booted()` answers `True` while a `Pipelex` owns the process globals. Each `make()` asks the *base* class whether a boot exists, which is what makes the two mutually exclusive in both directions — `set_runtime_hub`, `KajsonManager` and `log.configure` are all once-per-process, so booting one on top of the other would serve a half-populated class registry.
- **Teardown ordering is explicit, not a template hook.** The plugin-contributed teardown callbacks must run *before* `pipeline_manager.teardown()`, so a worker's in-flight resources release first. `RuntimeBoot` exposes `_teardown_plugin_callbacks()` and `_teardown_runtime()` as separate phases and `Pipelex.teardown` sequences them with its own step in between, because that order is the whole reason the split exists in this shape.

Enforcement mirrors the packages': `pipelex.runtime_boot` is declared in `KERNEL_LAYER_PACKAGES` (a *module* entry, beside `pipelex.runtime_hub`) and listed in the closure test's `KERNEL_LAYER_ENTRY_POINTS`. `pipelex.pipelex` is deliberately **undeclared** — it is interpreter-side by construction, like `pipelex.cli`, so declaring it would fail the rule by design rather than enforce anything. `tests/unit/pipelex/cli/dev/test_hub_layering_guard.py` asserts both directions, because "surely the boot should be kernel-layer" is a plausible future edit.

## Placement, not coupling

Splitting the hub removes the *lookups* that crossed the boundary. It does not move a type that was simply filed in the wrong package — and a misfiled type drags its whole package into every closure that names it. Two were resolved this way; both are worth knowing as the pattern to apply to the next one.

**`JobMetadata` moved to `pipelex/system/`.** It is an argument to essentially every `cogt` call, yet it lived in `pipelex/pipeline/` — which made `cogt → pipeline` the fattest remaining edge and pulled `graph.trace_context` into every closure that touched inference. `JobMetadata`, `JobCategory` and `UnitJobId` now live in `pipelex/system/job_metadata.py`, and `JobMetadataError` moved from `pipeline/exceptions.py` to `pipelex/system/exceptions.py`. `cogt → pipeline` is now **zero statements**.

`TraceContext` moved with it, to `pipelex/system/trace_context.py`: it is the transport `JobMetadata` carries, so leaving it in `graph/` would only have renamed the inversion to `system → graph`. Its one dependency on the graph package — `DataInclusionConfig`, nested under `[...graph_config.data_inclusion]` in the TOML — moved down to `pipelex/system/data_inclusion_config.py`, which `graph_config.py` now imports. The TOML shape is unchanged; only the class's home moved, which is what keeps `trace_context` from re-importing `mermaid_config` and `reactflow_config` through `GraphConfig`.

**The templating primitives moved down into `tools/`.** `TemplateCategory`, `TemplatingStyle`, `TagStyle` and `TextFormat` sat under `cogt/templating/` while eight `tools/jinja2/` and `tools/mermaid/` modules imported them — so `tools`, the intended bottom layer, depended on `cogt`. None of the three modules holding them named anything from `cogt`, so this was pure misfiling:

- `TextFormat` and `TemplatingStyle` / `TagStyle` → `pipelex/tools/templating/`, which imports nothing from `pipelex` beyond its own sibling. It is a leaf.
- `TemplateCategory` → `pipelex/tools/jinja2/`, because its entire payload is a map of jinja2 filters. Filing it in `tools/templating/` would have made that package import `tools/jinja2` while `tools/jinja2` imports it back — a cycle inside one layer. As placed, the edges run one way: `tools/mermaid → tools/jinja2 → tools/templating`.

What stays in `cogt/templating/` — `TemplateBlueprint`, the sigil preprocessor, the rendering entrypoint — belongs there: a blueprint is language-layer, and the rest imports `tools/jinja2` downward.

**`resolved_fields` moved down into `core/concepts/`.** The neutral resolved-field layer — one structure field becoming a `ResolvedType` tree — lived under `pipelex/codegen/` while `core/concepts/structure_generation/generator.py` imported it, which put `pipelex.codegen` into the closure of every core module that reached structure generation. It names nothing outside `pipelex.core`, so this was misfiling again: it now lives at `pipelex/core/concepts/resolved_fields.py`, and the codegen emitters import it upward.

**The two renderers regrouped, then left `core/` altogether — they are now `pipe_machinery/rendering/`.** `input_renderer` sat in `core/pipes/inputs/` beside genuinely kernel-layer modules while `output_renderer` sat alone in `core/pipes/output/`. Both render a `PipeAbstract` for a human or an agent, so both belong to the interpreter layer, and leaving `input_renderer` where it was would have forced the kernel-layer declaration to be a list of *modules* rather than packages. Regrouping them into one `rendering/` package was the first half of the fix and bought a real improvement — the boundary fell on package lines instead of module lines. It was only half, though: the declaration still had to name six `pipelex.core.*` sub-packages, because the boundary now fell on package lines *inside* `core/`. Moving the whole group out to `pipe_machinery/` is what collapsed those six entries to one. The general form: regrouping tells you where the boundary is, moving is what makes the boundary a package name.

The lesson generalizes: when a kernel-layer package imports an interpreter-layer one, check whether the *type* is misplaced before designing an indirection. None of these needed a resolver slot or a protocol — only a `git mv` and an import rewrite. And when a package straddles the boundary, moving the odd module out is usually cheaper than teaching the guard about exceptions — which is exactly how the built-in plugins were split, [below](#where-the-built-in-plugins-split).

## The class-registry exception

`get_class_registry` is the one kernel-layer accessor that is *not* implemented in `runtime_hub`. Its implementation lives in `pipelex/system/registries/class_registry_access.py`, and `pipelex.runtime_hub.get_class_registry` delegates to it.

The reason is a genuine cycle. `core.concepts.concept` needs the active class registry, and it sits **inside** `runtime_hub`'s own import closure:

```
runtime_hub → cogt.llm.llm_worker_abstract → system.telemetry.otel_factory
            → core.pipes.pipe_output → core.stuffs.stuff → core.concepts.concept
```

so `concept.py` cannot import `runtime_hub` at module level. Hosting the accessor in a module that imports nothing from `pipelex` is what lets `core/concepts/` use a plain top-level import — replacing three identical `importlib.import_module("pipelex.hub")` shims that were invisible to every import lint and to pyright's module graph.

**Use `pipelex.runtime_hub.get_class_registry` everywhere.** The exception is a module that must stay import-light *with respect to* `runtime_hub` — that one imports the leaf directly. `core/concepts/` is the case, for two different reasons: `concept.py` sits **inside** `runtime_hub`'s own closure, so the public accessor is literally unreachable there; `concept_factory.py` and `structure_generation/generator.py` sit outside it, and routing them through `runtime_hub` would re-couple `core.concepts` — a declared kernel-layer package — to the whole cogt/plugin stack for one function. Do not "fix" those two to the public accessor: nothing checks this mechanically, and the closure test would not catch it either, since it measures interpreter modules rather than weight.

### Library scoping crosses downward, at install time

A run may pin a per-library `ClassRegistry` rather than the process-global Kajson one. Knowing *which* library is current is method-layer knowledge, so the kernel layer holds only a slot: `class_registry_scoping`, defaulting to a resolver that returns `None`. `set_interpreter_hub` installs the real resolver, which means scoping is live exactly when a `InterpreterHub` exists — an invariant a caller cannot forget to wire. The runtime teardown phase (`RuntimeBoot._teardown_runtime`, reached from every `teardown`) calls `class_registry_scoping.reset()` so a torn-down library manager is never reachable through a still-pinned library id. It lives on the kernel half because releasing the slot belongs to whichever teardown runs — on a kernel-only boot nothing installed a resolver and the reset is a no-op.

This is the same shape as `HubSlot.ISOLATED_EXECUTION_PROBE`: a defaulted callable a higher layer replaces at boot, never an unset attribute. A process that only ever builds a `RuntimeHub` therefore degrades to the process-global registry instead of raising. That used to describe only the doctor path and most unit tests; since [the boot split](#where-the-boot-splits) it is a first-class entry point — `RuntimeBoot.make()` installs no `InterpreterHub`, and a test pins that scoping stays at its unscoped default on that boot.

## Enforcement

The boundary is enforced mechanically. Two things are checked, because the rule and the property it buys can fail independently.

### The rule — `make check-hub-layering`

```bash
make check-hub-layering   # alias: make chl
```

An AST guard (`pipelex-dev check-hub-layering`, core in `pipelex/cli/dev_cli/commands/hub_layering_guard.py`) that runs in `make agent-check`, in the `make check` aggregate, and in CI. It checks three rules over `pipelex/` and `tests/`:

1. **The layer rule.** A module in the declared kernel layer — `pipelex.cogt`, `pipelex.core`, `pipelex.errors`, `pipelex.graph`, `pipelex.kernel`, `pipelex.observer`, `pipelex.plugins`, `pipelex.providers`, `pipelex.reporting`, `pipelex.system`, `pipelex.test_extras`, `pipelex.tools`, `pipelex.tracing`, plus the `pipelex.runtime_hub` and `pipelex.runtime_boot` modules themselves — may not import `pipelex.interpreter_hub`. The declaration is compliant with an empty exception list, so the guard hard-blocks on **any** violation. Entries are matched exact-or-dotted-prefix, which is why a bare *module* is a legal entry beside the packages: without `pipelex.runtime_hub` spelled out, the module at the centre of the rule would be exempt from it, and without `pipelex.runtime_boot` the kernel layer's own composition root would be — see [Where the boot splits](#where-the-boot-splits). `pipelex.core` was six entries until its interpreter tenants moved out — see [Where core splits](#where-core-splits) for what that enumeration was working around.

    `graph`, `tracing`, `observer` and `errors` joined the list late, and the delay cost something — see [Known inversions](#known-inversions). They are the run-graph data model with its tracer and renderers, the trace-event assembler feeding it, the run-observation hooks, and the error taxonomy: machinery present at execution time whatever is loaded, which is the kernel layer's own definition. `pipelex.test_extras` is on the list for the same reason: it ships, `runtime_boot.py` imports it at boot, and it measures clean. `pipelex.kernel` is the same rule applied *ahead* of a breach instead of behind one: it was declared in the commit that created the package, because it holds the operator-execution semantics the interpreter's operator classes call into, and a single `interpreter_hub` reach anywhere in it would put the interpreter back into every programmatic caller — the property the package exists to provide. `pipelex.kit` is data files with no module to police; `pipelex.language`, `pipelex.runtime_bridge` and `pipelex.cli` all measure dirty and are interpreter-side by construction. That is every top-level package accounted for — which is the point, because the rule of thumb is that a measured-clean package should be *declared*, not left undeclared: **an undeclared package is not neutral, it is unpoliced.**
2. **The dead-module rule.** *No* scanned module may reference `pipelex.hub`. It was deleted rather than aliased so a stale import fails loudly; this closes the one hole in that guarantee.
3. **The transitive rule.** No kernel-layer module may *reach* `pipelex.interpreter_hub` either, through any chain of module-level imports. Rules 1 and 2 are per-file and see one hop; this one resolves the whole module-level import graph of `pipelex/` and does reachability over it, reporting the shortest chain and the line of its first hop. It exists because the one-hop rules missed a live breach for a whole release: four modules of the declared kernel-layer `pipelex.plugins` package reached the interpreter hub through `runtime_bridge`, `pipeline` and `pipe_operators`, and both gates stayed green (see [Known inversions](#known-inversions)). A *direct* import is left to rule 1, which names the more specific remedy, rather than reported twice.

    The rule scopes itself to what an import actually loads: module-level imports only, `TYPE_CHECKING` blocks, function bodies and `# hub-layering: ignore`d statements excluded — the same carve-outs rule 1 grants, applied to the graph. It also models what an import loads *on the way*: reaching `pkg.sub.mod` executes `pkg/__init__.py` and `pkg/sub/__init__.py` first, so those are edges too and a breach in a package init is reached by every importer of anything beneath it. And a module-level **string** naming the interpreter hub is an edge, since a dynamic import loads it just the same — rule 1 catches that string in a kernel-layer module, but an *intermediary* sits in no declared layer, so without this the graph would miss it. A finding cannot be silenced **where it is reported** — the marker suppresses one reviewed *import*, never a module's transitive reach, so a module the declaration claims is kernel-layer while the graph says otherwise gets moved or has its dependency inverted.

    The rule also refuses to pass vacuously: if the scan finds no `pipelex.interpreter_hub` among the modules it parsed — a renamed hub, or a scan root the module names cannot be derived from — it raises instead of reporting zero. Reverse reachability from a module the graph does not contain is empty by construction, so without that check a rename would silently turn the whole rule off while the gate stayed green.

All three rules match **imports and bare string literals**, and the string half is the load-bearing one. A missed *import* of a deleted module is an immediate `ImportError`; a missed *string* is not, and it is invisible to every import-graph tool and to pyright's module graph. Both forms that actually occurred here were strings: three `importlib.import_module("pipelex.hub")` shims hiding a cycle from every lint, and one `mocker.patch("pipelex.hub.get_console", ...)` that broke a whole CLI test suite with an `AttributeError` raised nowhere near a hub. Matching is exact-or-boundary against the module path, so `pipelex.runtime_hub` never matches `pipelex.hub` and prose that merely *mentions* a module is not a reference.

The guard resolves relative imports (`from ...interpreter_hub import …`) against the importing module's own package, so the forbidden arrow cannot be spelled around. A path assembled at runtime from f-strings or concatenation is beyond any AST scan; nothing in the tree does that.

Two deliberate carve-outs:

- **`if TYPE_CHECKING:` blocks are exempt from the layer rule** — the rule is about what *loads*, and a type-only import loads nothing. Only `TYPE_CHECKING` and `typing.TYPE_CHECKING` open such a block: an attribute of any other receiver (`settings.TYPE_CHECKING:`) is an ordinary runtime condition and earns no exemption. The `else` branch is not exempt either, nor is `if not TYPE_CHECKING:`, and the dead-module rule applies inside `TYPE_CHECKING` too (a deleted module exists in no phase).
- An inline `# hub-layering: ignore` comment anywhere on the offending statement suppresses it, mirroring `# kw-only: ignore`. It applies to every rule that is about what *loads*, so a suppressed import is also not an edge in rule 3's graph. The only two in the tree are on the guard's own declarations of the two hub paths, which name them rather than importing them.

`tests/` is scanned for the dead-module rule only. `tests.*` sits in no declared layer, so a test may freely patch `pipelex.interpreter_hub` — while a stale `pipelex.hub` patch target, the landmine above, still fails the check. The transitive rule likewise walks `pipelex/` only: a test module has no closure to protect.

### The property — the import-closure test

`tests/unit/pipelex/test_kernel_layer_import_closure.py` imports each kernel-layer entry point in a **subprocess** and asserts that no module from its `INTERPRETER_PACKAGES` set — and no `pipelex.interpreter_hub` — landed in `sys.modules`. This is the [measurement above](#why-the-boundary-exists), pinned. It exists separately from the lint because the two fail independently: a kernel-layer module reaching directly into `pipe_operators`, without touching a hub, breaks the property while the lint stays green.

The entry points are the inference layer, `runtime_hub` itself, `providers.builtins`, and the heaviest module of each kernel-layer `core/` package. `providers.builtins` earns its place by history, under its former name `plugins.builtins`: it instantiates every built-in vendor adapter, and `pipelex.plugins` — then the largest declared kernel-layer package, vendors and mechanism together — had no entry point here when the transitive breach happened. The guard now follows the import graph, but that is *static* analysis and cannot see a dynamic import, so the package that bit us gets a runtime-truth check too. One entry point still covers both halves of what has since become two packages: importing the manifest pulls in every built-in adapter, and they reach the mechanism they register through.

Its predicate is now a single membership test over whole top-level packages — no per-module list and no exclusions. It used to need both: core's Pipe-machinery modules had to be named one by one (most were caught only *transitively*, through the `pipe_operators` / `libraries` they pull in, and `pipe_blueprint` pulled in none of it, so a kernel-layer import of it would have passed a package-only predicate), and `core.bundles.exceptions` had to be excluded by exact match because its validation-error data landed in every kernel closure. Both disappeared when the modules moved: the machinery to `pipe_machinery`, the parser to `mthds_parsing`, and the one leaf model to `core.exceptions`. **A package-granular predicate is honest only once the packages match the layers** — that is the whole argument for the moves, stated as a check.

The predicate now names **every** interpreter package. `pipeline` and `pipe_run` were the last two absent, and the reason was placement, not a broken arrow: four leaf models of theirs landed in every kernel closure, so naming them would have failed every entry point over an address. The remedy was the one `mthds_parsing` used — move the leaves first, then widen the predicate — and where each leaf went is decided by what it *is*, not by which package used to hold it:

| leaf | was | is now | why there |
| --- | --- | --- | --- |
| `SpecialPipelineId` | `pipeline.pipeline_models` | `system.job_metadata` | it is the vocabulary of `pipeline_run_id`, and that is `JobMetadata`'s field |
| `PipeRunMode` | `pipe_run.pipe_run_mode` | `system.pipe_run_mode` | a run-scope directive, not one of a method's values |
| `PipeRunParamKey` | inside `pipe_run.pipe_run_params` | `system.pipe_run_param_key` | same, and `core.concepts` needs it as its reserved-names list |
| `PipeRunError` | `pipe_run.exceptions` | `core.pipes.exceptions` | the kernel layer *subclasses* it — `PipeRunInputsError` and `OptionalValueAbsentError` derive from it in `core.pipes.inputs` |

Note the last row does not follow the first three into `system/`. Consistency of destination is not the criterion — what the symbol *is* decides. Three are run-scope directives and enums; the fourth is the base of a pipe-error family whose other members already live in `core/pipes/`, and filing it with them also lands it in the same error-reference section as the two subclasses that derive from it.

The detector runs in a subprocess, so its logic lives in a `textwrap.dedent` string that no linter or type checker sees — and a predicate that has quietly stopped detecting would make every entry point pass *vacuously*. Three guards close that, each covering what the others cannot:

- **A negative control.** `pipelex.interpreter_hub` is dirty by definition, and the test asserts it comes back *reported as offending modules*, not merely non-zero — so a broken predicate fails rather than going quietly green. It proves the detector detects, but only that *some* name matches.
- **Every configured name is a real package directory** under `pipelex/`. A typo'd or retired name matches nothing, and because the predicate is a membership test that is invisible rather than loud. (The package set is a module-level constant handed to the subprocess as argv, so the names are at least lint-visible; nothing checks a string against disk.)
- **The set matches the `INTERPRETER` set** in this page's [verification snippet](#why-the-boundary-exists) — a reader who runs a stale copy measures zero and believes it. The two had silently disagreed before anyone compared them by machine.

Each subprocess is time-bounded, because an unbounded one turns a deadlock into a hung suite instead of a failure.

Alongside them, `tests/unit/pipelex/test_hub_lifecycle.py` pins that a boot installs both singletons, and that the production `Pipelex.teardown` — not a stand-in calling `class_registry_scoping.reset()` directly — really releases the scoping an `InterpreterHub` installed, and that the next boot re-installs it. `tests/unit/pipelex/test_runtime_boot_closure.py` pins the boot-time half of the property: it *runs* `RuntimeBoot.make()` in a subprocess and asserts zero interpreter modules loaded, no `InterpreterHub` installed, and scoping left unscoped. A subprocess is required rather than preferred — both hubs are sticky class-attribute singletons that teardown never clears, so an in-process check would answer with a stale hub and pass vacuously. Booting rather than importing is what catches a boot step resolving out of an interpreter-contributed registry that is empty on that boot.

`tests/unit/pipelex/kernel/test_kernel_boot_contract.py` takes the same escalation one step further and is the only gate that closes the function-local blind spot [the aggregate inversion](#known-inversions) exposed. It boots kernel-only, *calls* a Pipelex-kernel op on that boot, and sweeps `sys.modules` **after** the call rather than after the import. The order is the whole mechanism: a deferred import inside a kernel function lands in `sys.modules` during the call and nowhere else, so the static guard and the import-closure test both stay green on it — verified by injecting one, not assumed. For `pipelex.kernel` this is also the package's contract rather than a consequence of it, since the reason the kernel exists is that a programmatic caller can run operator semantics with no interpreter and no library loaded.

### This document — the `hub-layering-convention` drift contract

The guard and the closure test keep the *code* honest; neither can tell whether **this page** still describes it. That is a [drift contract](drift-contracts.md): `hub-layering-convention` names the guard and both hub modules as triggers and this document as the review target, so adding or moving a hub accessor — or changing the declared kernel layer — obliges a recorded review here before the change can land. It carries no verify command, because `check-hub-layering` already gates `make check` and CI.

Two checks make that review mechanical rather than a re-read, and are the ones to run:

- Extract every public module-level symbol from `runtime_hub` and `interpreter_hub` and confirm each appears in the partition tables above. An undocumented accessor is the failure mode this contract exists to catch.
- Import `KERNEL_LAYER_PACKAGES` from the guard and confirm every declared package is named under [The rule](#the-rule-make-check-hub-layering).

## Known inversions

Named so this document stays honest.

**The one that was fixed, kept as the worked example.** Four modules of the declared kernel-layer `pipelex.plugins` package used to reach `interpreter_hub` — `direct/direct_plugin.py` and `pipe_func/pipe_func_plugin.py` as the leaves (their job is to *construct* interpreter-layer objects), `builtins.py` and `discovery.py` purely as aggregators over them. (That `builtins.py` is today `pipelex/providers/builtins.py`; it moved with the vendor adapters it aggregates when the mechanism and the adapters were split.) None imported the hub *directly*, so the guard did not flag them, and the closure test had no entry point for the package, so nothing else did either: from the moment the declaration landed it claimed a property the measurement contradicted, with both gates green. The remedy was placement, not indirection — the two leaves moved to `pipelex/interpreter_plugins/` and the composition moved to the layer allowed to weld (see [Where the built-in plugins split](#where-the-built-in-plugins-split)) — and the guard grew the [transitive rule](#the-rule-make-check-hub-layering) so the next one cannot hide the same way. Three lessons generalize: a package-granular declaration is a claim about the whole package, an aggregator inherits every breach it aggregates, and a rule that stops at one hop will eventually be walked around.

**The one the worked example predicted, found by declaring the package.** `graph/graph_rendering.py` imported `pipeline.dry_run_pipeline` at module level, and importing it pulled in `interpreter_hub` plus ~76 interpreter modules. Every gate stayed green, for a reason worth stating plainly: rules 1 and 3 both filter their candidates through `is_kernel_layer`, and `pipelex.graph` was not declared — so the module had been outside the rules' domain the whole time. This is the `pipelex.plugins` breach again, one level up: not a package whose declaration overreached, but a package with **no** declaration at all. The remedy was placement, as before. The module split along "do I need a loaded method?", which is this boundary's own question: the pure renderer (`GraphFormat`, `render_graph_from_spec`) stayed in `pipelex.graph`, and the bundle-driven half that has to dry-run a method first (`generate_graph_for_bundle`, `generate_view_for_bundle`) became `pipelex.pipeline.bundle_graph_rendering`, next to the `dry_run_pipeline` it exists to wrap. The lesson is the one this page now states under [The rule](#the-rule-make-check-hub-layering): **an undeclared package is not neutral, it is unpoliced** — omitting an entry makes the guard quieter, never louder.

**The one an aggregate module hid.** Five built-in vendor adapters — `providers/{anthropic,bedrock,google,mistral}/*_list.py` — imported `MissingDependencyError` from `pipelex.exceptions` rather than from its definition site in `pipelex.system.exceptions`. `pipelex/exceptions.py` is the *public all-errors aggregate*: it re-exports every interpreter package's exceptions, so importing anything from it loads `pipeline.exceptions` → `validation_errors` → `fixes.planner` and more. Each of those five declared kernel-layer modules therefore loaded 15 interpreter modules. Neither gate could see it: the guard's rules only chase `interpreter_hub`, and the aggregate never reaches it; the closure test's `providers.builtins` entry point does not cover these modules either, because the vendor plugins import them *inside a function*, which hides them from the static import graph and from the aggregator's own closure at once. The fix is the repo's existing rule — *direct full-path imports everywhere*, i.e. import from where the symbol is defined, never from a convenience aggregate — plus one new entry point (`providers.anthropic.anthropic_list`) so the runtime-truth check now covers the shape. The generalizable lesson: **a module that re-exports across layers is a layer boundary with the sign filed off**, and a deferred import will hide it from every static gate you have. Since then the rule is mechanical rather than prose: `tests/unit/pipelex/test_kernel_layer_exceptions_aggregate_gate.py` AST-walks every package declared in `KERNEL_LAYER_PACKAGES` and fails on any import of `pipelex.exceptions` — module-level or function-local, absolute or relative — as well as on any bare string naming it, since an `import_module` argument or a patch target loads the aggregate through a path no import node records. The next aggregate reference in the kernel layer is caught by CI instead of by review. The aggregate itself remains a legitimate public surface for callers *outside* the kernel layer; the ban is scoped to the layer whose closure it would silently widen.

- **`cogt` names four specific vendors, deliberately.** `cogt/config_cogt.py` imports `AnthropicConfig`, `GoogleConfig`, `MistralConfig` and `OpenAIConfig` from their `pipelex/providers/` packages. Both sides are kernel-layer, so this breaches no hub rule — it is a placement inversion: the vendor-neutral engine names four vendors. It is accepted rather than fixed, because the alternative is worse. The main config model is statically typed end-to-end (`system/configuration/configs.py` and the `pipelex.toml` files are structurally synced, and boot fails loud when they disagree), so making vendor config sections plugin-contributed would trade compile-time typing for a dynamic registry — a real loss to buy a cosmetic win. The edge is one-way and shallow: a config class, not a worker.
- **The img-gen one that stopped being an inversion.** `cogt/img_gen/img_gen_args_factory.py` used to import two vendor img-gen factories at module level, for taxonomy and geometry helpers. It was never a registry problem: dispatch is keyed on `AspectRatioTaxonomy`, a `cogt`-owned enum that names model families and no providers, and no family is served by a single adapter — the `openai` and `azure_openai` decks both ship GPT Image models, and the gateway worker resolves Gemini geometry through the same mapping the native Google worker uses — so there was no provider key to dispatch on, and a registrar slot would have bought a plugin-API bump for two import sites. Fixed by placement instead: the helpers are now `cogt/img_gen/img_gen_gemini_mapping.py` and `cogt/img_gen/img_gen_gpt_mapping.py`, and the Google worker and the gateway factory import *inward* to them. One `cogt → specific vendor` edge of this kind remains — `model_backends/backend_factory.py` reaching for `VertexAIFactory` — already deferred inside a function, and it self-resolves when VertexAI support is removed.
- **What pins the two bullets above.** The complete `cogt → pipelex.providers` list — the four config imports plus that one deferred factory import — is a golden set in `tests/unit/pipelex/cogt/test_cogt_dependency_boundaries.py`, together with the stronger fact that `pipelex/cogt/**` imports no vendor SDK at all. A sixth edge fails it, and nothing else in the repo would notice: both packages are kernel-layer, so the guard and the closure test are blind to an edge between them by construction. Golden sets rather than a rule, because these are *accepted* inversions — there is no predicate separating the sanctioned edges from a new one, and the point is to make the next one a diff a reviewer reads. Note the tempting generalization — *no vendor SDK anywhere in `pipelex/` outside `providers/`* — is **false**, and a test asserting it would be red on day one: `tools/pdf/pypdfium2_renderer.py`, `tools/storage/`, `tracing/` and `reporting/` import pypdfium2 and the AWS/GCP SDKs on purpose, because `providers/storage/` is a registration shim whose implementations live under `pipelex/tools/`. The scope that holds is `cogt`.
- `plugins/pipe_func_executor_registry.py` is typed by a protocol from `pipe_operators/` and defers that import under `TYPE_CHECKING` precisely because `pipelex.config` imports it and the inference layer imports `pipelex.config` — a module-level import there would drag the interpreter back into every inference closure. Being type-only it loads nothing, so it breaches neither rule — the sanctioned deferral pattern. The placement inversion itself is unfixed.
- **The one that stopped being an inversion.** `core/`'s Pipe-touching half — `pipe_abstract`, `pipe_blueprint`, `pipe_factory`, the renderers — imported the interpreter directly while sitting under a package the kernel layer wanted to declare. That was never a wrong *import* (it is a fact about what a pipe is), it was a wrong *address*, and it forced the declaration to enumerate core's data-model packages instead of naming `pipelex.core`. Fixed by placement: the modules are now `pipelex.pipe_machinery` and `pipelex.mthds_parsing`, and the six `pipelex.core.*` entries collapsed to one. Several of them still import `interpreter_hub` on purpose, to inject downward — that is the sanctioned direction and always was.
- Broader measured inversions, out of scope for the hub boundary: `cogt → core`, `system → cogt`, `plugins → runtime_bridge`, and `cogt/model_backends/model_lists.py` importing `pipelex.cli.exceptions.PipelexCLIError`. What remains of `tools → cogt` is `tools/pdf/pypdfium2_renderer.py` reaching for `cogt.extract` and `cogt.image` types; the templating half of that cluster is gone (see [Placement, not coupling](#placement-not-coupling)).

## For consumers outside this repo

`pipelex.hub` is gone; there is no shim. Every external importer must choose a layer. The symbols that crossed the repo boundary are split as follows — kernel-layer ones now come from `pipelex.runtime_hub`, interpreter-layer ones from `pipelex.interpreter_hub`, per the tables above. `get_pipelex_hub` splits into `get_runtime_hub` and `get_interpreter_hub`, so each call site must pick the container it actually meant.


## Registration Surface

# Registration surface

A pipe kind is not one class. Adding `PipeFoo` means touching a handful of separate places, none of which the others infer — there is no plugin registry for pipe kinds and no scan of the tree. This page is the checklist, and the explanation of which of those places are genuinely load-bearing versus which look like duplication and are not.

## What a pipe kind is made of

Each kind lives in its own package under `pipelex/pipe_operators/<kind>/` or `pipelex/pipe_controllers/<kind>/`, and holds three classes:

| file | class | role |
| --- | --- | --- |
| `pipe_foo_blueprint.py` | `PipeFooBlueprint` | the MTHDS language surface — what a `.mthds` file parses into |
| `pipe_foo.py` | `PipeFoo` | the runnable pipe |
| `pipe_foo_factory.py` | `PipeFooFactory` | blueprint → pipe |

`PipeSignature` follows the same shape from `pipelex/pipe_signature/`, with the caveat noted below.

## The checklist

1. **The kind's package** — the three classes above, plus an `exceptions.py` if the kind raises its own errors (see the error-class location convention in the Python standards).

2. **The type tag** — add the value to `PipeType` in `pipelex/pipe_machinery/pipe_blueprint.py`, and give it an arm in the `category` property that maps it to `PipeCategory.PIPE_OPERATOR` or `PipeCategory.PIPE_CONTROLLER`. The match is exhaustive with no `case _`, so the type checker walks you to every remaining site — that is the point of writing it that way.

3. **The blueprint union** — add `PipeFooBlueprint` to `PipeBlueprintUnion` in `pipelex/mthds_parsing/pipelex_bundle_blueprint.py`. It is a pydantic discriminated union keyed on the `type` field, so a kind absent from the union is a parse error on any `.mthds` file that names it, not a silent skip.

4. **The registration manifest** — add `PipeFoo` and `PipeFooFactory` to `PipeRegistryModels` in `pipelex/pipe_machinery/registry_models.py`, in `PIPE_OPERATORS` / `PIPE_OPERATORS_FACTORY` or `PIPE_CONTROLLERS` / `PIPE_CONTROLLERS_FACTORY`. This is the one whose omission is **silent**: boot succeeds either way, and the failure surfaces later as a kajson deserialization error on a pipe. `tests/unit/pipelex/test_registry_models_split.py` pins the manifests against the class registry for exactly that reason.

5. **The spec layer** — add `PipeFooSpec` under `pipelex/builder/pipe/`, then register it in **both** `pipe_spec_map.py` (`pipe_type_to_spec_class`, the authoring lookup) and `pipe_spec_union.py` (`PipeSpecUnion`, the discriminated union). The two are separate on purpose: the union carries `PipeSignatureSpec`, the map does not, because a signature is not a user-selectable type.

## Why `core/` no longer holds the manifest

`PipeRegistryModels` used to live in `pipelex/core/registry_models.py` alongside core's value model. Importing it loads every pipe operator, every pipe controller and their factories — so a module filed under `core/` pulled in the whole interpreter, and `core` read as a dependant of the pipe packages it is supposed to sit below. The manifest now lives in `pipelex/pipe_machinery/`; `core.registry_models` keeps `CoreRegistryModels` (the stuff-content classes) and imports nothing from `pipe_operators` / `pipe_controllers` / `pipe_signature`.

The two manifests land in one registry from the two halves of the boot: `CoreRegistryModels` is registered by `RuntimeBoot.setup` in `pipelex/runtime_boot.py`, `PipeRegistryModels` by `Pipelex.setup` in `pipelex/pipelex.py` — see [Where the boot splits](hub-layering.md#where-the-boot-splits). They must stay **disjoint** — a class in both is a sign one half was edited without the other — and because registration is name-keyed dict insertion, which side registers first carries no meaning.

See [Hub Layering](hub-layering.md) for the layer boundary this split serves.

## The spec/blueprint parallel is deliberate

Steps 3 and 5 look like the same list written twice. They are not, and collapsing them would be a mistake:

- **Blueprints are the MTHDS language.** `PipeFooBlueprint` defines what an author may write in a `.mthds` file. It is the reference for the standard.
- **Specs are an authoring convenience for AI agents.** `PipeFooSpec` is a simplified shape with convenience fields that `to_blueprint()` transforms into blueprint fields. Spec-level fields deliberately differ from blueprint-level ones — `PipeComposeSpec.target_format` is a spec convenience with no blueprint counterpart, for instance.

The two layers evolve for different reasons and answer to different consumers, so they carry their own unions. When adding validation or a field, decide which layer owns it first: language rules go on blueprints, authoring convenience goes on specs. `pipelex/builder/CLAUDE.md` is the canonical statement of that split.


## Drift Contracts

# Drift Contracts

Drift contracts turn the "keep docs in sync with code" policy from advisory prose into a deterministic, mechanically-checked obligation. A **drift contract** declares: *when these trigger files change, these review targets must be re-examined against the change, and the examination must be recorded.* The tool cannot judge whether a doc is still correct — that takes a human or an agent — but it can prove, deterministically, that someone looked at this exact content and said so on the record.

## The three synchronization tiers

Not every code↔doc relationship deserves a review contract. This repo distinguishes three tiers, and a drift contract is the right tool only for the last one:

| Tier | Mechanism | Judgment needed |
|---|---|---|
| **Derived** | Regenerate and diff: `check-config-sync`, `check-mthds-schema`, `check-gateway-models`, `generate-error-pages`, `generate-error-identity` | None — the output is a pure function of its sources |
| **Linkage** | Referential integrity of declared cross-references (the spec ↔ verifying-test back-link pair) | None — links either resolve or they don't |
| **Review** | Drift contracts: "this changed → that must be looked at, and the look recorded" | Yes — someone must judge whether the prose still holds |

The standing rule: **anything mechanizable becomes a derived check.** If a review contract keeps opening and the review is always the same mechanical comparison, delete the contract and write a derived check instead — it needs no judgment and can't be rubber-stamped. The best review contract is one you eventually mechanize out of existence. Corollary: generated artifacts (`docs/errors/`, `derived/`, `tests/data/errors/error_identity.txt`, gateway model docs) must never appear in a contract's `review` list — reviewing generator *output* is how you end up editing generated files.

## How it works

Three pieces:

- **`drift.toml`** (repo root, human-authored) declares the contracts: `triggers` (globs over tracked files), optional `exclude`, `review` targets, and optional `verify_commands`.
- **`.drift/acks/<contract-id>.toml`** (tool-written, committed) records the last fulfilled review: a content digest, the reviewer, a timestamp, a rationale, and the per-file trigger snapshot.
- **`pipelex-dev drift plan|check|ack`** computes obligations, checks them, and records acks. Make wrappers: `make drift-plan`, `make drift-check` (in the `make agent-check` and `make check` aggregates; in CI it currently runs as an **advisory** job that is visible on the PR but does not block merges — it will be promoted to a required check), `make drift-ack CONTRACT=… RATIONALE="…"`.

The validity rule is a single equality, with no base ref, no git diff, and no timestamps:

> A contract is **fulfilled** iff the stored ack digest equals the digest recomputed from the current tree. Otherwise it is **open**.

The digest covers both the contract's own definition and the `(path, blob OID)` pairs of every tracked file matching its triggers — so editing a trigger file, adding or deleting a matched file, *or editing the contract itself* all open the contract. A brand-new contract has no ack, so it is open until its targets are reviewed once and acked: adoption is explicit, contract by contract.

## Where the gate acts

One view of every surface the contract system touches, from the working tree to the merge:

| Surface | What happens there |
|---|---|
| **Make targets** | `make drift-check` (alias `dc`) is the pass/fail gate; `make drift-plan` (`dp`) is the read-only diagnosis; `make drift-ack CONTRACT=… RATIONALE="…"` (`da`, optional `BY=` for agents) records the review. All three wrap `pipelex-dev drift`. |
| **Quality checks** | `drift-check` runs inside the `make agent-check` and `make check` aggregates, alongside the other repo gates — a local check cannot pass with an open contract or a rotten manifest, so an open contract surfaces before CI. Mind the index: the digest reads staged content, so unstaged trigger edits are invisible to the gate until `git add`. |
| **Commits** | The digest is computed from the **git index**, and `drift ack` stages the ack file it writes — so the ack, the code change, and any doc fix land in the same commit, reviewed as one diff. |
| **Branches / merges** | After a merge, `drift check` recomputes the digest over the merged tree. An ack that was valid on either side but does not cover the merged trigger content fails the check until the merged state is reviewed and re-acked (see [Merges](#merges)). |
| **PRs / CI** | The `lint-drift` CI job runs `make drift-check` on every PR — currently advisory (visible, not merge-blocking; promotion to a required check is planned). The committed ack file appears in the PR diff, so the reviewer sees the rationale next to the change it covers. |

The gate acts on *content*, never on *time*: there is no base ref, no diff against a branch, no timestamp freshness — only the digest equality below.

## The ack workflow

When `make drift-check` fails (locally or in CI), run:

```bash
make drift-plan
```

It prints one Markdown packet per open contract: the description, exactly which trigger files were added/removed/modified since the last ack, the review targets, the verify commands, and the previous reviewer's rationale. To fulfill a contract:

1. **Review the targets against the trigger changes.** Actually read them. Update whatever is stale. "Nothing to update" is a legitimate outcome — say so in the rationale.
2. **Stage the trigger files:** `git add` them (staging is enough; you don't need to commit first, and other unstaged changes elsewhere are fine). **The digest is computed from the git index, not the working tree** — an unstaged edit or an untracked file is invisible to the ack, and `drift ack` warns when it sees one that matches the triggers. For a contract with `verify_commands` that warning escalates to a hard error: the verify commands run on the working tree, so a matching unstaged edit or untracked file means they certify content the digest does not cover. This coverage check runs both **before** the verify commands (so an already-dirty tree fails fast) and **after** them (so a trigger the verify commands themselves format or generate is caught too, not silently left out of the digest).
3. **Record the ack:**

    ```bash
    make drift-ack CONTRACT=config-docs RATIONALE="Documented the new activity_queues setting; other config pages unaffected."
    ```

    This first runs the contract's `verify_commands` (each one `shlex`-split and run without a shell, from the repo root; the first failure aborts the ack), then recomputes the digest from the index, writes the ack file, and stages it — the ack lands in the same index the check reads, so the local gate and the commit you are building agree. If staging the written ack fails (e.g. a locked git index), the ack file is removed rather than left unstaged, so a later `drift check` reports a missing ack instead of a false green.

4. **Commit the ack file together with the change it covers** (it is already staged). The ack surfaces in the PR diff, so the reviewer sees the rationale next to the change — that is the audit trail.

`reviewed_by` defaults from `git config user.name`; agents pass `BY=<identity>` (e.g. a model name or session URL). The rationale is required and is the on-the-record review decision — write an honest sentence, not "docs fine".

## Why there is no bypass

`drift ack` has no `--skip-verify`, and `drift check` has no override label. Every escape hatch here is a rubber-stamp invitation, and the legitimate escape already exists: acking with a rationale that says "no doc change needed" is cheap, honest, and auditable. If a contract opens so often that acking feels like ceremony, the contract is mis-scoped — narrow its triggers or mechanize it into a derived check; don't add a bypass.

## Merges

Two branches that each re-acked the same contract can produce a literal merge conflict in its ack file — but don't rely on that: a line-wise auto-merge can just as well splice the two acks into one that neither branch reviewed. The guarantee to rely on is different: **after the merge, `drift check` recomputes the digest over the merged tree**, and a spliced or stale ack simply fails the check, because the merged trigger content was never reviewed as a whole. Resolution is the same either way: finish the merge, review, then re-ack on the merged tree.

## Manifest hygiene

`drift check` hard-fails on manifest rot, not just open contracts: a trigger glob or review target matching zero tracked files, triggers fully shadowed by excludes, duplicate or malformed contract ids, an ack file with no matching contract, or an ack whose `contract` field disagrees with its filename. A rename that leaves a glob dead is caught on the same PR that should fix it — edit `drift.toml` first, then re-ack.

Keep the manifest small. Contracts are added one at a time, where staleness has actually hurt; growth should follow pain, not tooling enthusiasm.


## Changelog

--8<-- "CHANGELOG.md"