Error Model

In Pipelex, an error is data, not a control-flow accident. Every failure is classified once — at the layer that knows the most about it — and that classification travels intact to every consumer: the human reading a Rich panel, the agent parsing JSON, a distributed worker's retry engine, and the HTTP adapter picking a status code.

This page covers the contract that makes that possible: the ErrorReport schema, the classification enums, how inference workers classify SDK exceptions, how classification survives every wrapping layer, and how it survives serialization across a distributed worker boundary.

Design Principle

Three rules hold across the codebase, and everything else builds on them.

Single-rooted hierarchy. Every custom exception inherits from PipelexError (pipelex/base_exceptions.py). There is one root, so one to_error_report() contract covers the whole tree.

Classify at the source, never lose it. The layer that catches a third-party exception knows the most about it. It classifies there. Every layer above is a wrapper — it adds context (pipe code, stack) but inherits the classification rather than re-deriving or discarding it.

No broad catches in business logic. except Exception is allowed only at CLI entry points and async task roots. Ruff rule BLE001 enforces this — an unexpected exception crashes loudly instead of being silently swallowed.

The Layer Model

An error rises through a series of layers. Each layer has exactly one job.

Classification happens once, at Layer 1. Layers 2–5 are wrappers: they attach context as they catch and re-raise, but the error_category, error_domain, model, and provider set at Layer 1 reach Layer 5 unchanged (see Cause-Chain Enrichment).

ErrorReport — the Serialization Schema

ErrorReport (pipelex/base_exceptions.py) is the single source of truth for error serialization. It is a frozen Pydantic model with extra="forbid".

PipelexError.to_error_report() is the entry point. to_dict() serializes, dropping None fields; from_dict() is its strict inverse.

The identity triple, and why renaming an error class is a wire break

error_type, title and type_uri are the three identity fields on every report. title and type_uri are presentation, and each has a declaration hatch — set _declared_title or _declared_type_uri directly in a subclass body and that value is used verbatim instead of the auto-derived one (inheritance is deliberately bypassed via cls.__dict__, so a parent's curated title never captures its subclasses).

error_type has no such hatch: it is type(self).__name__, the Python class name with no indirection. That makes it the machine contract — consumers outside this repo switch on that string. Renaming an error class therefore breaks them silently: their build stays green and the branch simply stops matching, falling through to a generic error path.

The guard against that is a committed snapshot of the full (error_type, title, type_uri) set at tests/data/errors/error_identity.txt, regenerated with make generate-error-identity (alias make gei) and gated by tests/unit/pipelex/errors/test_error_identity_snapshot.py. A rename cannot land without producing a reviewable one-line-pair diff on that file at the moment it is made — which is also the moment to plan the matching consumer updates.

validation_errors — structured bundle-validation diagnostics

A bundle-validation failure (ValidateBundleError) aggregates per-error data across stages and projects it onto validation_errors as a list of typed ValidationErrorItems, so the structured error report an HTTP API surfaces carries machine-mappable diagnostics (not just a single detail string). Each item's category is one of the closed ValidationErrorCategory set:

• blueprint_validation — interpreter / blueprint-validation faults. A blueprint-stage PipeValidationError raised inside a pydantic model validator (e.g. the PipeBatch input_item_name == input_list_name collision, or the SubPipe batch_over == batch_as collision — both batch_item_name_collision) is wrapped by pydantic as a value_error; the blueprint categorizer unwraps it (ctx["error"]) so the item keeps its structured error_type and pipe_code / domain_code locators instead of degrading to the no-error_type residual. The item stays in blueprint_validation (not pipe_validation) because the fault genuinely surfaced at the parse boundary, before any pipe was instantiated — only the error_type is recovered, not the stage. This category also serves as the last-resort residual: a parse-level failure (a TOML-syntax error, an empty blueprint, a bundle-elaborator failure) is raised with only a message and no categorized data, so when nothing else produced an item the builder projects that message as one blueprint_validation item (no error_type, no source — the bundle could not become a blueprint at all).
• pipe_factory — pipe-factory failures (e.g. a missing concept).
• pipe_validation — pipe/concept validation (missing input variable, type mismatch).
• dry_run — the residual dry-run failure (DryRunError / PipeRunError) with no structured locator. It is projected as one message-only item only when no categorized error has data. It is graph-level, so it typically carries no source.

Together the two residuals make the structured-info invariant total: every invalid verdict carries a non-empty validation_errors[], never a bare message. The builder tries the channels in order — categorized data, then the dry_run residual (the more specific channel), then the blueprint_validation fallback — and emits exactly one residual only when no earlier channel produced an item.

Besides category and message, each item carries whatever identity fields its stage produced — error_type, pipe_code, concept_code, domain_code, field_path, field_name, variable_names, missing_concept_code, declared_concepts, and a source (the declaring file path, or the per-content source the in-memory load path was given) that hands a consumer the owning file for cross-file diagnostic placement. When the error has a deterministic remedy, the item also carries a suggested_fix.

Signatures are never an error. An unimplemented PipeSignature reached during validation is a runnability fact, not a validation failure: the validator no longer raises on it. The assembled library's outstanding signatures ride the validation report's pending_signatures, and is_runnable = not pending_signatures. allow_signatures is a sweep-mechanics flag only (whether signature pipes are mock-run and listed in validated_pipes) — it does not change the verdict, so strict ≡ lenient in the report body. The "is this a failure?" decision moves to the consumer: the CLI exits non-zero on not is_runnable unless --allow-signatures; the HTTP caller reads is_runnable. (The execute/run path is different: running a stub still raises PipeSignatureNotExecutableError.)

Host-wiring guards are programmer errors, not content verdicts. validate_bundle's "provide exactly one of mthds_contents / mthds_file_path" guard and resolve_crate_from_contents's mthds_sources-length-mismatch guard raise PipelexUnexpectedError (→ 500, redacted under STRICT), not ValidateBundleError — a caller wiring bug must not be reported as if the submitted bundle were invalid. The empty-mthds_contents guard stays caller-facing (it can legitimately reflect an end user submitting no bundles).

ValidationErrorItem and the builder are the single source of truth across surfaces: build_validation_error_items() (pipelex/pipeline/validation_errors.py) is called by both ValidateBundleError.to_error_report() (the API path) and the agent CLI's extract_validation_errors() (the CLI JSON envelope), so the two structured shapes cannot drift. The item lives in pipelex/base_exceptions.py alongside ErrorReport — not next to the source error-data models — because ErrorReport references it as a typed field and the root exceptions module must not import the pipelex.core error modules.

The error_type registry — the closed vocabulary of faults

An item's error_type names the fault it reports, and that vocabulary is closed: pipelex/validation_error_types.py holds it in full, enumerated as VALIDATION_ERROR_TYPES. A consumer that needs to know which faults the language surface can report — a coverage gate, a test corpus, a client mapping errors onto its own UI — reads that registry instead of collecting strings from whichever diagnostics it happens to have seen.

The registry is the union of the enums the runtime already raises, never a second list beside them: PipeValidationErrorType and PipeFactoryErrorType are the two stage vocabularies, and ValidationResidualErrorType names the one residual channel with no stage enum of its own. A member added to any of the three is in the registry the moment it is declared. ValidationErrorItem.error_type is typed against their union, so an unregistered string cannot be constructed or parsed onto an item — which is what makes the enumeration closed rather than merely documented, and what publishes the vocabulary into the OpenAPI schema pipelex-api serves for /validate.

Two spellings live in that one vocabulary, deliberately. The stage enums are snake_case codes (missing_input_variable); the dry-run residual is DryRunError, the name of the exception that produced it, because that residual is raised as an error object rather than classified into a code. Normalizing it would be a wire break across every consumer that pins the string, and it would buy nothing — the enumeration is closed either way.

Membership means a value is reachable on the wire, not that it is a useful thing to exercise. optional_force_redundant rides warnings and never an invalid verdict, and the two unknown_* fallbacks fire on states no author can ask for. A consumer building coverage over the registry excludes those on its own side with a stated reason, rather than pruning them from the runtime truth here — that is what the MTHDS Test Corpus vocabulary generator will do when its error.* namespace arrives with this registry.

validation_errors is one of the fields kept under STRICT disclosure (it is in _STRICT_KEPT_FIELDS): the items describe the caller's own submitted bundle, not server internals, so redacting them would gut the hosted path's diagnostics.

report = exc.to_error_report()
report.to_dict()         # {"error_type": "LLMCompletionError", "message": "...", ...}
ErrorReport.from_dict(d) # strict inverse — raises ValidationError on a malformed dict
report.http_status       # 422 / 429 / 500 — for HTTP adapters

suggested_fix — structured deterministic fixes

When a validation error has a deterministic remedy, its ValidationErrorItem carries a suggested_fix — a SuggestedFix (pipelex/suggested_fix.py, deliberately stdlib+pydantic-only so pipelex.base_exceptions can import it without a cycle; naming is brand-neutral, fixes are a language-level concept):

• fix_code — the kebab-case rule id (e.g. match-sequence-output). The planner's KNOWN_FIX_CODES set is the validation set for user-facing rule filters (--select / --ignore); an unknown code is rejected loudly, never lenient-ignored, because a typo'd filter selects behavior.
• description — human-readable statement of the change.
• safety — safe fixes may be auto-applied; unsafe ones require explicit opt-in.
• source — the file the ops target, when known (multi-file libraries). An applier must only apply ops to the file they target.

• ops[] — the fix itself, as semantic TOML patch ops addressed by table path (FixOpKind: set_key, ensure_table, delete_key, delete_table, rename_table_key, move_key, remap_value; each op's table_path follows the same conventions as the items' field_path). The ops are the machine contract; any rendered diff or 💡 Suggested fix: line is presentation.

  The op vocabulary is a discriminated union on kind: one model per kind, each declaring exactly the fields its own semantics need and forbidding the rest, so {"kind": "delete_key", …, "new_key": "x"} is a parse error rather than a stray field the applier silently ignores. Two aliases are published from the same union — FixOp, every kind, which is what ops[] is typed as, and MigrationOp, the structural kinds only (delete_key, delete_table, rename_table_key, move_key, remap_value), which is what a configuration migration ledger is parsed against. The narrow alias is what keeps a materializing op — one that writes a value the file did not have — out of a ledger that is replayed over every user file on every run.

The fix planner (pipelex/pipeline/fixes/planner.py) translates enriched typed error data into SuggestedFix payloads — pure functions keyed strictly on error_type + structured fields, never on message strings. Each rule fires only when its enrichment is present (set only at the raise sites that know the correct value), so the same error type raised elsewhere without enrichment is structurally suppressed. The planner runs inside build_validation_error_items(), so every consumer of the validation report — CLI, API, MCP — sees fixes with zero extra plumbing.

Applying fixes is the runtime's job too: the applier (pipelex/pipeline/fixes/applier.py) mutates a tomlkit DOM in place per op (guarded — an op whose target table is absent is skipped and reported, never raised) and then reflows the whole file to canonical MTHDS style, and the convergence loop (pipelex/pipeline/fixes/fix_loop.py) runs validate → apply SAFE fixes → re-validate to a fixed point, reporting non-convergence loudly. The user-facing surface is pipelex fix bundle.

On the hosted API the same payload rides the wire verbatim as validation_errors[].suggested_fix; how it appears in HTTP error responses is documented on the API side, in pipelex-api's docs/error-responses.md → "Suggested fixes".

Classification Enums

Two StrEnums drive every downstream decision.

InferenceErrorCategory

Defined in pipelex/cogt/exceptions.py. Drives retry decisions — is_retryable is True only for TRANSIENT.

class InferenceErrorCategory(StrEnum):
    TRANSIENT = "transient"
    # ... CONFIGURATION, CONTENT, CAPACITY, AMBIGUOUS ...
    UNKNOWN = "unknown"

    @property
    def is_retryable(self) -> bool:
        match self:
            case InferenceErrorCategory.TRANSIENT:
                return True
            case _:  # all other categories
                return False

ErrorDomain

Defined in pipelex/base_exceptions.py. Set as a class-level attribute on the exception, drives HTTP status.

error_domain_to_http_status() is the pure mapping table. ErrorReport.http_status layers one rule on top: a provider 429 (provider_metadata.status_code == 429) takes precedence over the domain, so the API can emit a Retry-After header.

class PipelexConfigError(PipelexError):
    error_domain = ErrorDomain.CONFIG     # class-level — every instance carries it

Worker Classification

Layer 0 → Layer 1. Every inference worker under pipelex/providers/*/ catches its SDK's typed exceptions and re-raises a categorized CogtError.

The Uniform Shape — Extract / Classify / Render

Every inference worker's SDK-exception handler collapses to a three-step pipeline: Extract turns the SDK exception into a provider-blind ProviderErrorMetadata, Classify maps that metadata to a category + user-action, and Render picks the CogtError subclass to raise.

except (APIError, APIConnectionError, APITimeoutError) as exc:
    metadata = extract_openai_metadata(exc)
    classification = classify_inference_error(metadata)
    raise render_inference_error(
        metadata=metadata,
        classification=classification,
        family=InferenceErrorFamily.LLM,
        model_desc=self.inference_model.desc,
        model_handle=self.inference_model.name,
    ) from exc

The three steps live in three modules. Only the per-provider Extract functions stay plugin-local; Classify and Render are single shared functions.

Provider-specific nuance is normalized away in Extract (e.g. Google's code becomes status_code; AWS Bedrock error codes are mapped to HTTP statuses), so Classify has no provider branching. HTTP status drives classification; status-less errors dispatch on the SDK exception type name. The tests/unit/pipelex/cogt/inference/test_provider_classification_parity.py meta-test walks every ProviderName against the extract-fn registry so adding a new provider without wiring it fails fast.

ProviderErrorMetadata and UserAction

Every raised inference error carries structured SDK metadata and typed advice.

class ProviderErrorMetadata(BaseModel):
    provider: str
    sdk_exception_type: str
    status_code: int | None = None
    request_id: str | None = None
    retry_after_seconds: float | None = None
    provider_error_code: str | None = None
    body: Any | None = Field(default=None, exclude=True)   # may carry secrets

UserAction pairs a discrete UserActionKind (WAIT_AND_RETRY, CHECK_BILLING, CHECK_CREDENTIALS, CHANGE_INPUT, CHANGE_MODEL, CONTACT_SUPPORT, UNKNOWN) with a free-form detail string — so the CLI can render consistent guidance while keeping provider-specific text.

The instructor Unwrap

On structured-generation paths, instructor wraps the real SDK exception in an InstructorRetryException. extract_underlying_sdk_exception() recovers it, so it routes through the same per-provider categorization as the plain-text path. A genuinely unrecognized inner exception (e.g. a pydantic.ValidationError from a schema mismatch) lands in UNKNOWN rather than being mis-labelled as a CONTENT-policy violation.

Model and Provider Attribution

Inference-failure leaf errors (LLMCompletionError, ImgGenGenerationError, …) are raised deep inside a plugin and do not know which model handle invoked them. Each worker family fills that in at its public-method chokepoint:

def fill_model_and_provider(self, model_handle: str | None, *, backend_name: str | None) -> None:
    """Fill model_handle / backend_name from the worker, only when still unset."""

Cause-Chain Enrichment

A wrapper exception — PipeRunError → PipeRouterError → PipelineExecutionError — carries no error_category of its own. to_error_report() enriches the report from the __cause__ chain, so the inference classification survives every wrapping layer.

def _enrich_error_report_from_cause(self, report: ErrorReport) -> ErrorReport:
    cause = self.__cause__
    if not isinstance(cause, PipelexError):
        return report
    cause_report = cause.to_error_report()
    return ErrorReport(
        error_type=report.error_type,                                  # keep own identity
        message=report.message,
        error_category=report.error_category or cause_report.error_category,
        error_domain=report.error_domain or cause_report.error_domain,
        # ... retryable, user_action, model, provider, provider_metadata ...
    )

A wrapper keeps its own error_type and message but inherits every classification field it does not set itself.

Crossing a Distributed Worker Boundary

The error model is built to survive serialization. Because ErrorReport round-trips through to_dict() / from_dict(), a failure that happens on a remote worker can reach the submitting process with its full classification intact — not just a message string.

The runtime itself stays transport-agnostic: the machinery that carries an error across a worker boundary ships in the host-runtime plugin for each distributed backend, not in core. A backend plugin is responsible for three things.

Packing. Convert a PipelexError into the transport's failure type and stash to_error_report().to_dict() in its details payload, so worker and submitter code keep the full classification rather than a bare message. The same step derives the transport's retry decision from InferenceErrorCategory.is_retryable.

Recovering. On the submitter side, walk the returned failure, pull the packed dict, and rebuild the ErrorReport. Recovery is total: when no report dict is found — a non-Pipelex exception, a worker crash, a timeout — the plugin synthesizes a fallback report so the recovery path always has structured classification to surface.

A fail-safe floor. Ensure a domain error that escapes the conversion path fails the unit of work terminally rather than hanging. In a durable-execution system the default for an unconverted exception might be to retry forever, so "convert all the errors we know about" is not enough — the floor must hold for the errors, and the code paths, that nobody enumerated.

Net effect: a pipe failing on a remote worker reaches the CLI and HTTP adapters with the same error_category / retryable / model / provider / user_action as the identical failure run locally — and a failure that escapes conversion fails loud and bounded instead of hanging.

See Runtime Bridge & Transport for the boundary these converters span; the per-backend converters themselves live in the host-runtime plugins.

Interfaces

CLI

The agent CLI (pipelex-agent) emits a structured error to stderr, markdown by default and JSON with --error-format json. When --error-format is omitted it inherits the value of --format (the success-output flag) — so --format json still flips both as it did before the split. Both exit with code 1.

The human CLI (pipelex) renders a Rich error panel — red banner, structured fields, the user_action tip, doc/Discord links — through the shared display_error_panel() helper in pipelex/cli/error_handlers.py.

Validate exit-code policy (0 / 1 / 2)

The validate surface — both the bare pipelex validate {bundle,method,pipe} group and the agent CLI's pipelex-agent validate — exits with three codes that mirror the hosted /validate 200-verdict-vs-non-2xx-no-verdict split:

The verdict lives in the structured is_valid field, not the exit code. The exit code is a convenience signal for naive shell/CI/Makefile use (set -e, cmd && next, if cmd; then); machine consumers (hooks, the Codex hook, runners) MUST read is_valid (and error_domain) from the JSON for their block/warn decisions rather than branching on the exit code. Decoupling the verdict from the exit code is what keeps any future exit-code change non-breaking. The 1-vs-2 split is also additive for flat consumers: both stay non-zero, so anything that only tests zero-vs-non-zero is unaffected.

Implementation: the agent CLI threads exit_code through agent_error(...) (agent_output.py, default 1); the validate commands pass exit_code=2 at every no-verdict site and keep the default 1 on the ValidateBundleError arm and the signature gate. The bare CLI sets the code directly via typer.Exit(...) in cli/commands/validate/* and via the exit_code parameter on handle_model_choice_error / handle_model_availability_error in cli/error_handlers.py. Shared boot handlers (make_pipelex_for_cli's gateway/inference/telemetry/model-deck-preset paths) stay exit 1 — they are shared across run/build/validate and out of the validate-policy scope.

API

pipelex is a library — there is no API server in the package. Downstream HTTP repos consume the ErrorReport:

• error_domain_to_http_status(error_domain) — pure domain → status table.
• ErrorReport.http_status — full property, layering the provider-429 passthrough on top.

A downstream FastAPI exception handler calls ErrorReport.http_status and is a trivial adapter — it must not redefine the mapping.

Inputs and Outputs

Inputs. to_error_report() takes a live PipelexError. ErrorReport.from_dict() takes a to_dict() payload — strictly, raising ValidationError on drift. (A distributed-worker bridge adds a cross-boundary recovery helper that walks a returned failure's __cause__ chain and rebuilds the report; it lives in the host-runtime plugin, not core.)

Outputs. to_error_report() returns an ErrorReport; to_dict() returns a None-free dict. Side effects: telemetry events emitted on pipeline failure at Layer 3; the agent CLI writes to stderr and raises typer.Exit(...) — code 1 by default, or the validate surface's 0/1/2 policy (see Validate exit-code policy).

Architecture

flowchart TB
    SDK["Layer 0 — SDK exception<br/>(openai.RateLimitError)"]
    W["Layer 1 — Worker classifies<br/>is_quota_exhaustion_*() → CogtError<br/>+ InferenceErrorCategory + ProviderErrorMetadata"]
    WRAP["Layers 2-3 — Wrappers<br/>PipeRouterError → PipelineExecutionError<br/>(attach pipe context)"]
    REPORT["ErrorReport<br/>via to_error_report() + cause-chain enrichment"]

    SDK -->|"raise ... from exc"| W
    W -->|"raise ... from exc"| WRAP
    WRAP --> REPORT

    REPORT --> RICH["Human CLI<br/>Rich panel"]
    REPORT --> AGENT["Agent CLI<br/>JSON / Markdown"]
    REPORT --> HTTP["HTTP adapters<br/>.http_status"]

    W -.->|"pack on worker"| TEMP["Distributed worker bridge (plugin)<br/>report packed into transport details"]
    TEMP -.->|"recover on submitter"| REPORT

    classDef src fill:#fff3e0,stroke:#e65100,color:#000
    classDef cls fill:#e8eaf6,stroke:#3949ab,color:#000
    classDef out fill:#e8f5e9,stroke:#2e7d32,color:#000
    class SDK src
    class W,WRAP,REPORT,TEMP cls
    class RICH,AGENT,HTTP out

Implementation

Class Hierarchy

PipelexError is the single root. CogtError is the inference branch — it overrides to_error_report() to add error_category, retryable, user_action, provider_metadata, and reads model_handle / backend_name from the instance.

Exception
└── PipelexError                  base_exceptions.py — error_domain, user_action, to_error_report()
    ├── PipelexConfigError         → error_domain = CONFIG
    ├── PipelexSetupError          → error_domain = CONFIG
    ├── CogtError                  cogt/exceptions.py — error_category, provider_metadata
    │   ├── LLMCompletionError      ← per-instance category from the worker
    │   ├── ImgGenGenerationError   ← per-instance category
    │   ├── ModelNotFoundError      ← sibling family raised on provider HTTP 404
    │   │   ├── LLMModelNotFoundError / ImgGenModelNotFoundError
    │   │   └── ExtractModelNotFoundError / SearchModelNotFoundError
    │   └── ... (see worker classification) ...
    ├── PipelineExecutionError      pipeline/exceptions.py — error_domain = RUNTIME
    └── ... (one exceptions.py per package) ...

Factory-time vs Runtime

The "outcome" exceptions (LLMCompletionError, ImgGenGenerationError, ExtractJobFailureError, SearchJobFailureError) intentionally carry no class-level error_category — their category is genuinely per-instance, decided by the worker.

Reference

Quick-Ref

# Produce a report from any PipelexError
report = exc.to_error_report()          # enriched from the __cause__ chain
payload = report.to_dict()              # None-free dict for serialization

# Consume a report
report.http_status                      # 422 / 429 / 500
report.user_action_detail()             # free-form advice text, or None
report.error_category                   # "transient" / "capacity" / ...

# Round-trip across a boundary
ErrorReport.from_dict(payload)           # strict inverse of to_dict()

# Retry decision
InferenceErrorCategory.TRANSIENT.is_retryable   # True — only TRANSIENT

File → Purpose

Behavior Summary

Next Steps

• Pipe Routing & Execution — the layer model errors rise through
• Runtime Bridge & Transport — the process boundary the error bridge spans (the per-backend error converters live in the host-runtime plugins)
• Inference Configuration — transport_max_retries and the Tier 1 retry policy
• Agent CLI — the JSON / markdown error contract