Inference Backend Plugins

Every model call in Pipelex — an LLM completion, an image generation, a document extraction, a web search, a judgment — is served by an inference worker. Which worker handles a given model is decided entirely by data: a model's sdk field selects a backend, and a backend plugin is what teaches Pipelex how to build the worker for that sdk.

Core names no backend by import or by string. The built-in drivers (OpenAI, Anthropic, Mistral, Bedrock, Google, FAL, HuggingFace, OpenRouter, Portkey, Docling, …) are plugins too — they ride the exact same seam an out-of-tree plugin would. This page documents that seam, the Inference SPI a plugin compiles against, and how to write one.

The seam in one view

boot (Pipelex.setup)
  └─ build_registrar(config, builtin_plugins=BUILTIN_PLUGINS, …)   # pure, import-light
       ├─ for each built-in plugin (both layers' halves, composed)
       └─ for each installed entry point in the requested groups
            #  "pipelex.plugins.kernel" alone on a kernel-only boot; both groups on a full one
            └─ plugin.register(registrar)        # side-effect-free
                 └─ registrar.add_inference_backend(family=…, sdk=…, make_worker=…)
  └─ InferenceBackendRegistry(registrar.inference_backends)   # stored on the hub

run time (e.g. LLMWorkerFactory.make_llm_worker)
  └─ model_handle = ModelHandle.make_for_inference_model(...)
  └─ make_worker = get_inference_backend_registry().lookup(family=LLM, sdk=model_handle.sdk)
  └─ worker = make_worker(inference_model=…, backend=…, sdk_clients=…, reporting_delegate=…)

The worker factories (LLMWorkerFactory, ImgGenWorkerFactory, ExtractWorkerFactory, SearchWorkerFactory, DocGenWorkerFactory, JudgmentWorkerFactory) hold no match over SDK strings. They build a ModelHandle, resolve the InferenceBackend config, look up the backend's make_worker by (family, sdk), and call it. A lookup miss raises a friendly InferenceBackendNotFoundError ("… Is its plugin installed and enabled?").

The contract: PipelexPlugin

A plugin is any object satisfying the @runtime_checkable PipelexPlugin protocol:

class PipelexPlugin(Protocol):
    name: str  # unique, lowercase identifier (e.g. "openai")
    targets_api: int  # must equal PLUGIN_API_VERSION

    def register(self, registrar: PluginRegistrar) -> None: ...

register is the only method core calls, and it is side-effect-free: it may call the registrar's menu methods and nothing else — no hub access, no I/O, no client/SDK construction. This is what makes build_registrar safe to run more than once (it runs at boot, and again whenever the pipelex plugins list diagnostic command re-discovers plugins to print what each contributed).

targets_api is checked against PLUGIN_API_VERSION. A mismatch fails loud with PluginApiVersionMismatchError — a single coarse integer gate, not semver-range matching.

Registering an inference backend

A backend plugin's register calls one menu method per (family, sdk) it serves:

registrar.add_inference_backend(
    family=InferenceFamily.LLM,  # LLM | IMG_GEN | EXTRACT | SEARCH | DOC_GEN | JUDGMENT
    sdk="acme",  # the model's `sdk` string
    make_worker=_make_acme_worker,  # a MakeWorkerFn (a plain callable)
    check_llm_request=_check_acme_request,  # optional, LLM family only: a CheckLLMRequestFn
)

check_llm_request is how validation learns what the worker refuses. Every LLM worker runs a check_request classmethod before each provider call, which refuses, from the model's spec and the job params alone, a request it would refuse anyway: a reasoning setting the model's thinking_mode refuses, a reasoning budget where it takes only an effort, a thinking budget the call's max_tokens cannot hold, or a structured output it cannot generate. When a method loads, every PipeLLM and PipeStructure resolves its model setting to the model the deck serves and calls the check its backend registered, so such a step is refused as an llm_setting_refused_by_model item before a run spends anything. The item is shown to whoever wrote the setting, in the caller view too, so the refusal's text must name nothing your plugin keeps private: raise an LLMSettingRefusedError(msg) (pipelex.cogt.exceptions), worded for whoever wrote the setting, to have it shown as written, and any other refusal your check raises is named on the item by its code's sentence alone, its text riding only the operator's diagnostics. A built-in backend's refusal is shown with the model named by its deck handle. The registered check is import-light like make_worker, called with inference_model, job_params and is_structured as keyword arguments, and delegates to the worker class's own check, which builds no SDK client:

def _check_acme_request(*, inference_model: InferenceModelSpec, job_params: LLMJobParams, is_structured: bool) -> None:
    from acme_pipelex.acme_worker import AcmeLLMWorker

    AcmeLLMWorker.check_request(inference_model=inference_model, job_params=job_params, is_structured=is_structured)

An LLM backend that registers no check is held to LLMWorkerAbstract.check_request, the rule every worker shares: a model whose spec declares thinking_mode = "none" takes no reasoning setting. A worker that refuses more overrides check_request from the same helpers its calls resolve the request with, so validation and the call never disagree. A check raising MissingDependencyError, because the backend's SDK is not installed, leaves the step to the run. A plugin that builds one of the built-in workers for its own sdk registers a check calling that worker's check_request, as the built-in plugins do.

A registry key is (family, sdk). The same sdk string may appear in two families (e.g. google serves both LLM and IMG_GEN); they are distinct keys. A duplicate (family, sdk) fails loud with DuplicateInferenceBackendError naming both contributing plugins.

One plugin may register across several families from a single register — openai and google serve LLM + IMG_GEN, mistral serves LLM + EXTRACT, linkup serves EXTRACT + SEARCH, and typesafe serves JUDGMENT alone. This is the cross-family-vendor coordination point: one plugin, many backends. DOC_GEN is the family of the document engines a PipeDocGen step prints with, local libraries rather than inference, which the built-in reportlab plugin and the Pipelex document generation plugin serve (see Document Engine Plugins).

A plugin whose backend runs without an external service, such as a document engine, may also declare the models it serves rather than leave them to a backend file. add_internal_model(name=…, spec=…) declares a model in the internal backend, the spec being the table a backend file would hold for it, and add_doc_gen_default(doc_gen_format=…, source=…, model=…) declares the model deck's default engine for a document format and source. Both are stored at registration and merged by the model manager at boot, and Document Engine Plugins gives their rules.

MakeWorkerFn — the locked call shape

A backend is a typed callable, not a one-method object. Its signature is the contract:

def _make_acme_worker(
    *,
    inference_model: InferenceModelSpec,
    backend: InferenceBackend,
    sdk_clients: SdkClientRegistry,
    reporting_delegate: ReportingProtocol | None,
) -> InferenceWorkerAbstract: ...

The factory always passes all four keyword arguments. A stateless backend simply ignores the ones it doesn't need (# noqa: ARG001 on the unused parameter — see the built-in pypdfium2 and azure_rest plugins).

Two invariants shape what goes inside the closure:

• Import-light. The plugin module must import no backend SDK at module load. Do the SDK import inside the closure (# noqa: PLC0415), so merely discovering the plugin never pulls a heavy/optional dependency. Booting Pipelex with the built-ins registered imports none of anthropic, mistralai, google.genai, boto3, fal_client, huggingface_hub, docling, linkup, typesafe_sdk, … — enforced by a subprocess import-blocker guard.
• Fail at use, not at boot. Guard an optional dependency with require_sdk(...) inside the closure. A missing extra then raises MissingDependencyError (naming the package and the pipelex[<extra>] install hint) only when the backend is actually used.

Client memoization goes through sdk_clients.get_or_create(handle=…, build=lambda: …) — the registry caches one SDK client per ModelHandle, so repeated worker construction reuses the connection.

Listing models — an optional capability

Alongside its worker factory, a backend plugin may register a model lister — the callable behind pipelex show models <backend>. It is optional: a backend that cannot enumerate its models simply never calls add_model_lister, and the listing command reports that SDK as unsupported-for-listing. The contract grows by one optional method, so progressive disclosure is preserved.

registrar.add_model_lister(sdk="acme", lister=_list_acme_models)

A ListModelsFn mirrors MakeWorkerFn — import-light to reference, lazy inside. It is always async (the listing loop awaits it) and is keyed by sdk alone (listing is per-SDK, not per-(family, sdk)):

async def _list_acme_models(
    *,
    sdk: str,
    backend_name: str,
    backend: InferenceBackend,
    flat: bool,
    any_listed: bool,
) -> None:
    from my_pkg.acme_list import list_acme_models  # noqa: PLC0415

    await list_acme_models(sdk=sdk, backend_name=backend_name, backend=backend, flat=flat, any_listed=any_listed)

The same import-light / fail-at-use rules apply: a missing optional extra must raise MissingDependencyError only when the backend is actually listed — use the require_sdk(...) helper, or an equivalent find_spec guard inside the function the lister delegates to (the built-in listers do the latter, reusing the guard already in their list_*_models functions). If a registered lister's client variant cannot enumerate models at runtime (e.g. a Bedrock-backed Anthropic client), raise ModelListingUnsupportedError(sdk=…): the loop treats that as the same soft "unsupported for listing" outcome as a missing lister — never a hard failure. A duplicate sdk lister fails loud with DuplicateModelListerError naming both plugins.

Authoring a backend plugin (minimal example)

Writing an Inference Plugin walks through a complete plugin you can install and run with no API key, from its entry point to a method served by its model. The example below shows what a plugin wrapping a real SDK adds to that.

A complete LLM backend plugin for a hypothetical acme SDK:

# acme_plugin.py
from pipelex.cogt.inference.inference_worker_abstract import InferenceWorkerAbstract
from pipelex.cogt.model_backends.backend import InferenceBackend
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.plugins.contract import PLUGIN_API_VERSION
from pipelex.plugins.inference_backend_registry import InferenceFamily, require_sdk
from pipelex.plugins.model_handle import ModelHandle
from pipelex.plugins.registrar import PluginRegistrar
from pipelex.plugins.sdk_client_registry import SdkClientRegistry
from pipelex.reporting.reporting_protocol import ReportingProtocol

_ACME_MISSING_MSG = "The acme SDK is required in order to use Acme models."


def _make_acme_worker(
    *,
    inference_model: InferenceModelSpec,
    backend: InferenceBackend,
    sdk_clients: SdkClientRegistry,
    reporting_delegate: ReportingProtocol | None,
) -> InferenceWorkerAbstract:
    require_sdk(spec="acme", extra="acme", msg=_ACME_MISSING_MSG)

    from acme import AcmeClient  # noqa: PLC0415

    from my_pkg.acme_llm_worker import AcmeLLMWorker  # noqa: PLC0415

    model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)
    sdk_instance = sdk_clients.get_or_create(
        handle=model_handle,
        build=lambda: AcmeClient(api_key=backend.api_key),
    )
    return AcmeLLMWorker(
        sdk_instance=sdk_instance,
        inference_model=inference_model,
        reporting_delegate=reporting_delegate,
    )


class AcmePlugin:
    name = "acme"
    targets_api = PLUGIN_API_VERSION

    def register(self, registrar: PluginRegistrar) -> None:
        registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="acme", make_worker=_make_acme_worker)

Shipping it as an out-of-tree plugin

Declare an entry point in the pipelex.plugins.kernel group; discovery finds it automatically once the distribution is installed (no central enable-list — presence is the source of truth):

# pyproject.toml of your plugin package
[project.entry-points."pipelex.plugins.kernel"]
acme = "my_pkg.acme_plugin:AcmePlugin"

The group declares your layer, and the choice is not cosmetic. An inference backend is kernel-layer — it constructs a worker, never a Pipe — so it belongs in pipelex.plugins.kernel, and a kernel-only boot reads that group and finds it. Publish the same plugin under pipelex.plugins.interpreter and a kernel-only boot never even imports your module, so your backend is simply absent. The reverse mistake fails loudly instead: a pipelex.plugins.kernel plugin whose register reaches an interpreter-layer capability — add_orchestrator, add_bundle_validator, add_pipe_func_executor, add_pipe_func_package_offer, or the three hub slots that hand back a Pipe-aware object (claim_pipe_router, claim_pipe_run, claim_pipe_func_executor) — raises PluginLayerViolationError naming the capability and the group to move to. The other slots are kernel-tier and a kernel-group plugin may claim them freely: claim_content_generator, claim_task_manager, claim_isolated_execution_probe, claim_method_file_store. The restriction is one-directional — an interpreter-group plugin may contribute kernel-tier capabilities too.

The single pre-split pipelex.plugins group is retired. It is not ignored: a plugin still published there fails startup with RetiredPluginEntryPointGroupError, because the alternative — being silently undiscovered — is the expensive failure to diagnose.

The entry point may resolve to a plugin instance or a zero-argument factory. A broken entry point is isolated and reported as BrokenPluginError, never a silent skip.

Disabling a discovered plugin

build_registrar skips (and logs) any plugin whose name appears in the core-owned runtime.plugins.disabled denylist:

# .pipelex/pipelex.toml
[runtime.plugins]
disabled = ["acme"]

For external entry points the denylist is matched against the entry-point name before the plugin is loaded, so a broken or dependency-missing installed plugin can still be disabled to recover startup (it would otherwise raise BrokenPluginError at load). A working external plugin that sets a name different from its entry-point name is denylistable by either name.

Disabling a core-unconditional plugin (openai) is a configuration error, not a no-op — it raises CoreUnconditionalPluginDisabledError. There is intentionally no allowlist: a plugin's presence (built-in or installed entry point) is what enables it.

Use pipelex plugins list to see every discovered plugin, the entry-point group it was found under, what each contributed, and its denylist state. The Group column is the first thing to read when a plugin is missing: a built-in shows —, and an external plugin that resolved to the wrong layer shows it there.

Declaring the error codes a plugin raises

Every error a plugin raises reports a public registry code, a closed plugin's included: a caller reads the code, its page and its next step whatever plugin raised it. A plugin's error class declares its code as its error_code, one code, an ErrorCodeSet whose raise sites choose one with .with_error_code(...), or an InheritsCause that takes its cause's code, and every code comes from the MTHDS core (CoreCode) or from Pipelex's registry (PipelexErrorCode). A plugin defines no code of its own. A failure a worker builds by hand with an InferenceErrorCategory and no code reports the category's own code (InferenceErrorCategory.default_error_code) when no cause carries one, so that a transient failure, which the run repeats, never reports a failure no retry clears.

The plugin also declares, in its register, every code its errors can carry:

from mthds.protocol.problem.registry import CoreCode

from pipelex.errors.registry.pipelex_codes import PipelexErrorCode


def register(self, registrar: PluginRegistrar) -> None:
    registrar.declare_error_codes(codes=(CoreCode.PROVIDER_FAILED, PipelexErrorCode.DOCGEN_PRINT_TIMEOUT, "pipelex.docgen.template_invalid"))
    # ...and the plugin's backends

A code is given as its constant or as its string. Each one is parsed at boot, and a code neither registry defines refuses the boot with PluginErrorCodeNotRegisteredError (pipelex.plugin.broken), naming the plugin and the code, before any of the plugin's codes is bound. A release of the plugin built against codes this Pipelex does not define therefore fails where its operator sees it, never on a caller's run. Several plugins may declare the same code, since a public code means one thing whoever raises it. The registrar keeps the bound codes per plugin in declared_error_codes, and the seam belongs to no layer, so a pipelex.plugins.kernel plugin declares its codes as an interpreter plugin does.

The Inference SPI

What an out-of-tree backend plugin imports is the contract. The published surface is listed below, one module per row, grouped by what a plugin does with it. The plugin contract and the worker contracts are what every backend plugin needs; the rest is what a plugin reaching an OpenAI- or Anthropic-compatible service, or a gateway with error codes of its own, builds on.

The plugin contract

The worker contracts, their jobs and their outputs

A worker implements the provider half of a call, _gen_text or _extract_pages for instance, and records the usage the provider answered with on its job's report. The base around it logs the event every call ends with, Inference call ends, with that usage, its cost, its duration and its outcome, so a worker logs nothing for the call itself (see Summary events). A provider that bills by the request or by the page rather than by the token is recorded with record_unit_priced_usage(tokens_usage=…, pricing_unit=PricingUnit.REQUEST, nb_units=1), which writes each unit as a million tokens in and out so the per-million rates price one unit, and marks the usage so the event keeps the call's cost and writes no token counts; a worker never writes those counts itself.

Building on an OpenAI- or Anthropic-compatible service

Errors: distilling, classifying and rendering a failure

The runtime a worker runs in

The SPI is a documented, versioned module/symbol list gated by PLUGIN_API_VERSION — not an __init__.py re-export shim (the repo bans re-exports; import by full path). Anything a plugin needs to import outside this surface is a design gap to resolve, not an accident to live with: either the symbol is published here, or the plugin keeps its own copy of it. Pipelex's own out-of-tree inference plugin is held to it by a test, in that plugin's repository, that scans its imports against this table.

Fail-loud guarantees

Related

• Writing an Inference Plugin — a complete plugin, installed and run end to end with no API key
• Orchestrator Plugins — the other per-call seam, riding the same discovery/denylist machinery
• Storage Provider Plugins — the config-selected-singleton seam (storage backend by runtime.storage.method)
• Secrets Provider Plugins — the config-selected-singleton seam (secrets backend by runtime.secrets.method)
• Pipe Routing & Execution — where worker construction sits in the run path
• Error Model — how these errors render and dereference
• Architecture Overview