Secrets Provider Plugins

Every credential Pipelex reads — an API key, a service-account file path, a signing secret — is resolved through a single secrets provider set on the hub at boot. Which provider that is comes entirely from data: one config field, runtime.secrets.method, names a backend, and a secrets plugin is what teaches Pipelex how to build the provider for that method.

Core names no secrets backend by import or by string. The built-in env provider (reads secrets from environment variables) is a plugin too — the always-on SecretsPlugin — riding the exact same seam an out-of-tree pipelex-secrets-<backend> package (Vault, AWS Secrets Manager, GCP Secret Manager, …) would. This page documents that seam, the factory contract a plugin registers, and how to write one.

This is the second application of the mechanism the storage provider seam introduced; secrets reuses it wholesale, so the two pages describe the same shape with the nouns swapped.

A different shape from the inference/orchestrator seams

The inference-backend and orchestrator registries are selected per call — a model's sdk, a request's orchestration_mode. Secrets is not: it is a process-global singleton selected by its own config key, independent of the orchestrator. So it rides the same keyed registry + config-selected singleton mechanism storage uses:

Plugins register N provider factories into a registry keyed by an open method token. At boot, core reads runtime.secrets.method, looks that token up in the registry, and calls the factory to produce the one provider set on the hub.

This is deliberately not a hub slot (those are orchestrator-coupled, claimed only when boot_orchestrator == plugin.name). Secrets selection has nothing to do with the orchestrator, so it gets its own registry and its own config key — the sibling of runtime.storage.method.

The seam in one view

boot (Pipelex.setup)
  └─ build_registrar(config, builtin_plugins=BUILTIN_PLUGINS, …)   # pure, import-light
       ├─ for each built-in plugin                   (SecretsPlugin is one)
       └─ for each installed entry point in the requested groups
            #  "pipelex.plugins.kernel" alone on a kernel-only boot; both groups on a full one
            └─ plugin.register(registrar)            # side-effect-free
                 └─ registrar.add_secrets_provider(method=…, factory=…)
  └─ SecretsProviderRegistry(registrar.secrets_providers)    # stored on the hub
  └─ secrets_provider = registry.get_required(method=runtime.secrets.method)(runtime.secrets)
  └─ set_secrets_provider(secrets_provider)          # the one provider every consumer reads

There is no match runtime.secrets.method: anywhere in boot — the token set is open, so validation is the registry lookup. Adding a backend means registering a factory for its token; nothing in core changes. Every downstream consumer (get_secret(...), telemetry credentials, the storage gcp arm, search/extract workers) keeps calling get_secrets_provider() and is unaffected by which method was selected.

Where in boot secrets is resolved

Secrets is resolved early — right after plugin discovery, after the gateway/terms precondition gate but before the telemetry factory (its first in-process consumer) and well before storage selection. Two consequences that this ordering guarantees:

• Telemetry receives the config-selected secrets provider when it is constructed.
• Storage's gcp factory reads the config-selected secrets provider from the hub at its apply-point (it resolves GCP_CREDENTIALS_FILE_PATH) — secrets is already on the hub by then. This ordering is pinned by an integration test that boots a non-env secrets method alongside gcp storage and asserts the gcp arm read the external provider.

Injection precedence

Pipelex.setup resolves the secrets provider in this order:

1. an explicit setup(secrets_provider=...) parameter (test/host injection) — always wins;
2. the config-selected registry factory (get_required(method=runtime.secrets.method)(config)).

There is no separate core default: the built-in SecretsPlugin is the default supplier of the env method, so an ordinary boot always resolves through step 2.

The contract: SecretsProviderFactoryFn

A secrets backend is a typed callable — whole secrets config in, provider out (pipelex/plugins/secrets_provider_registry.py):

SecretsProviderFactoryFn = Callable[[SecretsProviderConfig], SecretsProviderAbstract]

Passing the whole SecretsProviderConfig (not a pre-resolved sub-config) is deliberate: it lets a factory read whatever it needs — an SDK-backed provider (Vault, AWS Secrets Manager) reads its own settings — at the boot apply-point, never at registration. The built-in env factory ignores the config entirely.

A plugin contributes one factory per method it serves by calling the registrar menu in its register, passing the token as a raw string:

registrar.add_secrets_provider(method="vault", factory=_make_vault_secrets_provider)

register is side-effect-free — it may call the registrar menu and nothing else (no hub access, no I/O, no SDK import). That is what keeps build_registrar safe to run more than once (at boot, and again for the pipelex plugins list diagnostic).

Import-light registration, deferred dependency

The factory is a plain callable stored at registration and invoked only at the boot apply-point. Two invariants follow, and they are what let an SDK-backed provider ship without weighing down every boot:

• Import-light. The plugin module must import no backend SDK at module load. The built-in SecretsPlugin needs none (the env provider reads environment variables); an out-of-tree pipelex-secrets-vault plugin must keep its hvac/boto3/etc. import inside the provider's methods (or its factory), never at module top-level, so discovery stays import-light even when the extra is installed.
• Fail at use, not at boot. An optional dependency raises MissingDependencyError (naming the package and the pipelex[<extra>] install hint) when the backend is used, not when it is registered or selected.

The built-in SecretsPlugin

pipelex/providers/secrets/secrets_plugin.py is the reference secrets plugin. It is core-unconditional — a secrets provider is required infra, so it joins the kernel layer's KERNEL_CORE_UNCONDITIONAL_PLUGIN_NAMES — composed into CORE_UNCONDITIONAL_PLUGIN_NAMES at the boot entrypoint — and cannot be disabled into a boot with no secrets provider:

class SecretsPlugin:
    name = "secrets"
    targets_api = PLUGIN_API_VERSION

    def register(self, registrar: PluginRegistrar) -> None:
        registrar.add_secrets_provider(method="env", factory=_make_env_secrets_provider)

The env factory ignores its config and returns a bare EnvSecretsProvider — the config parameter exists only to conform to SecretsProviderFactoryFn; an SDK-backed external provider is where reading the config earns its keep:

def _make_env_secrets_provider(config: SecretsProviderConfig) -> SecretsProviderAbstract:
    return EnvSecretsProvider()

env is a single-impl backend: EnvSecretsProvider resolves get_required_secret / get_optional_secret from environment variables, and its versioned reads (*_specific_version) raise NotImplementedError because environment variables have no version history. A full external provider (Vault, AWS Secrets Manager) implements those versioned reads.

Selecting a method by config

runtime.secrets.method is an open str token (Decision S1), not a closed enum. The built-in uses "env"; an external pipelex-secrets-<backend> plugin registers its own (e.g. "vault"). A config naming an external method parses fine — the token is stored verbatim and its installability is validated later, at registry lookup:

# .pipelex/pipelex.toml
[runtime.secrets]
method = "vault"          # an out-of-tree provider — selected iff its plugin is installed

Whether that token names an installed provider is validated at registry lookup, not at parse: an unknown method surfaces as UnknownSecretsMethodError at boot, which is the right layer — it lists the registered methods so the fix is obvious.

Fail-loud guarantees

The duplicate detection is the same fail-loud _add helper the inference/orchestrator/storage menus use — a conflict names both contributing plugins, so a collision between two installed secrets packages is unambiguous.

Authoring an out-of-tree secrets plugin

A third-party secrets plugin is a distribution that:

1. implements a SecretsProviderAbstract subclass — get_required_secret / get_optional_secret, the versioned *_specific_version reads, and set_secret_as_env_var — deferring any SDK import into those methods (import-light);
2. defines a plugin class (name, targets_api, register) whose register calls add_secrets_provider(method="<token>", factory=...) — and nothing else;
3. advertises itself under the pipelex.plugins.kernel entry-point group — a secrets provider is kernel-layer, and the group is how a plugin declares that (see Inference Backend Plugins for what each group means and what happens if you pick the wrong one; the pre-split pipelex.plugins group is retired and now fails startup):

# pyproject.toml of your plugin package
[project.entry-points."pipelex.plugins.kernel"]
vault_secrets = "pipelex_secrets_vault.plugin:VaultSecretsPlugin"

Installing the distribution makes the method selectable (runtime.secrets.method = "vault"); uninstalling removes it. No core change, no central registration list — presence is the source of truth. A discovered plugin can be quarantined without uninstalling via the runtime.plugins.disabled denylist (matched against the entry-point name before load, so a broken install can still be disabled to recover startup — see Inference Backend Plugins for the shared discovery/denylist machinery).

Use pipelex plugins list to see every discovered plugin, the entry-point group it was found under, what each contributed, and its denylist state. The Group column is the first thing to read when a plugin is missing: a built-in shows —, and an external plugin that resolved to the wrong layer shows it there.

Related

• Storage Provider Plugins — the sibling config-selected-singleton seam this one mirrors; its gcp arm reads the secrets provider at boot
• Secrets Provider Injection — supplying a custom secrets provider from your own vault or secrets manager
• Inference Backend Plugins — the per-call sibling seam and the shared discovery/denylist machinery
• Orchestrator Plugins — the other per-call seam, plus the hub-slot mechanism these config-selected seams deliberately avoid
• Error Model — how these errors render and dereference
• Architecture Overview