Skip to content

Configuration

Overview

Pipelex uses a TOML-based configuration system with shipped defaults plus project-level overrides.

  • Shipped defaults: Pipelex ships default values that are maintained in the Pipelex repository (contributors will see them in the repo root pipelex.toml). This is the baseline used by the installed package.
  • Project overrides: a project that uses Pipelex typically customizes behavior via files created in .pipelex/.

You can create the project configuration files by running:

pipelex init config

Configuration Setup Notes

  1. pipelex init config creates a template configuration file with sample settings. It does not include all possible configuration options - it's meant as a starting point.
  2. Running pipelex init config will overwrite your existing pipelex.toml file without warning. Make sure to backup your configuration before running this command.

For a complete list of all possible configuration options, refer to the configuration group documentation below.

Where to edit configuration in a project

The main project configuration files are:

  • .pipelex/pipelex.toml: project customization (logging, reporting, feature flags, etc.)
  • .pipelex/telemetry.toml: custom telemetry destinations
  • .pipelex/inference/…: inference backends, routing profiles, and model presets

Overrides (advanced)

In addition to .pipelex/pipelex.toml, Pipelex can apply override files at the project root (not in .pipelex/) for machine- and environment-specific settings:

  1. pipelex_local.toml
  2. pipelex_{environment}.toml (example: pipelex_dev.toml) — selected by the PIPELEX_ENV environment variable (see Selecting the environment)
  3. pipelex_{run_mode}.toml (example: pipelex_normal.toml; unit tests may use tests/pipelex_unit_test.toml)
  4. pipelex_override.toml (recommended to gitignore)

Contributor details

For the full “where defaults live” and “how config is merged” explanation, see Configuration Internals.

Configuration Structure

The configuration is organized into three main sections:

  1. [pipelex] - Core Pipelex settings
  2. [cogt] - Cognitive tools and LLM settings
  3. [plugins] - Plugin-specific configurations

Each section contains multiple subsections for specific features and functionalities.

Configuration Override System

Pipelex uses a sophisticated configuration override system that loads and merges configurations in a specific order. This allows for fine-grained control over settings in different environments and scenarios.

The exact loading sequence is:

  1. Base configuration from the installed Pipelex package (pipelex.toml)
  2. Your project's base configuration (pipelex.toml in your project root)
  3. Local overrides (pipelex_local.toml)
  4. Environment-specific overrides (pipelex_{environment}.toml) - Example environments: local, dev, staging, prod — selected by the PIPELEX_ENV environment variable (see Selecting the environment)
  5. Run mode overrides (pipelex_{run_mode}.toml) - Example run modes: normal, unit_test
  6. Final overrides (pipelex_override.toml) (recommended to put in .gitignore)

Each subsequent configuration file in this sequence can override settings from the previous ones. This means:

  • Settings in pipelex_local.toml override the base configuration
  • Environment-specific settings override local settings
  • Run mode settings override environment settings
  • Super user settings override all previous settings

Override File Naming

  • Base config: pipelex.toml
  • Local overrides: pipelex_local.toml
  • Environment overrides: pipelex_dev.toml, pipelex_staging.toml, pipelex_prod.toml, etc.
  • Run mode overrides: pipelex_normal.toml, tests/pipelex_unit_test.toml, etc.
  • Final overrides: pipelex_override.toml

NB: The run_mode unit_test is used for testing purposes.

Selecting the environment

The pipelex_{environment}.toml overlay is picked at runtime from the PIPELEX_ENV environment variable.

Value Overlay file loaded
local pipelex_local.toml (also loaded as the local override layer; see above)
dev pipelex_dev.toml
staging pipelex_staging.toml
prod pipelex_prod.toml

If PIPELEX_ENV is unset, Pipelex defaults to dev. Any other value raises an error at startup — the accepted values are defined by the RunEnvironment enum in pipelex/system/runtime.py.

The selected environment is also stamped on OpenTelemetry spans as deployment.environment, so it doubles as the environment label for traces and metrics.

Set it the way you set any other env var — for example, in your shell, your .env file, your CI configuration, or your container runtime:

export PIPELEX_ENV=staging

Best Practices for Overrides

  1. Use the base pipelex.toml for default settings
  2. Use pipelex_local.toml for machine-specific settings
  3. Use environment files for environment-specific settings (dev, staging, prod)
  4. Use run mode files for normal or unit_test configurations
  5. Use pipelex_override.toml sparingly, only for temporary overrides (add to .gitignore)

Best Practices

  1. Version Control: Include your base pipelex.toml in version control
  2. Environment Overrides: Use environment-specific files for sensitive or environment-dependent settings
  3. Documentation: Comment any custom settings for team reference
  4. Validation: Run pipelex validate --all after making configuration changes
  5. Gitignore: Add local and sensitive override files to .gitignore