Skip to content

Configuration

Overview

Pipelex uses a TOML-based configuration system with shipped defaults plus project-level overrides.

  • Shipped defaults: Pipelex ships default values that are maintained in the Pipelex repository (contributors will see them in the repo root pipelex.toml). This is the baseline used by the installed package.
  • Project overrides: a project that uses Pipelex typically customizes behavior via files created in .pipelex/.

You can create the configuration files by running:

pipelex init config          # creates the global config at ~/.pipelex/
pipelex init config --local  # creates the project config at {project_root}/.pipelex/

Configuration Setup Notes

  1. By default pipelex init config targets the global ~/.pipelex/ directory; pass --local to create the project-level .pipelex/ instead.
  2. pipelex init config creates a template configuration file with sample settings. It does not include all possible configuration options - it's meant as a starting point.
  3. Running pipelex init config will overwrite your existing pipelex.toml file without warning. Make sure to backup your configuration before running this command.
  4. Credentials (~/.pipelex/.env) and the Gateway service-agreement file always remain in the global ~/.pipelex/ directory regardless of --local — only config, inference, and telemetry files are written to the project .pipelex/.

For a complete list of all possible configuration options, refer to the configuration group documentation below.

Where to edit configuration in a project

The main project configuration files are:

  • .pipelex/pipelex.toml: project customization (logging, reporting, feature flags, etc.)
  • .pipelex/telemetry.toml: custom telemetry destinations
  • .pipelex/inference/…: inference backends, routing profiles, and model presets

Overrides (advanced)

In addition to the base pipelex.toml, Pipelex applies override files from inside each .pipelex/ directory (both the global ~/.pipelex/ and the project's .pipelex/) for machine- and environment-specific settings:

  1. pipelex_local.toml
  2. pipelex_{environment}.toml (example: pipelex_dev.toml) — selected by the PIPELEX_ENV environment variable (see Selecting the environment)
  3. pipelex_{run_mode}.toml (example: pipelex_normal.toml; unit tests use tests/pipelex_unit_test.toml, the one override file read outside a .pipelex/ directory)
  4. pipelex_override.toml (recommended to gitignore)
  5. pipelex_temporary_override.toml (recommended to gitignore)

Contributor details

For the full “where defaults live” and “how config is merged” explanation, see Configuration Internals.

Configuration Structure

The configuration is organized into four main sections:

  1. [pipelex] - Core Pipelex settings
  2. [cogt] - Cognitive tools and LLM settings
  3. [plugins] - Plugin-specific configurations
  4. [migration] - Migration helpers for renamed settings

Each section contains multiple subsections for specific features and functionalities.

Configuration Override System

Pipelex uses a sophisticated configuration override system that loads and merges configurations in a specific order. This allows for fine-grained control over settings in different environments and scenarios.

The exact loading sequence is (later wins, per leaf key):

  1. Base configuration from the installed Pipelex package (pipelex.toml)
  2. Global base configuration (~/.pipelex/pipelex.toml)
  3. Global override sequence, from ~/.pipelex/: pipelex_local.toml, pipelex_{environment}.toml, pipelex_{run_mode}.toml, pipelex_override.toml, pipelex_temporary_override.toml
  4. Your project's base configuration ({project_root}/.pipelex/pipelex.toml)
  5. Project override sequence, from the project's .pipelex/: the same five files as step 3
  6. Programmatic overrides passed in code, if any

Notes on the override sequence:

  • pipelex_{environment}.toml (example: pipelex_dev.toml) is selected by the PIPELEX_ENV environment variable (see Selecting the environment)
  • pipelex_{run_mode}.toml — example run modes: normal, unit_test; under unit testing the run-mode overlay is sourced exclusively from ./tests/pipelex_{run_mode}.toml
  • Each subsequent configuration file in this sequence can override settings from the previous ones, and project-level files override the global ~/.pipelex/ layer

Override File Naming

  • Base config: pipelex.toml
  • Local overrides: pipelex_local.toml
  • Environment overrides: pipelex_dev.toml, pipelex_staging.toml, pipelex_prod.toml, etc.
  • Run mode overrides: pipelex_normal.toml, tests/pipelex_unit_test.toml, etc.
  • Final overrides: pipelex_override.toml

NB: The run_mode unit_test is used for testing purposes.

Selecting the environment

The pipelex_{environment}.toml overlay is picked at runtime from the PIPELEX_ENV environment variable.

Value Overlay file loaded
local pipelex_local.toml (also loaded as the local override layer; see above)
dev pipelex_dev.toml
staging pipelex_staging.toml
prod pipelex_prod.toml

If PIPELEX_ENV is unset, Pipelex defaults to dev. Any other value raises an error at startup — the accepted values are defined by the RunEnvironment enum in pipelex/system/runtime.py.

The selected environment is also stamped on OpenTelemetry spans as deployment.environment, so it doubles as the environment label for traces and metrics.

Set it the way you set any other env var — for example, in your shell, your .env file, your CI configuration, or your container runtime:

export PIPELEX_ENV=staging

Best Practices for Overrides

  1. Use the base pipelex.toml for default settings
  2. Use pipelex_local.toml for machine-specific settings
  3. Use environment files for environment-specific settings (dev, staging, prod)
  4. Use run mode files for normal or unit_test configurations
  5. Use pipelex_override.toml sparingly, only for temporary overrides (add to .gitignore)

Best Practices

  1. Version Control: Include your base pipelex.toml in version control
  2. Environment Overrides: Use environment-specific files for sensitive or environment-dependent settings
  3. Documentation: Comment any custom settings for team reference
  4. Validation: Run pipelex validate --all after making configuration changes
  5. Gitignore: Add local and sensitive override files to .gitignore