Skip to content

Configuration

Overview

Pipelex uses a TOML-based configuration system with shipped defaults plus project-level overrides.

  • Shipped defaults: Pipelex ships default values that are maintained in the Pipelex repository (contributors will see them in the repo root pipelex.toml). This is the baseline used by the installed package.
  • Project overrides: a project that uses Pipelex typically customizes behavior via files created in .pipelex/.

You can create the project configuration files by running:

pipelex init config

Configuration Setup Notes

  1. pipelex init config creates a template configuration file with sample settings. It does not include all possible configuration options - it's meant as a starting point.
  2. Using pipelex init config --reset will overwrite your existing pipelex.toml file without warning. Make sure to backup your configuration before using this flag.

For a complete list of all possible configuration options, refer to the configuration group documentation below.

Where to edit configuration in a project

The main project configuration files are:

  • .pipelex/pipelex.toml: project customization (logging, reporting, feature flags, etc.)
  • .pipelex/telemetry.toml: custom telemetry destinations
  • .pipelex/inference/…: inference backends, routing profiles, and model presets

Overrides (advanced)

In addition to .pipelex/pipelex.toml, Pipelex can apply override files at the project root (not in .pipelex/) for machine- and environment-specific settings:

  1. pipelex_local.toml
  2. pipelex_{environment}.toml (example: pipelex_dev.toml)
  3. pipelex_{run_mode}.toml (example: pipelex_normal.toml; unit tests may use tests/pipelex_unit_test.toml)
  4. pipelex_override.toml (recommended to gitignore)

Contributor details

For the full “where defaults live” and “how config is merged” explanation, see Configuration Internals.

Configuration Structure

The configuration is organized into three main sections:

  1. [pipelex] - Core Pipelex settings
  2. [cogt] - Cognitive tools and LLM settings
  3. [plugins] - Plugin-specific configurations

Each section contains multiple subsections for specific features and functionalities.

Configuration Override System

Pipelex uses a sophisticated configuration override system that loads and merges configurations in a specific order. This allows for fine-grained control over settings in different environments and scenarios.

The exact loading sequence is:

  1. Base configuration from the installed Pipelex package (pipelex.toml)
  2. Your project's base configuration (pipelex.toml in your project root)
  3. Local overrides (pipelex_local.toml)
  4. Environment-specific overrides (pipelex_{environment}.toml) - Example environments: dev, staging, prod -> based on the environment variable ENV in your .env file
  5. Run mode overrides (pipelex_{run_mode}.toml) - Example run modes: normal, unit_test
  6. Final overrides (pipelex_override.toml) (recommended to put in .gitignore)

Each subsequent configuration file in this sequence can override settings from the previous ones. This means:

  • Settings in pipelex_local.toml override the base configuration
  • Environment-specific settings override local settings
  • Run mode settings override environment settings
  • Super user settings override all previous settings

Override File Naming

  • Base config: pipelex.toml
  • Local overrides: pipelex_local.toml
  • Environment overrides: pipelex_dev.toml, pipelex_staging.toml, pipelex_prod.toml, etc.
  • Run mode overrides: pipelex_normal.toml, tests/pipelex_unit_test.toml, etc.
  • Final overrides: pipelex_override.toml

NB: The run_mode unit_test is used for testing purposes.

Best Practices for Overrides

  1. Use the base pipelex.toml for default settings
  2. Use pipelex_local.toml for machine-specific settings
  3. Use environment files for environment-specific settings (dev, staging, prod)
  4. Use run mode files for normal or unit_test configurations
  5. Use pipelex_override.toml sparingly, only for temporary overrides (add to .gitignore)

Best Practices

  1. Version Control: Include your base pipelex.toml in version control
  2. Environment Overrides: Use environment-specific files for sensitive or environment-dependent settings
  3. Documentation: Comment any custom settings for team reference
  4. Validation: Run pipelex validate all after making configuration changes
  5. Gitignore: Add local and sensitive override files to .gitignore