Skip to content

Configuration Internals (defaults and overrides)

This page is for contributors: it explains where Pipelex defaults live, how configuration is merged at runtime, and where to change the behavior in code.

For the user-facing configuration guide, see Configuration Overview.

Mental model (for contributors)

Pipelex configuration is merged in layers:

  • Shipped defaults: maintained by the Pipelex project and used as the baseline by the installed package.
  • Global config (~/.pipelex/): machine-wide settings for a developer, applied to every project on the machine.
  • Project config ({project_root}/.pipelex/): per-project settings, edited by teams using Pipelex.
  • Override files at each level (pipelex_local.toml, pipelex_{environment}.toml, pipelex_{run_mode}.toml, pipelex_override.toml, pipelex_temporary_override.toml): optional, typically gitignored, used for personal or ephemeral tweaks.

The key idea: global personal preferences layer under project-specific settings — both are loaded, and project values win on collisions. Nothing in ~/.pipelex/ is shadowed by the mere existence of a project .pipelex/.

Where defaults live (Pipelex repository)

  • Default values (baseline): pipelex/pipelex.toml, shipped inside the package
  • User-facing templates copied into .pipelex/ by the init flow: pipelex/kit/configs/…

Merge order (runtime)

The configuration loading/merging behavior is implemented in pipelex/system/configuration/config_loader.py. Files are deep-merged via load_toml_from_path_and_merge_with_overrides; the later a file appears in the load list, the higher its precedence per leaf key.

Load order:

  1. Package defaults — pipelex/pipelex.toml shipped with the installed package.
  2. Global base — ~/.pipelex/pipelex.toml.
  3. Global override sequence — from ~/.pipelex/, in this order:
    1. pipelex_local.toml
    2. pipelex_{environment}.toml
    3. pipelex_{run_mode}.toml (see unit-test special case below)
    4. pipelex_override.toml
    5. pipelex_temporary_override.toml
  4. Project base — {project_root}/.pipelex/pipelex.toml, if a project .pipelex/ exists and is distinct from ~/.pipelex/.
  5. Project override sequence — same five files as step 3, read from {project_root}/.pipelex/, if the project dir is distinct from the global dir.
  6. Programmatic extra_overrides passed into ConfigLoader.load_config() (if any).

Unit-test special case

When runtime_manager.is_unit_testing is true, the pipelex_{run_mode}.toml entry is sourced exclusively from ./tests/pipelex_{run_mode}.toml (e.g. tests/pipelex_unit_test.toml). Global and project run_mode files are not loaded, keeping test runs hermetic from machine-wide overrides.

The inference documents

.pipelex/inference/backends.toml and .pipelex/inference/routing_profiles.toml are not part of the pipelex.toml merge. Each is its own document with a shorter sequence, built by ConfigLoader.backends_file_paths() / routing_profiles_file_paths() and read by load_toml_from_base_and_overrides:

  1. The base — the project's file if the project .pipelex/inference/ has one, otherwise the global one. One file, winner takes all; a personal override cannot stand in for it.
  2. Global override — ~/.pipelex/inference/backends_override.toml (respectively routing_profiles_override.toml), if present.
  3. Project override — the same file under the project's .pipelex/inference/, if present and distinct from the global one.

The order differs from pipelex.toml's on purpose: there the project base beats the global override, here the global override beats the project base. A global inference override exists so one machine-wide choice ("run on this backend") reaches every project, and every project carries a tracked backends.toml, so a project base that won would defeat the file. The project override still wins over the global one. With an explicit config_dir (the doctor's --global, an init targeting one directory), the sequence is that directory's base and its own override, nothing layered.

Where to change things in code

  • Config merge logic: pipelex/system/configuration/config_loader.py
  • Override file sequences: ConfigLoader._override_files_for_dir and ConfigLoader._plugin_override_files_for_dir in that same file

Contributor guidelines for config changes

  • If you change defaults, update pipelex/pipelex.toml (and consider whether templates in pipelex/kit/configs/… should also be updated).
  • If you change templates, keep them user-focused and stable; avoid adding internal-only details.
  • If you change merge order or override semantics, treat it as a potentially breaking change and document it (changelog + migration notes if needed).
  • If you change the shape of a configuration model — renaming, moving or removing a key, or dropping an enumerated value — make check will stop you until the change is recorded in that surface's migration ledger. Adding a key is not a shape change and needs nothing, provided the key has a default: the defaults layer is what an old file falls back on, and a required key with no default breaks every existing file. The gate is make check-migration-schemas (alias cmig) and its regenerator is make up-migration-schemas (alias umig). The contract, including what a ledger may contain and what is guaranteed to a user whose files are migrated, is in The Migration Ledger.
  • The repository's own .pipelex/ dogfoods the templates, and make check-config-sync (alias ccs) holds the two directories to the same contents; make up-kit-configs (alias ukc) mirrors .pipelex/ back onto pipelex/kit/configs/. Both look past the same exclusions, declared together in pipelex/cli/dev_cli/config_sync_exclusions.py, so the check can never go red on something the sync would not fix. Three kinds of file are excluded: the per-developer overrides and the templates that intentionally differ between the two sides (CONFIG_SYNC_EXCLUDED_FILES, each with its reason beside it); the .gitignore pipelex writes into a configuration directory it owns, which is a runtime artifact of that directory and has no kit counterpart; and, by glob, the timestamped copies a migration leaves behind (CONFIG_SYNC_EXCLUDED_PATTERNS, built from the same infixes and stamp shape pipelex/migration/backup.py writes, so a rename there cannot leave a pattern matching nothing). That last one matters because migrating in a checkout is an ordinary thing to do: the run's own .gitignore already keeps those copies out of git status, and a sync check going red on them would put the dirtied-repository problem straight back, one gate over. The declaration sits under the dev CLI rather than beside the kit paths for a reason worth knowing before you move it: deriving the patterns from the namer means importing pipelex.migration, which reaches the pipeline layer, and pipelex/kit/paths.py is inside the kernel boot closure that hub layering keeps free of exactly that. When you add a file to .pipelex/ that genuinely has no kit counterpart, add it to one of those two sets with a comment saying why — do not delete it to get a green check.

Test Profile Configuration

The test profile system controls which AI models are included in parametrized tests. This is separate from the main Pipelex configuration.

Files

File Purpose Tracked
.pipelex-dev/test_profiles.toml Base profiles and model collections Yes
.pipelex-dev/test_profiles_override.toml Local customizations No (gitignored)

How it works

  1. Collections define reusable lists of models organized by provider
  2. Profiles reference collections or specify models directly
  3. The preprocessing command resolves references and generates fixture files

Key code locations

  • Profile loading & merging: pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py
  • Generated fixtures: tests/integration/pipelex/fixtures/_generated_model_sets.py

Common tasks

# Regenerate fixtures with default (dev) profile
make regenerate-test-models

# Use a specific profile
make regenerate-test-models TEST_PROFILE=full

For full documentation, see Test Profile Configuration.