CSV Input & Output

Pipelex can read a CSV file directly into a typed list and write a list output back out as CSV. A CSV maps to a ListContent of a flat concept: each row becomes one instance of the concept, and each column maps to one of its fields. No new language construct is involved — you describe the row concept exactly as you would any structured concept.

This covers reading a .csv referenced from your inputs.json, writing a result with --save-csv, and the rules and limitations of the v1 codec.

The flat-concept rule

A concept used with CSV must be flat: every field is a scalar. The accepted types are:

• text → str
• integer → int
• number → float
• boolean → bool
• date → datetime.date (an ISO YYYY-MM-DD value)
• datetime → datetime.datetime (an ISO YYYY-MM-DDTHH:MM:SS value)

Optional fields are allowed, and so are choice-constrained scalars — but only string-valued ones (a text-typed choices field, a string Literal, or a StrEnum). A non-string choice such as Literal[1, 2, 3] or an IntEnum is rejected: it would serialize to a CSV string but cannot coerce back from one, so accepting it would break the round trip. Any nested concept, list, dict, union, or Any-typed field is likewise rejected with a clear error that names the offending field — a CSV cell has no room for nested structure, so you must project to a flat concept first.

[concept.Person]
description = "A person record read from a CSV row"
[concept.Person.structure]
name       = { type = "text",    required = true,  description = "Full name" }
job        = { type = "text",    required = true,  description = "Occupation" }
country    = { type = "text",    required = true,  description = "Country of origin" }
birth_year = { type = "integer", required = true,  description = "Year of birth" }
death_year = { type = "integer", required = false, description = "Year of death, if deceased" }

Reading a CSV as input

Reference the file from inputs.json with a url whose path ends in a tabular suffix (.csv, or .xlsx once the Excel backend is available), under a flat structured concept. Pipelex detects the tabular suffix and reads the file into a ListContent of that concept — one CSV file becomes one list, and the concept names the row type:

{
  "people": {
    "concept": "demo.Person",
    "content": { "url": "people.csv" }
  }
}

{ "people": "people.csv" }

{ "people": { "url": "people.csv" } }

Given a people.csv:

name,job,country,birth_year,death_year
Ada Lovelace,Mathematician,United Kingdom,1815,1852
Grace Hopper,Computer Scientist,United States,1906,1992
Vint Cerf,Computer Scientist,United States,1943,

people arrives as a ListContent[Person] with three items. birth_year is coerced to int, and Vint Cerf's blank death_year cell becomes None (which is why death_year is declared optional).

Header and column rules

• The header row is required, and headers must match the concept's field names exactly — there is no implicit renaming.

• An extra column (one with no matching field) is an error.

• A missing required column is an error.

• A missing optional column is fine — that field is None for every row. (Pipelex's own writer always emits every column, so this only matters for hand-authored partial CSVs.)

• An empty cell becomes None before validation, so the field must be optional to accept it.

Type coercion

Cell strings are coerced through Pydantic's lax validation. The contract is pinned by tests, so you can rely on it:

A cell that cannot be coerced raises a clear error naming the 1-based row, the concept, and the field, so you can find the offending value.

Writing a result as CSV

Pass --save-csv <path> to any run command to write the pipeline's main stuff to a CSV file:

pipelex run pipe summarize_people --inputs inputs.json --save-csv summaries.csv
pipelex run bundle demo.mthds --save-csv out.csv
pipelex run method my_method --save-csv out.csv

The path is literal and cwd-relative — it is written exactly where you point it, not under --output-dir. Parent directories are created as needed. The output's main stuff must be a flat list; if it is not a list, or its row concept is not flat, you get a clear error rather than a malformed file. The header is derived from the declared row concept (not from the first item), so an empty list still writes a correct header-only file. On write, None is rendered as an empty cell and booleans as lowercase true/false, so a value round-trips back through a read unchanged.

Round-trip example

The pieces compose into a full CSV-in → process → CSV-out round trip. Here a PipeBatch runs once per person, a PipeLLM writes a one-sentence persona, and a PipeCompose assembles the output row from selected input fields plus that sentence. Both the input concept Person and the output concept PersonSummary are flat, so both sides satisfy the flat-concept rule:

[concept.PersonSummary]
description = "A person row reduced to name + country plus a one-sentence persona summary"
[concept.PersonSummary.structure]
name    = { type = "text", required = true, description = "Full name" }
country = { type = "text", required = true, description = "Country of origin" }
summary = { type = "text", required = true, description = "One-sentence persona summary" }

[pipe.summarize_people]
type             = "PipeBatch"
description      = "Summarize each person in the list"
inputs           = { people = "Person[]" }
output           = "PersonSummary[]"
branch_pipe_code = "summarize_person"
input_list_name  = "people"
input_item_name  = "person"

[pipe.summarize_person]
type = "PipeSequence"
description = "Summarize one person"
inputs = { person = "Person" }
output = "PersonSummary"
steps = [
    { pipe = "write_persona", result = "persona" },
    { pipe = "assemble_summary", result = "summary_row" },
]

[pipe.write_persona]
type = "PipeLLM"
description = "Write a one-sentence persona for the person"
inputs = { person = "Person" }
output = "Text"
prompt = "Write a one-sentence persona summary for $person."

[pipe.assemble_summary]
type = "PipeCompose"
description = "Assemble the summary row from selected input fields plus the persona sentence"
inputs = { person = "Person", persona = "Text" }
output = "PersonSummary"

[pipe.assemble_summary.construct]
name    = { from = "person.name" }
country = { from = "person.country" }
summary = { from = "persona" }

pipelex run pipe summarize_people --inputs inputs.json --save-csv summaries.csv

This reads people.csv into ListContent[Person], summarizes each person, and writes a summaries.csv with header name,country,summary and one row per input person.

Configuring delimiter and encoding

The codec defaults to a comma delimiter and UTF-8 encoding. Reads use utf-8-sig, which transparently strips a leading byte-order mark (BOM) — Excel's "CSV UTF-8" export always prepends one — and reads plain UTF-8 unchanged when there is no BOM. Writes use plain utf-8 (no BOM). Both delimiter and encoding are parameters of the codec functions (list_content_from_csv / csv_from_list_content), but there is no user-facing config surface (TOML default or CLI flag) in v1 — they are never guessed from the file. A user-facing knob for European semicolon-delimited CSVs is a planned follow-up.

Limitations in v1

• Local file paths only. A remote url with a tabular suffix — http(s)://, s3://, gs://, a signed URL — is actively rejected with a clear error, never opened. Reading remote tabular files is a deferred follow-up.

• .xlsx is recognized but not yet implemented. An .xlsx path is detected as tabular and routed to a clear "needs pipelex[tabular]" message; the Excel backend itself is not built in v1.

• Empty string is indistinguishable from None. An empty cell reads as None, and on write None is rendered as an empty cell, so a genuinely empty string "" cannot be told apart from a missing value through a CSV round trip.

• A bare { "url": "...csv" } reference reads as a table. Detection triggers only on the exact single-key wrapper shape — a content reference that is precisely { "url": <tabular path> }. So a concept whose only field is url, given { "url": "report.csv" }, is read as a CSV table rather than as a single record holding url = "report.csv"; use a non-tabular value if you need the single-record reading. A record with any sibling key (e.g. { "label": "Home", "url": "report.csv" }) is never affected — it stays an ordinary record, with its fields preserved.

• No formula-injection escaping. Cells are written verbatim to preserve data fidelity, so a value beginning with =, +, -, or @ is not escaped. If you hand a generated CSV to a spreadsheet application, treat untrusted content accordingly. An opt-in escape flag is a deferred follow-up.

Related Documentation

• Providing Inputs to Pipelines
• Executing Pipelines
• Pipe Output
• Run Commands