CSV Input & Output

Pipelex can read a CSV file directly into a typed list and write a list output back out as CSV. A CSV maps to a ListContent of a flat concept: each row becomes one instance of the concept, and each column maps to one of its fields. No new language construct is involved — you describe the row concept exactly as you would any structured concept.

This covers reading a .csv referenced from your inputs.json, writing a result with --save-csv, and the rules and limitations of the v1 codec.

The flat-concept rule

A concept used with CSV must be flat: every field is a scalar. The accepted types are:

• text → str
• integer → int
• number → float
• boolean → bool
• date → datetime.date (an ISO YYYY-MM-DD value)
• datetime → datetime.datetime (an ISO YYYY-MM-DDTHH:MM:SS value)

Optional fields are allowed, and so are choice-constrained scalars — but only string-valued ones (a text-typed choices field, a string Literal, or a StrEnum). A non-string choice such as Literal[1, 2, 3] or an IntEnum is rejected: it would serialize to a CSV string but cannot coerce back from one, so accepting it would break the round trip. Any nested concept, list, dict, union, or Any-typed field is likewise rejected with a clear error that names the offending field — a CSV cell has no room for nested structure, so you must project to a flat concept first.

[concept.Person]
description = "A person record read from a CSV row"
[concept.Person.structure]
name       = { type = "text",    required = true,  description = "Full name" }
job        = { type = "text",    required = true,  description = "Occupation" }
country    = { type = "text",    required = true,  description = "Country of origin" }
birth_year = { type = "integer", required = true,  description = "Year of birth" }
death_year = { type = "integer", required = false, description = "Year of death, if deceased" }

Reading a CSV as input

Reference the file from inputs.json with a url whose path ends in a tabular suffix (.csv, or .xlsx once the Excel backend is available), under a flat structured concept. Pipelex detects the tabular suffix and reads the file into a ListContent of that concept — one CSV file becomes one list, and the concept names the row type:

{
  "people": {
    "concept": "demo.Person",
    "content": { "url": "people.csv" }
  }
}

{ "people": "people.csv" }

{ "people": { "url": "people.csv" } }

Given a people.csv:

name,job,country,birth_year,death_year
Ada Lovelace,Mathematician,United Kingdom,1815,1852
Grace Hopper,Computer Scientist,United States,1906,1992
Vint Cerf,Computer Scientist,United States,1943,

people arrives as a ListContent[Person] with three items. birth_year is coerced to int, and Vint Cerf's blank death_year cell becomes None (which is why death_year is declared optional).

Header and column rules

• The header row is required, and headers must match the concept's field names exactly — there is no implicit renaming.

• An extra column (one with no matching field) is an error.

• A missing required column is an error.

• A missing optional column is fine — a field with a default_value takes its default for every row, and any other optional field is None for every row. (Pipelex's own writer always emits every column, so this only matters for hand-authored partial CSVs.)

• An empty cell means no value: a field with a default_value takes its default, any other optional field becomes None, and a required field refuses it.

Type coercion

Cell strings are coerced through Pydantic's lax validation. The contract is pinned by tests, so you can rely on it:

A cell that cannot be coerced raises a clear error naming the 1-based row, the concept, and the field, so you can find the offending value.

Writing a result as CSV

Pass --save-csv <path> to any run command to write the pipeline's main stuff to a CSV file:

pipelex run pipe summarize_people --inputs inputs.json --save-csv summaries.csv
pipelex run bundle demo.mthds --save-csv out.csv
pipelex run method my_method --save-csv out.csv

The path is literal and cwd-relative — it is written exactly where you point it, not under --output-dir. Parent directories are created as needed. The output's main stuff must be a flat list; if it is not a list, or its row concept is not flat, you get a clear error rather than a malformed file. The header is derived from the row concept's structure — the class the run produced its rows with, or, for a run that produced no rows, the declared concept resolved by reloading the method — so an empty list still writes a correct header-only file. On write, None is rendered as an empty cell and booleans as lowercase true/false, so a value round-trips back through a read unchanged, except an empty text: it is written as an empty cell too, so it reads back as the field's default, or as None for an optional field without one.

Round-trip example

The pieces compose into a full CSV-in → process → CSV-out round trip. Here a PipeBatch runs once per person, a PipeLLM writes a one-sentence persona, and a PipeCompose assembles the output row from selected input fields plus that sentence. Both the input concept Person and the output concept PersonSummary are flat, so both sides satisfy the flat-concept rule:

[concept.PersonSummary]
description = "A person row reduced to name + country plus a one-sentence persona summary"
[concept.PersonSummary.structure]
name    = { type = "text", required = true, description = "Full name" }
country = { type = "text", required = true, description = "Country of origin" }
summary = { type = "text", required = true, description = "One-sentence persona summary" }

[pipe.summarize_people]
type             = "PipeBatch"
description      = "Summarize each person in the list"
inputs           = { people = "Person[]" }
output           = "PersonSummary[]"
branch_pipe_code = "summarize_person"
input_list_name  = "people"
input_item_name  = "person"

[pipe.summarize_person]
type = "PipeSequence"
description = "Summarize one person"
inputs = { person = "Person" }
output = "PersonSummary"
steps = [
    { pipe = "write_persona", result = "persona" },
    { pipe = "assemble_summary", result = "summary_row" },
]

[pipe.write_persona]
type = "PipeLLM"
description = "Write a one-sentence persona for the person"
inputs = { person = "Person" }
output = "Text"
prompt = "Write a one-sentence persona summary for $person."

[pipe.assemble_summary]
type = "PipeCompose"
description = "Assemble the summary row from selected input fields plus the persona sentence"
inputs = { person = "Person", persona = "Text" }
output = "PersonSummary"

[pipe.assemble_summary.construct]
name    = { from = "person.name" }
country = { from = "person.country" }
summary = { from = "persona" }

pipelex run pipe summarize_people --inputs inputs.json --save-csv summaries.csv

This reads people.csv into ListContent[Person], summarizes each person, and writes a summaries.csv with header name,country,summary and one row per input person.

Configuring delimiter and encoding

The codec defaults to a comma delimiter and UTF-8 encoding. Reads use utf-8-sig, which transparently strips a leading byte-order mark (BOM) — Excel's "CSV UTF-8" export always prepends one — and reads plain UTF-8 unchanged when there is no BOM. Writes use plain utf-8 (no BOM). Both delimiter and encoding are parameters of the codec functions (list_content_from_csv / csv_from_list_content), but there is no user-facing config surface (TOML default or CLI flag) in v1 — they are never guessed from the file. A user-facing knob for European semicolon-delimited CSVs is a planned follow-up.

Limitations in v1

• Local file paths only. A remote url with a tabular suffix — http(s)://, s3://, gs://, a signed URL — is actively rejected with a clear error, never opened: its problem is request.unsupported, naming the input's field as inputs.<name> and the reason remote_table_url, and never the URL. A url that does not parse is input.file_unreadable, with the reason the URL does not parse. Reading remote tabular files is a deferred follow-up.

• Not uploaded on a hosted run. A run on the hosted Pipelex API uploads the local files named at document and image inputs only, so a .csv path at a structured-list input reaches the hosted runner as a path on your machine, which it cannot read. Give the rows as a JSON list of objects instead, or run with --local. --save-csv is refused on a hosted run too (see Running on the Hosted API).

• .xlsx is recognized but not yet implemented. An .xlsx path is detected as tabular and routed to a clear "needs pipelex[tabular]" message; the Excel backend itself is not built in v1.

• Empty string is indistinguishable from None. An empty cell reads as None, and on write None is rendered as an empty cell, so a genuinely empty string "" cannot be told apart from a missing value through a CSV round trip.

• A bare { "url": "...csv" } reference reads as a table. Detection triggers only on the exact single-key wrapper shape — a content reference that is precisely { "url": <tabular path> }. So a concept whose only field is url, given { "url": "report.csv" }, is read as a CSV table rather than as a single record holding url = "report.csv"; use a non-tabular value if you need the single-record reading. A record with any sibling key (e.g. { "label": "Home", "url": "report.csv" }) is never affected — it stays an ordinary record, with its fields preserved.

• No formula-injection escaping. Cells are written verbatim to preserve data fidelity, so a value beginning with =, +, -, or @ is not escaped. If you hand a generated CSV to a spreadsheet application, treat untrusted content accordingly. An opt-in escape flag is a deferred follow-up.

Related Documentation

• Providing Inputs to Pipelines
• Executing Pipelines
• Pipe Output
• Run Commands