Per-node Usage Attribution

A run produces two artifacts that describe the same thing from different angles: pipe_output.graph_spec (the shape of the run) and pipe_output.tokens_usages (what it cost). Both are built from one trace-event stream, and every usage event in that stream already names the graph node it belongs to. This page is about the projection that keeps that link — GraphSpec.usage and NodeSpec.usage — and about the three states a usage number can be in, which is the part consumers get wrong.

Two projections of one event stream

Every inference call emits a UsageReportEvent carrying node_id (from trace_context.parent_node_id) and the full internal AnyTokensUsage. That single stream is read twice:

• UsageAggregator (pipelex/tracing/usage_aggregator.py) projects it into a flat list — one record per call, node_id dropped. That list becomes pipe_output.tokens_usages, trimmed to TokensUsageRecord at the client boundary, and feeds the console cost table and the CSV.
• usage_attribution (pipelex/tracing/usage_attribution.py) projects it into per-node totals — node_id kept, calls folded together. Those totals become NodeSpec.usage and GraphSpec.usage, assembled by GraphSpecAssembler.

The two projections are deliberately separate — tokens_usages is a shipped, extra="forbid" client contract, and a graph is not the place to re-litigate its record shape. What they are not allowed to do is disagree on arithmetic, so both compute every dollar through the same compute_tokens_usage_cost (pipelex/cogt/usage/usage_cost.py) and define a token total the same way AggregatedCosts.total_nb_tokens does.

flowchart TB
    CALL["inference call completes"]
    EVENT["UsageReportEvent<br/>{ node_id, tokens_usage }"]

    subgraph LOG["event log — one stream, ordered by (chain, sequence)"]
        STREAM["PipeStart · PipeEndSuccess · Edge · UsageReport"]
    end

    subgraph ATTRIBUTED["attributed projection"]
        direction TB
        ASSEMBLER["GraphSpecAssembler<br/>node_id kept, calls folded"]
        GRAPH["NodeSpec.usage<br/>GraphSpec.usage"]
    end

    subgraph FLATTENED["flat projection"]
        direction TB
        AGGREGATOR["UsageAggregator<br/>node_id dropped, one record per call"]
        FLAT["PipeOutput.tokens_usages"]
        SINKS["cost table · CSV · API wire"]
    end

    CALL --> EVENT
    EVENT -- "the link exists here" --> STREAM
    STREAM --> ASSEMBLER
    STREAM --> AGGREGATOR
    ASSEMBLER --> GRAPH
    AGGREGATOR --> FLAT
    FLAT --> SINKS

When attribution happens, and why it waits

GraphSpecAssembler folds usage events during pass 1, keyed by the node_id the event named — with no node lookup. Resolution against the real node set happens in pass 2. This is not an optimisation: a UsageReportEvent can legitimately be read before the PipeStartEvent of the node it names, because the two can come from different workers and the stream is ordered by (chain, sequence), the chain being the attempts of the executions that wrote the event. "Is this node real?" is only answerable once every event has been seen.

A usage event of a superseded attempt, one a later attempt of its execution or of an execution that dispatched it replaced, names a node the graph does not show, since the graph is built from current events only (see Executions and Their Attempts). Its inference was spent all the same, so the assembler counts it in the run's total, as unattributed usage.

Pass 2 then does three things: resolves each accumulated total onto its node (or into the unattributed bucket), rolls subtrees up the parent_node_id chain, and computes the run total.

The three states of a usage number

This is the part that matters for anything rendering a graph. A missing number means one of three things, and they must not be conflated — $0.00 on an unrated node is a lie, and a controller that reports nothing because it was read through the wrong half looks like a bug.

Unrated is not an edge case: compute_tokens_usage_cost returns None whenever the model carries no rate table, and dry/mock runs hardcode an empty rate table — so every dry-run graph is unrated.

A note on token counts, for anyone rendering them. The counts are faithful to what the run reported, which is not the same as being meaningful to a reader. Two cases to handle before putting a number on screen: a dry run reports one synthetic call per would-be inference with zero tokens (or, under --mock-usage, with invented non-zero tokens — never display those, they measure nothing), and a PipeSearch reports its provider's billing unit in the token field, so a single search shows ~2,000,000 "tokens" for a correct price of $0.005. The cost is the number that survives both; mthds-ui accordingly displays cost only, and only for a real run.

The invariants

NodeUsageSpec (pipelex/graph/graphspec.py) states four invariants in its docstring. They are the contract; each has a named test in tests/unit/pipelex/tracing/test_usage_attribution.py.

1. usage is None is all-or-nothing across a graph. It means no usage was reported anywhere in the run — collection was off, or the run made zero inference calls. As soon as one usage event was seen, every node carries a spec, zeroed where nothing ran. A controller, a lifted pipe and a PipeFunc all get inference_calls=0, never usage=None. So the field never distinguishes "this node was not measured" from "that node was".
2. cost is None ⟺ rated_inference_calls == 0. Nothing else. "Made no call" and "made only unrated calls" both land here, and inference_calls tells them apart.
3. inference_calls > rated_inference_calls > 0 means cost is a lower bound, not a total — some of this node's calls carried no rate table. A UI that renders it as a complete figure is silently wrong about money.
4. total_tokens is input_joined + output, the same definition as AggregatedCosts.total_nb_tokens. It is not the sum of nb_tokens_by_category: input_cached is a subset of input, not additive, so summing the dict double-counts. Never sum the dict; read total_tokens.

The model that ran, vs the model that was asked for

A GraphSpec names a model in three places, and only one of them is the outcome:

execution_data.resolved_model is captured from LLMSetting.model (pipe_llm.py), which is a handle: alias resolution happens later, at inference time. The name overstates what it holds — roughly a third of the nodes in a typical corpus carry an unresolved @alias there. Only by_model survives alias resolution, deck defaults, and any fallback or retry that landed somewhere other than what was requested.

by_model is a list because one node routinely uses more than one model. A PipeLLM's text pass and its object-structuring pass resolve separately (that is what resolved_model_for_object is about), and a retry can land elsewhere again. Collapsing them into "the model" would be wrong in exactly the way a single cost across mixed rated and unrated calls is wrong. An entry is one model type and name, since a handle names one model per model type: a node calling gpt-6-luna as an LLM and as a judgment model has two entries. Entries are ordered most-used first, ties broken by name and then by model type, so a consumer can take by_model[0] as the dominant model without sorting. Per-model cost follows invariant 2: None iff no call to that model was priced.

The same breakdown rolls up: subtree_by_model tells you every model a controller's whole branch used.

Own and subtree

Every node carries both halves: its own inference, and its own plus every descendant's (subtree_*). A controller (PipeSequence, PipeBatch, PipeParallel) runs no inference itself, so its own numbers are always zero and only the subtree half says anything — read a controller through its own half and it reports nothing at all. Both tokens and cost roll up, so a consumer that shows either one has the branch figure available without walking the graph.

The rollup is computed once, in the assembler, for the same reason AggregatedCosts computes its totals once: several consumers read the same GraphSpec, and they must not each re-derive it and disagree.

  seq_node       calls 0 (0 rated)   subtree 3 calls (2 rated)  subtree_cost 0.0071  subtree_tokens 4210
    ├── llm_a    calls 1 (1 rated)   cost 0.0043   tokens 2130
    ├── llm_b    calls 1 (1 rated)   cost 0.0028   tokens 2080
    └── func_c   calls 0 (0 rated)   cost None     tokens 0      ← ran, made no inference

func_c has a spec present with cost=None and inference_calls=0 — invariants 1 and 2 together say "collected, ran nothing", never "unknown".

roll_up tolerates two malformed-parentage cases rather than blowing up, because a partial cross-worker event read can produce either: a parent_node_id naming an absent node makes the child a root (never a KeyError), and a cycle in the parentage chain is broken by an in-progress guard and logged (never an infinite walk). The walk is memoized and uses an explicit stack, so it is O(nodes) and independent of the interpreter's recursion limit.

The run total and the unattributed bucket

GraphSpec.usage is a GraphUsageSpec with two NodeUsageSpec fields:

• total — every usage the run reported, attributed or not. This is the graph's comparand for the cost report's own total.
• unattributed — the part that named no live node: the UNATTRIBUTED_NODE_ID ("unknown") fallback stamped when an inference runs outside any pipe context, or a node_id whose PipeStartEvent never made it into the stream. Surfaced as its own bucket rather than dropped, so the graph's total can never silently disagree with the cost report's.

On a healthy run unattributed is zero. A non-zero one is a real signal worth looking at. Its usual cause on the hosted runtime used to be a start event lost to the DynamoDB events table's item limit, an event carrying inputs or a prompt over 400 KB, whose node then never existed while its inference still counted. The DynamoDB event log now writes such an event with its bulky values left out (see Event-Log Transport and the Scoped Override), so that cause is no longer expected.

An incomplete graph is reported to the operator's error trackers. When usage names a node whose start event never made it into the stream, the assembler logs a warning for each such node and returns those nodes and their usage beside the graph (GraphSpecAssembler.assemble_with_never_started_usage); it reports nothing itself, so assembling a graph has no effect beyond its log lines. The run's tracing assembly (assemble_tracing in pipelex/pipe_run/tracing_assembly.py) reports the run through the error tracker seam as a handled problem under the code pipelex.run.graph_incomplete (pipelex/tracing/unattributed_usage_report.py), but only when its caller asks for it with reports_incomplete_graph, because whether a run reports is the run's own policy. PipeRun, which runs a pipe in a single process, asks for it under the policy it applies to a run's failure: a dry run never reports, nor do the runs a validation makes (reports_errors=False). It is the only caller in the runtime that asks, so an in-process run reports and nothing else does by itself. A durable-execution host, which assembles a run's tracing in an activity of its own, reports only when that activity passes reports_incomplete_graph=True from the run's mode; that is the contract a host takes on, and a host that does not pass it reports nothing. The dry-run rule is held by the assembly itself: a dry run's graph is never reported even when a caller asks, since its events state its mode, so a host that passes the flag still holds that rule without knowing whether the run is a validation's, a validation's runs being dry runs. A caller that leaves reports_incomplete_graph off, the default, reports nothing, which code that must stay free of side effects, such as a durable workflow's own code, relies on. Every call that asks reports again: a host that passes the flag and retries its tracing activity sends one report per attempt that reaches the end of the assembly, which the trackers group under the code as further events of the same issue; nothing deduplicates them across attempts. The report names the run (the graph's id is its run's id) and the request bound on the log context, the ids of the nodes that never started (at most twenty, with their count), the inference calls they made and their cost in millionths of a US dollar, left out when it is unpriced or too large to scale. It carries no content of the run: no input, output, prompt, pipe or model. Its domain is runtime and its status 500, because the runtime's own record failed to hold what it ran, which is a fault for whoever runs the runtime to look into rather than the caller's, so the trackers admit it. With no tracker installed nothing is built and the warning is the only effect, and a report that cannot be built, from the usage total and its cost conversion to the problem's validation, is logged at debug level and dropped, never costing the run its graph. The report carries no exception, and the built-in posthog tracker sends only reports that carry one, so it reaches a tracker that takes problem reports, such as an out-of-tree Sentry tracker, and not PostHog.

Two other sources of unattributed usage are not reported, because neither leaves the graph incomplete: the usage of a superseded attempt, which the graph leaves out on purpose, and the UNATTRIBUTED_NODE_ID fallback, which never named a node.

Cross-checking against the cost report

The two are computed from the same events but encode "unrated" differently: CostRegistry.aggregate_costs treats an unrated usage as zero-cost, while NodeUsageSpec.cost is None. Comparing them naively manufactures a disagreement. The correct check is:

• graph.usage.total.cost equals the cost report's total when total.rated_inference_calls == total.inference_calls;
• when they differ, the graph total is a lower bound and the report's total is the wrong comparand — compare rated_inference_calls against the report's row count instead.

The in-process tracer does not carry usage

There are two GraphSpec builders. GraphSpecAssembler produces every consumed spec. The in-process GraphTracer.teardown() output is discarded at all three of its call sites (runner.py, pipe_run.py, dry_run_in_process.py — each reads pipe_output.graph_spec, i.e. the assembler's), and GraphTracerProtocol has no method to receive usage at all.

So usage is deliberately assembler-only. That gap is asserted rather than normalized away: tests/unit/pipelex/tracing/test_assembler_equivalence.py runs a usage-bearing scenario through both builders, asserts they stay structurally equivalent, and asserts the divergence explicitly (assembler populated, tracer None). If someone later plumbs usage into the tracer, that test fails and tells them to delete the assertion — which is what a test documenting an intentional gap should do.

Files reference

Related pages

• Execution Graph Tracing — the GraphSpec model and the trace event streams.
• TokensUsage Wire Records — the client-facing shape of the other projection.
• Cost Tracking & Reporting — the console table and CSV export.