Skip to content

Refining Concepts

Concept refinement allows you to create more specific versions of existing concepts while inheriting their structure. This provides semantic clarity and type safety for domain-specific methods.

What is Concept Refinement?

Refinement is the process of creating a specialized concept from a more general one. When you refine a concept, the new concept:

  • Inherits the structure of the base concept
  • Adds semantic specificity to clarify its purpose

Think of it as creating a subtype: an Invoice is a specific kind of Document, and a Photo is a specific kind of Image.

Why Refine Concepts?

1. Semantic Clarity

Refined concepts make your pipeline's intent explicit:

# ❌ Less clear
[pipe.process_invoice]
inputs = { invoice = "Document" }

# ✅ More clear
[pipe.process_invoice]
inputs = { invoice = "Invoice" }

2. Self-Documenting Code

[pipe.extract_contract_terms]
type = "PipeLLM"
description = "Extract key terms from a contract"
inputs = { contract = "Contract" }  # Clear what type of document is expected
output = "ContractTerms"

3. Domain-Specific Methods

Build pipelines tailored to specific use cases:

domain = "finance"

[concept.Invoice]
description = "A commercial invoice"
refines = "Document"

[concept.Receipt]
description = "Proof of payment"
refines = "Document"

[pipe.process_invoice]
type = "PipeLLM"
inputs = { invoice = "Invoice" }
output = "InvoiceData"
# ... invoice-specific processing

[pipe.process_receipt]
type = "PipeLLM"
inputs = { receipt = "Receipt" }
output = "ReceiptData"
# ... receipt-specific processing

4. Type Validation

Using specific concept names helps catch errors early:

[pipe.analyze_invoice]
inputs = { invoice = "Invoice" }  # Only accepts Invoice
output = "Analysis"

Current Limitations

No structure on refined concepts (For Now)

When you refine a concept, you cannot add an inline structure or specify a structure_class_name. This limitation will be lifted in future releases.

Not allowed:

[concept.Invoice]
description = "A commercial invoice"
refines = "Document"
structure_class_name = "InvoiceModel"  # ❌ Not allowed

[concept.Invoice.structure]  # ❌ Not allowed
invoice_number = "Invoice ID"

Allowed: 

[concept.Invoice]
description = "A commercial invoice"
refines = "Document"  # ✅ Inherits DocumentContent structure

Basic Refinement Syntax

Define a refined concept using the refines field:

[concept.ConceptName]
description = "Description of the refined concept"
refines = "BaseConceptName"

For concepts in a different domain, use the fully qualified reference:

[concept.SpecializedConcept]
description = "A more specialized version of an existing concept"
refines = "otherdomain.BaseConceptName"

Refining Document

[concept.Invoice]
description = "A commercial document issued by a seller to a buyer"
refines = "Document"

[concept.Contract]
description = "A legally binding agreement between parties"
refines = "Document"

Both concepts inherit the DocumentContent structure (with a url field) but represent semantically distinct document types.

Refining Image

[concept.ProductPhoto]
description = "A photograph of a product for marketing purposes"
refines = "Image"

[concept.Screenshot]
description = "A screen capture image"
refines = "Image"

Each inherits ImageContent structure (url, caption, base_64, etc.) with specific semantic meaning.

Refining Text

[concept.Article]
description = "A written composition on a specific topic"
refines = "Text"

[concept.Summary]
description = "A condensed version of a longer text"
refines = "Text"

Building Concept Hierarchies

You can build hierarchies by refining concepts that have their own structures. The refined concept inherits the structure from the base concept:

domain = "crm"

# Base concept with structure
[concept.Customer]
description = "A customer in our system"

[concept.Customer.structure]
name = { type = "text", required = true, description = "Customer name" }
email = { type = "text", required = true, description = "Customer email" }

# Refined concept - inherits Customer's structure
[concept.VIPCustomer]
description = "A VIP customer with special privileges"
refines = "Customer"

# Another refined concept from the same base
[concept.InactiveCustomer]
description = "A customer who has not been active recently"
refines = "Customer"

Both VIPCustomer and InactiveCustomer will have access to the name and email fields defined in Customer. When you create content for these concepts, it will be compatible with the base Customer structure.

Cross-Package Refinement

You can refine concepts that live in a different package. This lets you specialize a shared concept from a dependency without modifying the dependency itself.

Syntax

Use the -> cross-package reference operator in the refines field:

[concept.RefinedConcept]
description = "A more specialized version of a cross-package concept"
refines = "alias->domain.BaseConceptCode"
Part Description
alias The dependency alias declared in your METHODS.toml [dependencies] section
-> Cross-package reference operator
domain The dot-separated domain path inside the dependency package
BaseConceptCode The PascalCase concept code to refine

Full Example

Suppose you depend on a scoring library that defines a WeightedScore concept:

Dependency package (scoring-lib):

METHODS.toml
[package]
address = "github.com/acme/scoring-lib"
version = "2.0.0"
description = "Scoring utilities."

[exports.scoring]
pipes = ["compute_weighted_score"]
scoring.mthds
domain = "scoring"

[concept.WeightedScore]
description = "A weighted score result"

[pipe.compute_weighted_score]
type = "PipeLLM"
description = "Compute a weighted score"
output = "WeightedScore"
prompt = "Compute a weighted score for: {{ item }}"

Your consumer package:

METHODS.toml
[package]
address = "github.com/acme/analysis-app"
version = "1.0.0"
description = "Analysis application."

[dependencies]
scoring_lib = { address = "github.com/acme/scoring-lib", version = "^2.0.0" }

[exports.analysis]
pipes = ["compute_detailed_score"]
analysis.mthds
domain = "analysis"

[concept.DetailedScore]
description = "An extended score with additional detail"
refines = "scoring_lib->scoring.WeightedScore"

[pipe.compute_detailed_score]
type = "PipeLLM"
description = "Compute a detailed score"
output = "DetailedScore"
prompt = "Compute a detailed score for: {{ item }}"

DetailedScore inherits the structure of WeightedScore from the scoring_lib dependency's scoring domain.

Important

The base concept must be accessible from the dependency. The dependency must export the pipes in the domain that contains the concept, or the concept's domain must be reachable via an exported pipe's bundle.

For more on how dependencies and cross-package references work, see Packages.

Type Compatibility

Understanding how refined concepts interact with pipe inputs is crucial.

How Refinement Affects Type Checking

Key Rule

A pipe that accepts a base concept will also accept any concept that refines it. Refined concepts are substitutable for their parent — this is standard subtype compatibility.

[pipe.extract_text]
inputs = { document = "Document" }  # Accepts Document, Invoice, Contract, or any concept that refines Document

Practical Example

[concept.Invoice]
refines = "Document"

[concept.Contract]
refines = "Document"

# This pipe accepts Document and any concept that refines it (Invoice, Contract, etc.)
[pipe.extract_from_document]
type = "PipeExtract"
inputs = { document = "Document" }
output = "Page[]"

# This pipe accepts Invoice (and any concept that refines Invoice)
[pipe.process_invoice]
type = "PipeLLM"
inputs = { invoice = "Invoice" }
output = "InvoiceData"

# This pipe accepts Contract (and any concept that refines Contract)
[pipe.process_contract]
type = "PipeLLM"
inputs = { contract = "Contract" }
output = "ContractData"

In this setup:

  • extract_from_document accepts Document, Invoice, Contract, or any other concept that refines Document
  • process_invoice accepts Invoice or any concept that refines Invoice
  • process_contract accepts Contract or any concept that refines Contract

Best Practices

1. Choose Meaningful Names

# ❌ Avoid generic or vague names
[concept.Document1]
refines = "Document"

# ✅ Use specific, descriptive names
[concept.Invoice]
refines = "Document"

2. Write Clear Descriptions

# ❌ Too vague
[concept.Invoice]
description = "A document"
refines = "Document"

# ✅ Clear and specific
[concept.Invoice]
description = "A commercial document issued by a seller to a buyer, detailing products or services provided and payment terms"
refines = "Document"

3. Don't Over-Refine

# ❌ Too specific, creates unnecessary complexity
[concept.SmallInvoice]
description = "An invoice under $100"
refines = "Document"

[concept.LargeInvoice]
description = "An invoice over $1000"
refines = "Document"

# ✅ Keep it general, handle specifics in processing logic
[concept.Invoice]
description = "A commercial invoice"
refines = "Document"

When to Refine vs. When to Create New Concepts

Refine When:

  • ✅ Your concept is semantically a specific type of an existing concept
  • ✅ The base concept's structure is sufficient for your needs
  • ✅ You want to inherit existing validation and behavior
  • ✅ You're building domain-specific methods with clear document/content types
  • ✅ You need to create specialized versions of an existing concept

Examples: 

[concept.Invoice]  # Clearly a type of Document
refines = "Document"

[concept.VIPCustomer]  # A specialized type of Customer
refines = "Customer"

Create New Concept When:

  • ✅ Your concept needs custom structure with multiple fields
  • ✅ Your concept doesn't naturally fit any existing concept
  • ✅ You need complex validation or computed properties