Collection Pipelines

A Collection Pipeline maps an agent over a list. It pulls records from a list-style tool — rows in a Google Sheet, issues from a tracker, contacts from a CRM — and fires one isolated agent run per record, then collects the results into a sink. It's the batch counterpart to a single agent run: instead of one prompt, one conversation, one answer, you get the same agent applied independently to hundreds of items, each with its own trace, cost, and retry boundary.

Reach for a pipeline when you want to enrich a catalog, triage a backlog, classify a dataset, or build a knowledge index from a tool — anything shaped like "do this for each of these."

The mental model

 Source ──▶ Agent ──▶ Sink
 (a list   (one run   (where the
  tool)     per item)  results go)

A pipeline is a saved definition. Each time you run it — manually, as a dry run, or on a schedule — it produces a run, and each run fans out into per-item executions you can monitor.

Creating a pipeline

Open Pipelines from the sidebar and click New pipeline. The builder is a four-step wizard: Data source → Agent → Sink → Run policy.

1. Data source

The source is a paged list tool on one of your integration configs.

The source is fetched headlessly — no LLM, no agent — so listing a large collection is cheap. The walker follows pagination until the source is exhausted.

2. Agent

Pick the per-item agent — the agent that runs once for each record — and, optionally, pin a revision (leave blank to always use the agent's current revision at run time).

Then map the record into the agent's inputs:

3. Sink

The sink decides what to do with each item's result.

Index asks for the target Index asset id.

Write back asks for the target sheet: Integration config id, Spreadsheet id, Sheet name, Output column (e.g. K), and First data row (1-based, default 2). Each result is written to the cell in that column at the row matching the record's read order. Failed or skipped rows are left untouched — the write-back never blanks a cell it has no answer for.

4. Run policy

The run policy is where you set the rails. Sensible defaults keep a first run safe.

Dry run before you commit

Every pipeline has a Dry run action that runs the agent over just the first few items (5 by default) without committing the sink's side effects beyond what those few items produce. Use it to confirm your field mappings, prompt, and sink target are right before you fan out across the whole collection. A dry run still shows up in the run monitor, marked as a preview.

When the mapping looks right, hit Run for a full production run.

Monitoring a run

Opening a run shows a live monitor while it's in flight:

• Counter cards — Total, Succeeded, Failed, Skipped, In flight, and Cost (USD).
• Items table — one row per item with its key, status (Pending → Running → Completed / Skipped / Failed), and a View trace link straight to that item's full agent execution.

A run itself moves through Fetching → Fanning out → Running → Completed (or Failed, or Halted when a budget cap or halt-on-error rule tripped). The Runs history lists every run of the pipeline with its origin (production, dry run, or scheduled), status, and per-status counts.

Cross-run deduplication

For the Act (none) sink — the one with real, non-idempotent side effects — Fruxon keeps a cross-run ledger of items already processed successfully. On the next run, items the pipeline has already acted on are skipped, so a record never gets a second ticket or a duplicate message. The dedup key is the record's Item id path.

The other sinks don't need this: Collect and Index are safe to re-run (Index fully replaces the snapshot), and Write back overwrites the same cell by position, so re-running just refreshes the value.

Running on a schedule

To run a pipeline unattended, wire a Schedule trigger to it instead of to an agent. The trigger fires the pipeline on its cadence — the same frequency / interval / timezone machinery as scheduled agent runs. Fruxon won't start a new scheduled run while the pipeline's previous run is still in flight, so a slow run can't pile up on itself. Scheduled runs only fire while the pipeline is Active.

See Agent Network → Triggers for how schedule triggers are set up.

API

Collection pipelines are managed through tenant-scoped REST endpoints. The token scopes are pipelines:read, pipelines:write, and pipelines:run.

See the API Reference for full request and response shapes.

Next steps

• Knowledge Base & Assets — the index a pipeline can write into, and tool-source ingestion
• Agent Network — schedule triggers for unattended runs
• Observability — every per-item run is a full trace
• Cost & Budgets — how per-run budget caps relate to agent budgets