Skip to content

Transient pipelines

A transient pipeline lets an agent compose a multi-step, cross-resource operation that the on-prem sidecar runs in-process - data flows step-to-step without returning to the model. Two gateway meta-tools drive it:

  • pipeline_discover (read-only) - lists the resources, operations, and transforms the caller may use, with input/output types, plus a conventions block for field mapping and JavaScript authoring.
  • pipeline_execute (write) - compiles and runs an ordered list of steps. Step 0 is the source (it iterates records); each later step's operation runs per incoming record, with an optional transform shaping that step's output. dryRun validates without executing.

Guarantees

  • Composition, not a catch-all. Every step names an existing, individually-annotated operation that is also exposed as its own read/write tool. pipeline_execute takes no HTTP method or endpoint, so a caller cannot construct arbitrary requests.
  • Per-step authorization. Each step is authorized against its own resource's permissions; connector credentials are supplied server-side, never in the spec.
  • In-process. Records never return to the model between steps - end with a Record-to-CSV transform to receive output.

A worked example

The task: "For every S2 person in the Contractors access group, look up their Salesforce contact and export a CSV of name, email, and contract end date."

The agent first calls pipeline_discover to get the exact resource, operation, and transform names (and their input/output types) it is permitted to use, then submits a spec to pipeline_execute:

{
  "steps": [
    {
      "resource": "S2",
      "operation": "Search Person Data",
      "operationSettings": { "UDF1": "Contractors" },
      "transform": "S2PersonToRecord"
    },
    {
      "resource": "Salesforce",
      "operation": "Get Contacts",
      "operationSettings": { "Filter": "Email = {{email}}" },
      "transform": "ContactToRecord"
    },
    {
      "transform": "RecordSetToCSVStringSet",
      "resource": "Salesforce",
      "operation": "Get Contacts"
    }
  ],
  "dryRun": true
}
  • Step 0 is the source. Its operation iterates records (each S2 Person); its transform S2PersonToRecord turns each into a generic Record.
  • Each later step runs once per incoming record. Get Contacts receives the Record, looks up the contact, and ContactToRecord shapes the result back to a Record.
  • A step's effective output is its transform's output. So the pipeline's inter-step types are Record → Record, and a Record-to-CSV terminal transform (RecordToCSVString / RecordSetToCSVStringSet) renders the accumulated records into the returned CSV file. (Terminate in a PDF transform instead to receive base64 PDF.)
  • operationSettings are operation parameters only - never connector credentials. JS field mapping (e.g. TransformJS / JSFilter) goes in transformSettings.

Iterate against the type-checker with dryRun

Set dryRun: true to resolve names, run the per-step authorization gate, and type-check the whole chain without executing anything. The response echoes the compiled, typed plan. Only when it validates should the agent resubmit with dryRun: false. For any write pipeline, the dry run is a safety rail, not a nicety.

Type-checking rules

The sidecar refuses to run a pipe that does not type-check - this is what makes a composed pipeline safer than an untyped script, not more dangerous:

  • Between steps, outputType(step N) must be assignable to inputType(operation N+1), using the InputType / OutputType declared on each operation/transform via [Property(...)].
  • Within a step, the operation's output must feed the transform's input; the step's output is then the transform's output.
  • object is universal. A type declared as object (for example the Identity pass-through transform, or RecordToRecord) is assignable to and from anything, so it bridges any two steps.
  • Record is the common currency. Any …ToRecord output feeds any RecordTo… input, which is how cross-system steps connect. See Transforms.
  • The check is lenient only where a type is undeclared - a connector or transform that omits its [Property] type is treated permissively rather than blocking the pipe.

Credential binding and per-step authorization

Every step carries the same execution-plane guarantees as a single tool call:

  • The spec carries operation/transform parameters only, never connector settings. The sidecar fills each step's connector credentials from its own Resources database.
  • Each step uses its own resource's authentication. A cross-system pipe orchestrates independently-authenticated stages, so "the Salesforce connector has no Google credentials" is a non-issue - the sidecar binds each stage's secret locally.
  • Reserved identity keys are injected, not accepted. The caller's identity (__Caller*) is injected from the authenticated transport for resources that opted into caller-identity, and any __-prefixed key supplied in the spec is stripped.
  • Every step is re-gated against its resource's McpGroups for the calling user, fail-closed - the same gate as the standalone tool.
  • Because the plan never carries upstream credentials, the cloud relay can transport a pipeline it can never itself read; all binding happens on the on-prem sidecar.

Result, limits, and errors

  • Result shape. pipeline_execute returns a JSON document: the terminal CSVFile (CSV rows, or base64 for a PDF terminal) plus a run summary - step count, recordsProcessed, truncated, and row count. Bulk records never stream back to the model; only the terminal artifact and summary do.
  • Run caps. The record count is capped by the gateway's MaxIterableResults; when the source produces more, the run completes with truncated: true.
  • Failure semantics. On a step failure the run aborts and the error reports which step failed (building step N vs. during execution). Combined with dryRun, this lets the agent fix a spec before any write occurs.
  • Auditing. Each step writes a pipeline_step audit record (resource/operation, executed or denied with reason) and the run writes a pipeline_run record (ok / rejected / error, with record count and truncation).

v1 limitations

A step whose resource requires a minted caller token (McpCallerTokenAudience) is rejected inside a pipeline (calling that operation as a standalone tool still works). Type-checking is lenient where a connector or transform omits its declared type.

Long-running runs

pipeline_execute runs synchronously and returns when the pipe completes. For a long-running integration invoked by handle, two tools manage it out-of-band:

  • get_integration_run (read) - poll a run's status/summary by its run handle.
  • stop_integration_run (write) - request cancellation of an in-flight run.