> ## Documentation Index
> Fetch the complete documentation index at: https://www.integrate.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# ETL: Building & Editing Packages with MCP Tools
> Use the Integrate.io MCP server to let Claude, Cursor, or the MCP Inspector author, edit, and clean up ETL packages without opening the web UI.
The Integrate.io MCP server exposes a set of package-authoring tools that AI agents can call to build a working ETL package end to end. Supported clients include Claude Desktop, Claude Code, Cursor, and the MCP Inspector. The agent discovers what the catalog supports, composes a dataflow JSON, creates the package, fixes wiring errors in place, and runs it. No web UI required.
These tools complement the read and run tools described in the [MCP Server overview](/etl/integrateio-mcp-server). Configure your client and mint a token there first.
## When to use these tools
* You want an AI agent to build a new source-to-destination package in a single tool call from a natural-language prompt.
* You want the agent to clone an existing pipeline as a template, then swap in different connections, tables, or paths.
* You want the agent to add, edit, or remove components on an existing pipeline without opening the package designer.
* You want the agent to rename a package or define its public variables in place.
* You want the agent to fix a broken edge or component without opening the package designer.
* You want the agent to detect the columns of a CSV or other delimited file on an SFTP, S3, or GCS connection before wiring it into a flow.
* You want the agent to archive packages that failed an authoring attempt so they don't clutter the active list.
If you only need read or run access, stick to the inspection and execution tools on the [overview page](/etl/integrateio-mcp-server).
## Tool reference
### build\_pipeline (mutation)
Builds a complete source → destination pipeline in one call from a high-level intent. The server discovers the source schema, picks the matching component types for each connection, wires `source → select → destination` with edges, maps fields 1:1, and persists the package. The agent only passes intent (which connections, which table or file path, which fields, any computed columns), so the build turn stays fast even on wide schemas.
Prefer `build_pipeline` over `create_package` for a straightforward "load X into Y" build. Use `create_package` or `add_package_components` when the shape isn't covered here: filters, joins, multiple sources, a file or cloud-storage destination, or edits to an existing package.
Package name shown in the dashboard.
Source intent. See below.
Destination intent. Must be a database or warehouse connection (for example Snowflake, Redshift, BigQuery, MySQL, Postgres). File and cloud-storage destinations are not yet supported here, use `create_package` for those.
`"all"` (default) carries every discovered source column. An array of source column names keeps only that subset.
Derived columns to append. Each entry: `{ "name": "
`dataflow` (default) or `workflow`.
Must belong to the calling account. Omit to create the package as **Unassigned**.
Free-form, up to 4096 characters.
`source` accepts:
* `connection_id` (integer, required).
* `path` (string) for file or cloud-storage sources (SFTP, S3, GCS, SharePoint).
* `table_name` (string) for database or warehouse sources.
* `schema_name` (string), optional, database sources only.
* `delimiter` (string), file field delimiter, default `,`.
* `header_row` (boolean), whether the file has a header row, default `true`.
* `source_path_field_alias` (string), optional, file and cloud-storage sources only. When set, adds an output column with this exact name carrying the source file path each row was read from. Use this when the agent wants the filepath as a column for lineage. Never hardcode the literal path as a field value.
`destination` accepts:
* `connection_id` (integer, required).
* `table_name` (string, required).
* `schema_name` (string), optional.
* `operation_type` (string), write mode (for example `append`, `truncate_and_insert`, `merge`). Defaults to `append`. Validated against the destination's allowed `write_modes` from `describe_component_type`.
* `create_table` (boolean), auto-create the table, default `true`.
* `merge_keys` (array of string), required for merge and upsert modes (`merge`, `merge_update_and_insert`, `insert_or_update`). Each name flags the matching destination column with `is_merge_key: true`. The platform never infers the key, so a merge without `merge_keys` will not upsert correctly. Append, overwrite, and truncate modes ignore this field.
Field names from the source are carried through exactly as discovered, no renames, recasing, or added prefixes.
The tool returns the new `package_id` and also runs `validate_package` automatically so the caller knows whether the build is clean on the first response.
#### Example: SFTP to Snowflake with a merge key and filepath column
```json theme={null}
{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/call",
"params": {
"name": "build_pipeline",
"arguments": {
"name": "Daily orders load",
"source": {
"connection_id": 47376,
"path": "/incoming/orders/*.csv",
"source_path_field_alias": "source_file"
},
"destination": {
"connection_id": 91002,
"table_name": "orders",
"schema_name": "public",
"operation_type": "merge",
"merge_keys": ["order_id"]
},
"computed": [
{ "name": "loaded_at", "expression": "CurrentTime()" }
]
}
}
}
```
#### Error envelopes
| Envelope | Cause |
| ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `source connection not found in this account` | Source `connection_id` is unknown to this account. |
| `destination connection not found in this account` | Destination `connection_id` is unknown to this account. |
| `build_pipeline currently supports database/warehouse destinations only; ...` | Destination is a file or cloud-storage connection. Use `create_package` instead. |
| `source.path is required for a file/cloud-storage source` | Missing `path` on a file source. |
| `source.table_name is required for a database source` | Missing `table_name` on a database source. |
| `no columns were discovered on the source — check the path/table and delimiter` | The schema importer returned no fields. Verify the path, table, or delimiter. |
| `operation_type '' is not valid for this destination; valid: ...` | Write mode isn't supported by this destination. The envelope lists the valid options. |
### create\_package (mutation)
Mints a new package from a full `data_flow_json` spec. Components and edges are persisted in a single transaction, partial packages are never written.
Package name shown in the dashboard.
`dataflow` or `workflow`.
Array of single-key wrapper hashes (see below).
Array of edge hashes referencing component `id`s.
Must belong to the calling account. Omit to create the package as **Unassigned**.
Free-form, up to 4096 characters.
Defaults to `2.0.0` (the modern package designer).
`variables` is not accepted. Packages that need package-level variables must be created through the REST API.
#### Component shape
Each component is a single-key wrapper hash. The key is the wrapper type (for example `database_source_component`), and the inner hash carries the configuration. Common fields:
* `id`, stable identifier referenced by edges. Convention is `component-`. Omit to let the server generate one.
* `name`, human-readable label shown on the canvas.
* `xy`, `[x, y]` position pair.
* `alias`, data-flow alias. Sources expose this; destinations consume it via `input_alias`.
* `connection`, connection metadata block. Uses the generic key `connection`, not type-specific keys. Shape: `{ "id": , "name": "", "type": "" }`. Without `name` and `type`, the dashboard can't render the connection chip. When `update_package_components` changes a `*_connection_id`, the server clears this block and re-embeds it from the new connection so the dashboard chip stays in sync with the connection the job will run.
* `_connection_id`, string form of the connection id (for example `"cloud_storage_connection_id": "47376"`).
* `specificComponentType`, vendor sub-type (for example `sftp_source_component`, `mysql_source_component`). Required when the wrapper covers multiple vendors so the dashboard renders the correct icon and form.
* `schema`, nested object `{ "fields": [{ name, alias, data_type }, ...] }`. Each field needs an `alias`.
#### Edge shape
Edges reference components by their `id` field:
```json theme={null}
{
"id": "edge-abc123",
"label": "edge-abc123",
"source": "component-aaa111",
"target": "component-bbb222",
"source_index": 1,
"order": 1
}
```
`id` and `label` are optional, the server generates them when absent.
#### Best practice: clone from an existing package
The safest way to compose a valid `data_flow_json` is to read a similar existing package first:
1. Call `list_packages` and find a pipeline with the same source and destination connection types.
2. Call `get_package(, include_full_graph: true)` and inspect its `data_flow_json`.
3. Use the result as a template. Swap in your own connection ids and table or file paths. Keep the structural keys.
Inline secrets in the source package (REST `api_key` query params, `Authorization` headers, basic-auth passwords, and similar) come back as `[REDACTED]` in the full graph. Structure and non-secret fields are preserved, so the result is still a valid template, refer real credentials through a connection or a secret variable when you re-author the package. See [Inline secret redaction in get\_package](/etl/integrateio-mcp-server#inline-secret-redaction-in-get_package).
Building from `describe_component_type` output alone is error-prone, it surfaces `attr_accessor` property names but can't expose nested sub-shapes (like `schema.fields`) or renderer conventions like `specificComponentType`. Pair it with a working template whenever possible.
#### Returns
On success: `{ package_id, package_version, name, flow_type, flow_version, workspace_id, created_at, warnings }`. The `warnings` array lists shape fixups the server applied (for example rewriting a vendor-named wrapper to the canonical wrapper plus `specificComponentType`).
On failure: `{ error: "..." }`, workspace not in account, name too long, save failure, and so on.
### clone\_package (mutation)
**Mutation.** Copies an existing package into a NEW package to use as a template, without editing the original. The clone copies the source's `data_flow_json`, variables, description, and flow type, is attributed to the calling user, and gets a new `package_id`. For workflow packages, sub-package references inside `data_flow_json` are preserved as-is, not recursively deep-cloned.
Use `clone_package` to START a new package from an existing one, not to edit the original. If the user wants to add, edit, or rewire components on a package that already exists, use `add_package_components`, `update_package_components`, or `update_package_edges`. Cloning (or clone-then-archive) to make an edit mints a new `package_id` and orphans the original's schedules, run history, and API references.
Package to copy.
Name for the new package (up to 128 characters).
Workspace to place the clone in. Must belong to the calling account. Defaults to the source package's workspace.
Description for the new package, up to 1024 characters. Defaults to the source package's description.
Return what would be created without writing.
Returns `{ new_package_id, source_package_id, name, flow_type, workspace_id, owner_id, created_at }` on success.
### add\_package\_components (mutation)
Appends one or more NEW components (and, optionally, new edges) to an existing package's flow. This is the "extend a package the user already has open" operation, for example "add a Select component" or "insert a Filter between the source and destination." It fills the gap between `create_package` (builds a whole package) and `update_package_components` (only edits components that already exist).
Do not recreate a package, or clone-then-archive it, to add to it. That mints a new `package_id` and orphans the original's schedules, run history, API references, and the user's open editor session. `add_package_components` edits the package in place.
Package to extend.
One or more single-key wrapper hashes, the same shape `create_package` uses. See [Component shape](#component-shape).
New edges wiring the added components in. Omit to add components unwired, then call `update_package_edges`.
Each component follows the same wrapper shape as `create_package`. Two placement details are specific to adding:
* `name` must be unique within the package, it is how `update_package_components` later targets the component.
* `xy` is optional when the component is wired to an existing one: the server places a newly-added component just below its parent and nudges the downstream chain down to make room. Provide an explicit `xy` only for a standalone component with no incoming edge (for example a new source).
Carry field names through exactly as `discover_schema`, `discover_file_schema`, or `preview_data` returned them, same spelling and case. Never add an `f_` prefix or rename a field; the platform makes Pig aliases valid and reserved-word-safe automatically.
The call is all-or-nothing and row-locked: if any new component's `name` or `id` collides with one already in the flow, nothing is written. Each successful call bumps the package version via PaperTrail and is reversible from the version history UI.
Returns `{ package_id, components_added, edges_added, new_version, updated_at }` on success, plus a `warnings` array listing any wrapper-shape fixups the server applied.
### update\_package\_components (mutation)
**Mutation.** Edits specific components that ALREADY EXIST in a package's `data_flow_json`. Each entry is `{ component_name, updates }`, where `component_name` matches the inner `name` field on a component and `updates` is a hash of fields to merge onto it. Untouched top-level fields are preserved, and nested hashes are merged recursively (deep merge), so a partial config update does not drop sibling keys. To ADD a new component use `add_package_components`, to REMOVE one use `remove_package_components`, and to re-wire edges use `update_package_edges`.
Package to update.
At least one entry. Each entry is `{ component_name, updates }`: `component_name` matches a component's inner `name`, and `updates` is a hash of fields to deep-merge onto that component. The `name` and `id` fields are protected and cannot be overwritten, since edges reference them. Do not place credentials in `updates`; connection secrets live in connection management, and putting them in `data_flow_json` leaks them through the version history.
Report which components would change without writing.
The call is all-or-nothing: if any `component_name` is not found in the flow, the entire update is rejected and nothing is persisted. When an `updates` hash changes a `*_connection_id`, the server clears the stale `connection` display block and re-embeds it from the new connection so the dashboard chip stays in sync. The package row is locked (SELECT FOR UPDATE) for the save so concurrent agent calls serialize correctly. Each successful call bumps the package version via PaperTrail and is reversible from the version history UI.
Returns `{ package_id, components_updated, new_version, updated_at }` on success.
### list\_component\_types (read)
Enumerates every component registered in the platform catalog. Returns one entry per concrete subclass with:
* `name`, internal name (for example `mysql_source_component`). Pass this to `describe_component_type`.
* `category`, `source`, `transformation`, or `destination`.
* `class_name`, Ruby class name, informational.
* `description`, header docstring from the component's source file.
* `wrapper_key`, the outer key to use in `data_flow_json`. May be `null` for transformations or unmapped classes.
* `specific_component_type`, value for the inner `specificComponentType` field. May be `null` when the vendor has its own top-level wrapper.
Optional `category` argument filters the result.
Use this when the account has no similar package to clone from. For mature accounts, `get_package` of an existing pipeline is often enough.
### describe\_component\_type (read)
Per-component introspection by internal name. Returns:
* `properties`, writable attribute names extracted from `attr_accessor` declarations.
* `required`, attribute names with presence validators.
* `description`, header docstring.
* `example`, fixture JSON example, or `null` if none exists.
* `wrapper_key`, `specific_component_type`, `valid_specific_types`, `wrapper_example`, copy-paste-ready wrapper recipe.
Required argument: `type` (the `name` from `list_component_types`). Returns `{ error: "unknown component type: " }` for unknown names.
### update\_package\_edges (mutation)
Replaces a package's edges array wholesale. Pass the complete desired edges array, existing edges are overwritten.
Package to update.
Full edges array. Each edge needs at minimum `source` and `target` matching a component `id` in the current package.
The tool pre-validates every `source` and `target` against the package's current components. If any edge references an unknown component, the entire call is rejected, no partial writes. Each write bumps the package version via PaperTrail and is reversible through the existing UI history.
Use this when `validate_package` surfaces an edge-related error after `create_package` and you need to fix wiring in place. For component-internal edits (changing a connection, adjusting a schema), use `update_package_components`.
### remove\_package\_components (mutation)
Removes one or more components from an existing package's flow by component `name`. Any edge whose `source` or `target` was a removed component is dropped at the same time, so the package never ends up with dangling edges.
Package to update.
Inner `name` values from the package's components. The same key `update_package_components` targets.
The call is all-or-nothing. If any name doesn't match a component currently in the flow, the entire call is rejected with the list of unknown names and nothing is removed. The package row is locked for the read-modify-write, and each successful call bumps the package version via PaperTrail and is reversible from the version history UI.
Use this to undo a mis-added component or to clean up an unused branch. To add a component, use `add_package_components`. To edit one in place, use `update_package_components`. To archive the entire package, use `delete_package`.
Returns `{ package_id, components_removed, edges_removed, new_version, updated_at }` on success.
### rename\_package (mutation)
Renames a package and, optionally, updates its description. Metadata-only. `data_flow_json` (components and edges) is left untouched.
Package to rename.
New package name (3 to 128 characters).
Free-form, up to 1024 characters.
The change is attributed to the calling user and recorded in the package version history, the same as a rename through the dashboard. Use this when an agent needs to retitle a package after a prompt-driven authoring step rather than asking the user to open the UI.
Returns `{ package_id, previous_name, name, updated_at }` on success.
### manage\_package\_variables (mutation)
Defines, updates, or removes a package's public variables. Public variables are the named defaults a package exposes; per-run overrides are still passed through `run_package`'s `variables` argument.
Package to update.
Hash of `{ variable_name => default_value }` to add or update. Values are stored as given; for Pig string literals, follow the embedded single-quote rule used by `run_package`. At least one of `set` or `remove` is required.
Variable names to delete. At least one of `set` or `remove` is required.
This is a partial update of the variables store, modeled on the dashboard's **Variables** modal. The pipeline graph is not touched. The package row is locked for the read-merge-write so a concurrent dashboard save can't drop a just-set variable.
Secret variables are intentionally not supported here. They are encrypted at rest and must be set through the dashboard so plaintext never flows through the MCP transcript. Putting a secret value in `set` would store it as a plaintext public variable.
Returns `{ package_id, variables }` (the full resulting public-variable map) on success.
File-schema detection is covered by `discover_file_schema` on the [Reading & Inspecting](/etl/mcp-tools-reading#discover_file_schema) page.
### delete\_package (mutation)
Archives a package, same semantics as the dashboard's **Archive** action. The `Job` row is preserved with status `archived`, removed from the active package list, and remains queryable via `list_packages(status: 'archived')`.
Package to archive.
If any active schedule still references the package, the archive transition is rejected. Disable the schedule first with `toggle_schedule(enabled: false)`.
Use this to clean up after an unsalvageable `create_package` attempt so failed iterations don't accumulate as dead rows.
Returns `{ package_id, status: 'archived', archived_at }` on success, or `{ error: "..." }` for package-not-found or rejected transitions.
## Version history
Every write through the package-authoring tools is versioned via PaperTrail. These two tools let an agent inspect that history and roll back a change.
### list\_package\_versions (read)
**Read-only.** Lists a package's saved version history (PaperTrail snapshots), oldest first, with the current state last. Summary only, not the full `data_flow_json` of each version. Up to 100 entries.
The package (job) id.
Each entry: `version` (the index to pass to `revert_package`), `package_version`, `description`, `author`, `created_at`, `component_count`, and `current`. To undo the most recent change, use the highest-numbered `current: false` entry (the one immediately before the current entry).
### revert\_package (mutation)
**Mutation.** Forward-only rollback to a prior version returned by `list_package_versions`. It restores the chosen version's `data_flow_json` as a NEW version, so the revert is itself undoable, and leaves the package's schedules and run history untouched.
Package to revert.
The index of a `list_package_versions` entry whose `current` is `false`. That list is oldest-first with the current state last, so to undo the most recent change pick the highest-numbered `current: false` entry (the one immediately before the current entry). Never pass the current entry.
Report what would be reverted without writing.
Returns `{ package_id, reverted_to_version, restored_from_package_version, new_package_version, updated_at }` on success.
## Recommended agent flow
```text theme={null}
list_connections # confirm source + destination connections (never auto-pick)
build_pipeline # one-call build for source -> destination (preferred)
# OR, for shapes build_pipeline doesn't cover:
discover_schema / discover_file_schema / preview_data # learn the source columns
list_packages / get_package() # find a template to clone
list_component_types / describe_component_type # only if no template exists
create_package # mint the new package
preview_transformation # sanity-check a transform's output mid-build (database sources)
validate_package # catch structural + per-component errors
update_package_components / update_package_edges / remove_package_components # fix anything validate_package flagged
rename_package / manage_package_variables # metadata + variable cleanup
delete_package # archive failed attempts
run_package # execute on an available cluster
get_run # poll until completed
```
## Example: create a minimal SFTP-to-S3 package
```json theme={null}
{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/call",
"params": {
"name": "create_package",
"arguments": {
"name": "SFTP nightly drop to S3",
"flow_type": "dataflow",
"components": [
{
"cloud_storage_source_component": {
"id": "component-sftp01",
"name": "nightly_drop",
"xy": [100, 100],
"alias": "raw",
"specificComponentType": "sftp_source_component",
"connection": { "id": 47376, "name": "[Prod] SFTP", "type": "sftp" },
"cloud_storage_connection_id": "47376",
"path": "/incoming/orders/*.csv",
"schema": {
"fields": [
{ "name": "id", "alias": "id", "data_type": "string" },
{ "name": "total", "alias": "total", "data_type": "double" }
]
}
}
},
{
"cloud_storage_destination_component": {
"id": "component-s3001",
"name": "s3_landing",
"xy": [400, 100],
"input_alias": "raw",
"specificComponentType": "s3_destination_component",
"connection": { "id": 52001, "name": "[Prod] S3", "type": "s3" },
"cloud_storage_connection_id": "52001",
"path": "s3://my-bucket/orders/"
}
}
],
"edges": [
{ "source": "component-sftp01", "target": "component-s3001" }
]
}
}
}
```
A successful response returns the new `package_id`. Pass it to `validate_package` next, fix any errors with `update_package_components` or `update_package_edges`, then run with `run_package`.
## Error envelopes
Mutating tools return a plain `{ "error": "..." }` object instead of a JSON-RPC error when the failure is expected and recoverable. Common cases:
| Tool | Envelope | Cause |
| --------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `create_package` | `workspace not found in this account` | `workspace_id` belongs to another account. |
| `create_package` | record-validation message | Name too long, invalid `flow_type`, save failure. |
| `add_package_components` | `component name(s) already exist in flow: ...` | A new component reuses a `name` already in the package. The whole call is rejected. |
| `add_package_components` | `component id(s) already exist in flow: ...` | A new component reuses an `id` already in the package. |
| `add_package_components` | `edge id already exists in flow: ...` | A supplied edge `id` collides with an existing edge. |
| `add_package_components` | `package not found` | `package_id` doesn't belong to the calling account. |
| `clone_package` | `source package not found` | `source_package_id` doesn't belong to the calling account. |
| `clone_package` | `target workspace not found in this account` | `target_workspace_id` belongs to another account. |
| `clone_package` | `could not save clone: ...` | The clone failed to persist (record-validation message). |
| `update_package_components` | `package not found` | `package_id` doesn't belong to the calling account. |
| `update_package_components` | `component(s) not found in flow: ...` | A `component_name` doesn't match any current component. The whole call is rejected, and the envelope lists the valid names plus a did-you-mean suggestion. |
| `update_package_components` | `cannot overwrite protected fields: ...` | An `updates` hash tried to change `name` or `id` (would orphan edge references). |
| `update_package_edges` | `unresolved component references: ...` | Edge references a component id that isn't in the package. |
| `delete_package` | `package not found` | `package_id` doesn't belong to the calling account. |
| `delete_package` | archive transition rejection | An active schedule still references the package. |
| `remove_package_components` | `component(s) not found in flow: ...` | A name in `component_names` doesn't match any current component. The whole call is rejected. |
| `rename_package` | record-validation message | `name` is too short, too long, or otherwise rejected by the model. |
| `manage_package_variables` | `provide \`set\` (hash) and/or \`remove\` (array), nothing to do\` | Both arguments were missing or empty. |
| `revert_package` | `package not found` | `package_id` doesn't belong to the calling account. |
| `revert_package` | `package has no prior versions to revert to` | The package has only its current version. |
| `revert_package` | version-range message | `version` is `0` (the creation event), the current version, or out of range. The envelope lists the valid range. |
| `revert_package` | `version can't be restored (no recoverable snapshot at that index)` | The chosen version can't be reified back into a package state. |
| `discover_file_schema` | `connection not found, or not a cloud-storage/SFTP connection` | The id belongs to a database connection (use `discover_schema` instead) or is from another account. |
| `discover_file_schema` | `schema-importer error: ...` | The importer couldn't read the file (auth failure, missing file, malformed delimiter). Fix the connection or arguments and retry. |
| `describe_component_type` | `unknown component type: ` | `type` argument doesn't match any registered component. |
Unrecoverable failures (auth, malformed JSON-RPC) return standard JSON-RPC error responses, see the [MCP Server overview](/etl/integrateio-mcp-server) for HTTP-level errors.
## Audit trail
Every write performed through these tools is captured in your account's audit history via PaperTrail. Package creates, edge updates, and archive transitions all record the calling user as the author. Component edits made by `update_package_components` are versioned and reversible through the existing package version history UI.