ETL: AI Transformation

Use the AI Transformation to apply LLM-powered data processing to fields in your Integrate.io ETL pipeline. Supports custom prompts and models.

The AI Transformation component applies LLM-powered transformations to fields in your Integrate.io ETL pipeline. It operates per record (one request per configured field/record), supports prompt interpolation, and emits typed outputs you define.

Overview

• Currently supports OpenAI Chat Completions models. Additional providers can be added on request.
• Accepts rows from an upstream component; outputs the responses to the prompts you configure.
• Prompts support #{field_name} interpolation from upstream columns.
• If field_name is not provided, it assumes your prompt is referring to the field for the row your prompt is on.
• Component alias is always unique.

LLM API Key Management Options

Two approaches:

1. Bring Your Own Key (BYO Key). Supply your own OpenAI API key. All usage billed through your provider account.
2. Integrate.io Managed Key. Use an Integrate.io managed key. Generates a unique credential for your account. Usage processed through this managed key.

Configuration

Model Settings

• Model selection: Choose a Chat Completions model (e.g., gpt-4o, gpt-4o-mini)
• Temperature: Controls randomness (0.0—2.0)
• Max tokens: Upper bound on response tokens
• API key: Managed Key (provisioned via OpenRouter) or Bring Your Own Key

To view your provisioned key: Settings > Your Settings > OpenAI API Key.

Select Fields & Preview

• Select field name: Pick the upstream column
• Set prompt: Write instruction. Use #{field_name} to interpolate values
• Set alias: Provide the output column name (unique by design)
• Preview: Click Preview to run on a small sample (5 rows by default)
• All AI prompt fields are returned as string datatype

Testing

1. Connect AI Transformation after a source or transform component.
2. Configure fields, prompts, and alias.
3. Click Preview; a small number of rows will be processed.
4. Inspect outputs and iterate.

Best Practices

• Keep prompts short and explicit; specify the desired output format.
• Use interpolation to add context from multiple fields.
• Use preview to validate behavior and manage token usage.
• You can use ChatGPT to help write prompts.

Managed Key Costs

When using an Integrate.io Managed Key, AI Transformation usage is billed based on the LLM provider’s consumption pricing. Charges are a direct pass-through of the provider’s API usage. Billed monthly in arrears. You can configure a cost limit to cap spend.

Cost depends on: number of rows processed, input data length, output size, and model selected.