ETL: Python Transformation

Use the Python Transformation to run custom Python code for data processing as part of your Integrate.io ETL pipeline. Supports pandas.

The Python Transformation component allows you to run custom Python code as part of your Integrate.io ETL pipeline. It transforms records without changing the schema of the input data.

Overview

• Accepts an array of JSON records as input
• Requires a defined transform function
• Input and output schemas must match exactly
• Useful for string manipulation, cleaning, enrichment, or any custom logic
• You can remove records from the array (filtering), but returned records must follow the original schema
• You may return an empty array, but not a different schema

Example Transformation Code

import json

def transform(event, context):
    """
    Transforms input records by converting all string values to uppercase.
    The output schema must match the input schema exactly.
    """
    try:
        data = event
        results = []

        for record in data:
            transformed_record = {}
            for key, value in record.items():
                if isinstance(value, str):
                    transformed_record[key] = value.upper()
                else:
                    transformed_record[key] = value
            results.append(transformed_record)

        return results

    except Exception as e:
        raise e

Example Input and Output

Input:

[
  { "id": 1, "name": "Belgian Waffles", "price": 5.95 },
  { "id": 2, "name": "Pancakes", "price": 4.95 }
]

Output:

[
  { "id": 1, "name": "BELGIAN WAFFLES", "price": 5.95 },
  { "id": 2, "name": "PANCAKES", "price": 4.95 }
]

Configuration

Batch Size: Define the number of records per batch. The total payload size should not exceed 6 MB.

Testing

Test your transformation in the Package Designer: provide a sample payload in table format, click Run Code, and view the transformed output. The test table mirrors the schema of the connected input component.

Variables

Use package, secret, and global variables with Python import format:

from package_variables import variable_name

All variables (including secrets and globals) are evaluated before passing to the function.

Best Practices

• Always ensure output matches input schema (field names and types).
• Valid to return fewer records or an empty array for filtering.
• Catch and log exceptions to help debug errors.
• Break complex logic into helper functions within the same script.