ETL: REST API Destination

How to configure the REST API component to write data in your Integrate.io ETL pipeline.

The REST API destination sends data from your pipeline to any HTTP endpoint. You can configure the request method, headers, body format, authentication, and batching to match your target API’s requirements.

Authentication

Choose one of the following authentication options:

• None - No authentication. Use this when the target API requires no credentials or when you pass authentication through custom headers.
• Basic - HTTP Basic authentication. Provide a Username and Password that are sent as a Base64-encoded Authorization header with each request.
• Connection - Use a saved connection for SSH tunnel access. When an SSH tunnel is active, HTTPS URLs are automatically rewritten to route through the tunnel’s local port.

Request Configuration

URL

Enter the target API endpoint URL. You can include dynamic values from your input data using the #{fieldName} syntax. For example:

https://api.example.com/contacts/#{contact_id}

The #{contact_id} placeholder is replaced with the value of the contact_id field from each input record. All field references in the URL are validated against the input schema.

Method

Select the HTTP method for the request:

• GET
• POST
• PUT
• PATCH
• DELETE

Headers

Add custom headers as name/value pairs. Header values also support the #{fieldName} syntax for dynamic values. Common headers like Content-Type are set automatically based on your chosen request body format, but you can override them here.

Request Body

Body format

• JSON - Sends records as JSON objects. This is the most common format for modern APIs.
• CSV - Sends records as comma-separated values.
• Form Data - Sends records as multipart form-data.
• URL-Encoded - Sends records as application/x-www-form-urlencoded.

Custom payload

Enable Custom Payload to define your own request body template instead of using the automatic field-to-body mapping. The template is a raw string where you can reference input fields. When the body format is JSON, the custom payload is validated for correct JSON syntax.

Nested attribute

Enable Send records as nested attribute to wrap your record data inside a named object. For example, if you set the nested attribute name to records, the request body becomes:

{
  "records": [
    { "field1": "value1", "field2": "value2" }
  ]
}

The default nested attribute name is records.

Static fields

Use JSON Constants to add static key-value pairs to every request body. These are included alongside the mapped record data.

Request Mode

Controls how input records are grouped into HTTP requests:

• Single per record - Sends one request per input record. Use this when the target API expects individual records (most common for create/update endpoints).
• Batch - Groups records into batches of a specified size and sends one request per batch. Set the Batch size to control how many records are included in each request. The batch size must be greater than 0.
• One request - Sends all records in a single request.

Field Mapping

Map input fields from your pipeline to the request body fields expected by the target API. Each mapping includes:

• Field name - The name used in the outgoing request body
• Column name - The source field from your pipeline input
• Field type - The data type (must match the schema field type)

When Set field name is enabled, field names are included in CSV and form-data formatted requests.

Advanced Settings

Request sleep interval

Set a delay (in milliseconds) between requests when using Single per record mode. Valid range: 0 to 60,000 ms. This is useful for respecting API rate limits.

Fail on error response

When enabled, the job fails if any API request returns an error response (HTTP 4xx or 5xx status). When disabled, the job continues processing remaining records even if individual requests fail.

Save responses to file

Enable this to write API response data to file storage. Configure the Response file path (an HDFS-compatible path) and Response file format to control where and how responses are saved. This is useful for logging, auditing, or downstream processing of API responses.

Related

REST API Source

Curl Function

Scheduling Package Execution