ETL: REST API Destination

Configure the REST API destination in Integrate.io ETL: send pipeline data to an HTTP endpoint with custom headers, auth, batching and payload templates.

The REST API destination sends data from your pipeline to any HTTP endpoint. You can configure the request method, headers, body format, authentication, and batching to match your target API’s requirements.

Authentication

Choose one of the following authentication options:

• None - No authentication. Use this when the target API requires no credentials or when you pass authentication through custom headers.
• Basic - HTTP Basic authentication. Provide a Username and Password that are sent as a Base64-encoded Authorization header with each request.
• Connection - Use a saved connection for SSH tunnel access. When an SSH tunnel is active, HTTPS URLs are automatically rewritten to route through the tunnel’s local port.

Request Configuration

URL

Enter the target API endpoint URL. You can include dynamic values from your input data using the #{fieldName} syntax. For example:

https://api.example.com/contacts/#{contact_id}

The #{contact_id} placeholder is replaced with the value of the contact_id field from each input record. All field references in the URL are validated against the input schema.

Method

Select the HTTP method for the request:

• GET
• POST
• PUT
• PATCH
• DELETE

Headers

Add custom headers as name/value pairs. Header values also support the #{fieldName} syntax for dynamic values. Common headers like Content-Type are set automatically based on your chosen request body format, but you can override them here.

Request Body

Body format

• JSON - Sends records as JSON objects. This is the most common format for modern APIs.
• CSV - Sends records as comma-separated values.
• Form Data - Sends records as multipart form-data.
• URL-Encoded - Sends records as application/x-www-form-urlencoded.

Custom payload

Enable Custom Payload (the Custom input payload type) to write your own request body template instead of using the automatic field-to-body mapping. The template can use input fields and package variables:

• #{fieldName} is replaced with the value of fieldName from each record. Typing #{ in the editor lists the input fields.
• $variableName is replaced with the value of a package variable, the same way it is in the URL. Variable values are inserted as-is, without escaping. A $ followed by a name that isn’t a package variable makes the job fail.

A template that uses #{fieldName} sends one request per record, so it requires Single per record as the request mode. The editor switches to it automatically.

{
  "contact": { "id": #{contact_id}, "name": "#{full_name}", "tags": #{tags_json} },
  "source": "$source_tag"
}

How field values are escaped

Each value is escaped for the place its placeholder sits in the body, so record data can fill in a value but cannot change the shape of the payload.

Body format	Placeholder inside quotes	Placeholder outside quotes
JSON	Sent as text. Quotes, backslashes and control characters (line breaks, tabs) are escaped. An empty value is sent as "".	Inserted as raw JSON. Use this for numbers, booleans, or a text field that already holds JSON. The value must be one JSON value; anything else stops the job at that record, after the earlier records were sent. An empty value (not one made of spaces) is sent as null.
URL-Encoded	Not applicable	The value is URL-encoded, so characters like & and = cannot add form fields.
CSV	A " in the value is doubled.	The value is wrapped in quotes when it contains a comma, quote or line break.
CSV with a JSON or URL-encoded Content-Type header	Same as the JSON or URL-Encoded row	Same as the JSON or URL-Encoded row
CSV with an XML Content-Type header	Not applicable	&, <, >, " and ' are escaped as XML entities.
Form Data, or CSV with another Content-Type	Not applicable	Inserted as-is.

With the CSV body format the payload is sent exactly as you write it, so the Content-Type header you set decides how values are escaped. A body filled in from #{fieldName} placeholders is sent exactly as rendered in every format.

For JSON, put a placeholder in quotes ("#{full_name}") for any text field. Outside quotes, a record’s value must be exactly one JSON value: a number, true or false, null, an object or an array. Plain text outside quotes stops the job at that record, and so does text that would add keys, such as 1, "role": "admin". The editor warns about every placeholder outside quotes unless it knows the field is a number or a boolean. A field whose type it can’t tell, such as an Aggregate output, is included.

A date field, or a map or bag field (for example the output of JsonStringToMap), can’t be used as a placeholder directly: package validation fails with a “Cannot explicitly cast” error, and the job can’t run. Convert it in a Select component first, for example with ToString(created_at) for a date or ToJson(field) for a map or bag, and reference the new field.

If you already escape values for JSON before this destination, for example with REPLACE in a Select component to turn " into \", remove that step. Values inside quotes are now escaped automatically, so pre-escaped values would be escaped twice and arrive with extra backslashes.

Validation

When the body format is JSON and the template uses #{fieldName}, package validation checks that the template is valid JSON with the placeholders filled in. A template that is not valid JSON is reported when you validate the package, not halfway through a job. If a record’s value still breaks the body when the job runs (for example, plain text in a placeholder outside quotes), the job error names the field. It never includes the record’s value.

Nested attribute

Enable Send records as nested attribute to wrap your record data inside a named object. For example, if you set the nested attribute name to records, the request body becomes:

{
  "records": [
    { "field1": "value1", "field2": "value2" }
  ]
}

The default nested attribute name is records.

Static fields

Use JSON Constants to add static key-value pairs to every request body. These are included alongside the mapped record data.

Request Mode

Controls how input records are grouped into HTTP requests:

• Single per record - Sends one request per input record. Use this when the target API expects individual records (most common for create/update endpoints).
• Batch - Groups records into batches of a specified size and sends one request per batch. Set the Batch size to control how many records are included in each request. The batch size must be greater than 0.
• One request - Sends all records in a single request.

Field Mapping

Map input fields from your pipeline to the request body fields expected by the target API. Each mapping includes:

• Input Field - The field from your pipeline input
• Destination Column - The name used in the outgoing request body
• Field type - The data type (must match the schema field type)

Send the value of one field

When the body format is JSON and you don’t use a custom payload, you can send the value of a single field instead of the full record. The checkbox label depends on the request mode:

• Single per record - Send only the value of one field. Each request body is the field’s value for that record.
• Batch and One request - Send values from one field as an array. Each request body is an array that holds the field’s value from each record.

When you select the checkbox, the field mapping is limited to one row, and Integrate.io sends the value of that row’s input field. A value that holds a JSON object or array as text is sent as JSON. For example, if the input field holds {"id": 1, "name": "Alice"}, the request bodies look like this:

// Single per record
{ "id": 1, "name": "Alice" }

// Batch or One request
[{ "id": 1, "name": "Alice" }, { "id": 2, "name": "Bob" }]

In Single per record, a text value is sent as is, without JSON quoting. Inside an array or a nested attribute, it is sent as a JSON string. If you also send records as a nested attribute (see Nested attribute), the value or array is wrapped in it together with any static fields, for example {"records": "value"}.

If a record has no value (NULL) in that field, Integrate.io sends null in its place. How the API handles null is up to the API; Google Sheets, for example, skips that row. Preview with data shows a warning when a previewed record has no value.

Advanced Settings

Request sleep interval

Set a delay (in milliseconds) between requests when using Single per record mode. Valid range: 0 to 60,000 ms. This is useful for respecting API rate limits.

Fail on error response

When enabled, the job fails if any API request returns an error response (HTTP 4xx or 5xx status). When disabled, the job continues processing remaining records even if individual requests fail.

Save responses to file

Enable this to write API response data to file storage. Configure the Response file path (an HDFS-compatible path) and Response file format to control where and how responses are saved. This is useful for logging, auditing, or downstream processing of API responses.

FAQ

How do I send a field value as a JSON string?

Put the placeholder in quotes: "name": "#{full_name}". The value is escaped automatically, so quotes, backslashes and line breaks in the data are sent correctly. To send a date, convert it in a Select component first, for example ToString(created_at) AS created_text, then use "created": "#{created_text}".

How do I send a number, boolean or JSON object from a field?

Leave the placeholder outside quotes: "count": #{count} or "meta": #{meta_json}. The value is inserted as raw JSON, so it must be exactly one JSON value. An empty value (not one made of spaces) is sent as null.

Can I use #{fieldName} with Batch or One request mode?

No. A custom payload with field placeholders is built from one record at a time, so it requires Single per record. A custom payload without placeholders works with every request mode.

Can I use package variables in the custom payload?

Yes. Reference them as $variableName, inside or outside quotes, as you would in the URL.

Related

REST API Source

API Call Transformation

Curl Function

Scheduling Package Execution