ETL: Clusters Overview

Learn about Integrate.io ETL clusters, the compute environments that run your data pipeline jobs. Understand cluster types, sizing, and lifecycle.

When you create a new cluster, you can choose from different cluster sizes (the number of nodes) according to the computing power you require for different types of jobs and packages (if you have a contract that allows you to do so). Generally speaking, the relationship between the number of nodes and computing power is linear: if a job takes four hours to run a four-node cluster, it will take one hour to run on a 16 node cluster. Integrate.io is built to handle large data volumes at scale. By distributing processing across multiple nodes in a cluster, even pipelines moving billions of rows can be completed efficiently without manual infrastructure management or custom code. All accounts include one free sandbox cluster for testing jobs on relatively small amounts of data during the package development cycle. Since the sandbox cluster is single-node and neither scalable nor highly available, it is not production-grade. Trial users who haven’t signed a contract yet have unlimited access to a sandbox cluster and can run up to 4-node production clusters max of 200 prod hours (with more hours available upon request) during the trial period.