ETL: Salesforce PK Chunking

Enable PK Chunking for large Salesforce data extractions in Integrate.io ETL. Improve query performance when extracting millions of records.

What is Salesforce PK Chunking?

PK Chunking is a Salesforce feature built for large datasets. Salesforce Primary Key (PK) is the object’s indexed record ID. Steps to use PK Chunking:

1. Query the target table to identify a number of chunks of records

2. Send separate queries to extract the data in each chunk

3. Lastly combine the results.

Enabling Salesforce PK Chunking

By enabling this feature, Salesforce automatically splits the Bulk API Query job into multiple batches. We then poll for the progress of each of the batch then process them in parallel once all are done, resulting in significantly reduced job run times. The parallelism depends on cluster node count.

The default chunk size is 100,000, but you can use the ‘chunkSize‘ header field to configure smaller chunks or larger ones up to 250,000. It’s best to experiment with chunk sizes to figure out what works best for your needs.

Read more here: Salesforce PK chunking documentation