ETL: Promoting Packages with GitHub

How to promote Integrate.io ETL packages from a Dev account to QA and Prod through GitHub pull requests and GitHub Actions.

Package promotion moves Integrate.io ETL packages from a Dev account to QA and then Prod using GitHub: a pull request for each step, a check that shows what will change, and GitHub Actions deploying each merge. Nobody edits packages in QA or Prod by hand, and every promoted version traces back to the Dev version, the reviewed pull request and the run that deployed it.

Promotion builds on the GitHub integration. Set that up first on the Dev account.

How It Works

1. Sync from Dev. Sync to GitHub commits a package to the Dev branch (for example dev), which mirrors the Dev account and never deploys anywhere.
2. Promote to QA. Open a pull request from dev into qa. The Promotion plan check shows what will happen in QA and validates the package files. Merging runs the Promote workflow, which creates or updates each package in the QA account, applies QA’s variables and secrets, and validates it.
3. Promote to Prod. Open a pull request from qa into main. Same check, against Prod. Merging promotes the exact files that were deployed to QA.

Each promotion is recorded in the package’s version history, for example Promoted from dev 1.7 via qa by GitHub Actions, commit <sha>, run <url>.

What’s in the repository

Path	What it holds
promotion.json	The environments, their order, which branch deploys to which account, and each package’s id in each target. Read from the default branch only, and changed only through pull requests.
integrate-io.json	The package index: the Dev account and each package’s id there, in promotion order. Written by Sync to GitHub.
packages/*.json	The exported packages. Secret names only, never their values.
env/<env>.json	Per-environment variable values, and which GitHub secret supplies each package secret.
.github/workflows/	The workflows: Promotion plan (the pull request check), Promote, Export from source (optional) and the tool’s tests.
scripts/iio_promote.py	The promotion tool the workflows run.

Requirements

• A Dev account linked to the repository through the GitHub integration, plus a QA account and a Prod account (or any other chain, such as Dev, QA, UAT, Prod).
• Package JSON export (the source control feature) on the Dev account, which Sync to GitHub uses. Contact Integrate.io support to enable it.
• Connections in QA and Prod with exactly the same names and types as in Dev. Packages are matched to connections by name, so credentials stay in Integrate.io and never pass through GitHub.
• Schedules in QA and Prod created once and set to always run the latest package version. Promotions don’t change schedules.
• GitHub Actions enabled on the repository, and permission to manage its branches, environments and secrets.

Set Up Promotion

1. Enable promotion from the Dev account

1. Open Source Control

   In the Dev account, go to Settings > Account > Source Control. Under Promotion, click Enable promotion.

2. List the environments

   List the environments in promotion order. Each row has an environment name, the Integrate.io account, and the branch that deploys to it. Use + Add environment for more rows. The last row deploys from the repository’s default branch (usually main).

   Environment	Integrate.io account	Branch
   dev	your Dev account	dev
   qa	your QA account	qa
   prod	your Prod account	main

   You need access to every account you list.

3. Propose the change

   Click Propose change. Integrate.io commits the workflows, the promotion tool, promotion.json and the env/ files to a branch of their own.

4. Create and merge the pull request

   Click Create pull request to open GitHub’s pull request form, then create it, review it and merge it into the default branch. Back in Integrate.io, click Check again. The page then shows the promotion order and the repository checks.

2. Link the QA and Prod accounts

In each target account (QA, then Prod):

1. Connect the same repository

   Go to Settings > Account > Source Control, click Connect GitHub and choose the same repository. These accounts are read only: Sync to GitHub is off there.

2. Confirm its environment

   If the account is already listed in promotion.json, the page shows its environment. If it isn’t, use Add this account to promotion.json: enter the environment name, its branch and which environment it comes after, then click Propose change and merge the pull request it opens.

3. Create the CI users and API keys

The workflows call the Integrate.io API with a CI user’s API key, read from a secret named IIO_API_KEY. The plan check runs the pull request’s own code, so give it a key that can only read, and keep the key that changes QA and Prod where only a reviewed merge reaches it.

CI user	Role in each account (with custom roles)	Its API key goes in
Read-only, for example [e-mail]	Dev, QA, Prod: view and list packages, connections and package groups	The repository secret IIO_API_KEY
Promoting, for example [e-mail]	QA, Prod: also create and update packages	An environment secret IIO_API_KEY on each target GitHub environment (qa, prod)

An API key belongs to a user, so one key works in every account its user belongs to. Without custom roles, users can’t be limited to reading: then one CI user’s key in the repository secret works for everything, and you should limit write access to the repository to the people allowed to deploy.

4. Configure GitHub

1. Protect the target branches

   Protect each target branch (qa, main): require pull requests and the plan status check (the job in the Promotion plan workflow). Leave the Dev branch (dev) open to pushes, because Sync to GitHub commits to it directly.

2. Use merge commits

   For pull requests between promotion branches, use Create a merge commit. Squash and rebase merges rewrite the history the next promotion builds on, and later pull requests then conflict.

3. Create the GitHub environments

   Create one GitHub environment per target, named as in promotion.json (qa, prod):

   • Deployment branches: only that environment’s branch (qa for qa, main for prod). This rule is what stops any other branch from deploying or reaching the promoting key. It isn’t optional.
   • Secrets: the promoting user’s IIO_API_KEY, and each secret named in env/<env>.json, as environment secrets, not repository secrets.
   • Required reviewers on prod, if you want a second approval.
4. Add the repository secret and variable

   Add the read-only user’s key as the repository secret IIO_API_KEY. Add the repository variable IIO_API_URL only if your account isn’t on https://api.xplenty.com.

5. Protect promotion.json

   Add a CODEOWNERS entry for promotion.json so the owners of Prod approve every change to where packages deploy, for example /promotion.json @your-org/prod-owners.

5. Check the setup

On any linked account’s Source Control page, the repository checks list what’s ready and what isn’t: promotion.json, the promotion workflows, the accounts, each branch’s protection, the IIO_API_URL variable, CODEOWNERS and the template version. Items such as the IIO_API_KEY secret and the environment secrets can only be confirmed in GitHub, so the page links to them.

Promote a Package

1. Sync the package from Dev

   In the Dev account, use Sync to GitHub on the package. See Sync a package to GitHub.

2. Open the pull request

   On the Dev account’s Source Control page, click Open pull request: dev → qa (or open it in GitHub). An open pull request picks up later syncs, so you don’t need a new one for each change.

3. Review the plan

   The Promotion plan check shows, for each package, whether merging creates or updates it, and which variables and secrets apply. It blocks the merge if a connection, package group or secret the package needs is missing in the target, or if the package fails validation.

4. Merge

   Merge the pull request. The Promote workflow updates the packages in QA. Then open a pull request from qa into main to promote the same files to Prod.

Variables and secrets per environment

Set environment-specific values in env/<env>.json on the default branch:

{
  "variables_mode": "preserve",
  "packages": {
    "customers-to-warehouse": {
      "variables": {"batch_label": "'prod'"},
      "secrets": {"api_token": "PROD_CUSTOMERS_API_TOKEN"}
    }
  }
}

Field	What it does
variables	Values that differ in this environment, applied after every import. Variables not listed keep the target package’s current value.
secrets	Package secret name, mapped to the name of the GitHub environment secret that holds its value.
variables_mode	preserve (default) keeps the target’s existing values and adds variables new in Dev. replace overwrites them with Dev’s values before variables is applied.

Packages new to a target

The first promotion of a package looks for a package with exactly the same name in the same package group in the target account, and creates one if there is none. The package group must already exist there. Its new id is recorded in promotion.json through the next promotion pull requests, after which renames are safe. Secrets the environment file doesn’t map must be set on the new package in the target account; the run summary lists them.

Limitations

• A failed promotion isn’t rolled back. If validation fails after the import, restore the previous version from the package’s version history.
• Deletions aren’t promoted. Removing a package from integrate-io.json doesn’t delete it in QA or Prod.
• Packages are promoted one at a time, in the order listed in integrate-io.json. If one fails, the run stops and earlier packages stay promoted. Re-running is safe.
• Workflows find the packages they run by name in the target, so the packages a workflow runs are promoted before it. Renaming a child workflow in Dev can stop the next promotion of its parent until the child’s new name is promoted.
• Validation runs in Dev before the merge (with Dev’s connections), and again in the target after the import.

FAQ

Q: Can I promote through more environments, such as UAT?

Yes. The chain comes from promotion.json, so Dev, QA, UAT, Prod works the same way: add the environment, its branch, its GitHub environment and its env/<env>.json.

Q: Do secret values pass through GitHub?

Package files never contain secret values. Values you choose to supply from GitHub are stored as environment secrets and sent to the target account during the Promote run only.

Q: Who can change where packages deploy?

Only a merged pull request into the default branch, because every workflow reads promotion.json from there. A CODEOWNERS entry makes Prod’s owners approve those changes.

Q: Do I need to change my schedules after a promotion?

No, as long as the schedules in QA and Prod always run the latest package version.

Q: Can I edit packages in QA or Prod directly?

You can, but the next promotion replaces them with the version from Dev. Make changes in Dev and promote them.

Related

GitHub Integration

Migrating from Development to Production

Package Version Control

Run Package Task