> ## Documentation Index
> Fetch the complete documentation index at: https://www.integrate.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# ETL: Creating a Workflow

> Create workflow packages in Integrate.io ETL to orchestrate multi-step tasks. Define sequential and conditional task execution with error handling.

<Frame>
  <iframe src="https://fast.wistia.com/embed/iframe/78fd4qurqq" allowfullscreen playinline class="wistia_embed" width="640" height="360" />
</Frame>

## To create a workflow:

<Steps>
  <Step>
    On the main menu, click **Packages**.
  </Step>

  <Step>
    Click **New package**.

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-1.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=1c6ae52eb82c243ac4637a2d4cde9914" alt="Packages menu with New package button" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-1.png" />
    </Frame>
  </Step>

  <Step>
    Optionally, set a name and/or description
  </Step>

  <Step>
    Select **workflow** option from Type dropdown
  </Step>

  <Step>
    Click **Create package** to create workflow package.

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-2.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=e23920094b48e14979422419d41a41ac" alt="New package dialog with workflow type selected" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-2.png" />
    </Frame>
  </Step>

  <Step>
    Click  **+ Add Task** button.

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-3.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=67e45db4e32b4830a8fca27b7b2bba28" alt="Add Task button on the workflow canvas" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-3.png" />
    </Frame>
  </Step>

  <Step>
    Choose task.

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-4.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=0695f75ad1418c0cb8cca13057666c1a" alt="Task type selection menu with available task options" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-4.png" />
    </Frame>
  </Step>

  <Step>
    Add additional tasks as required
  </Step>

  <Step>
    Connect tasks to create sequence of task execution. Click the connect icon on the dotted line to set the execution condition:

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-5.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=54bf89d20310c51dedacff76a0d93962" alt="Workflow tasks connected with execution condition options" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-5.png" />
    </Frame>

    * **On success (default)** - task will be executed once the preceding task was executed successfully
    * **On failure** - task will be executed once the preceding task execution failed
    * **On completion** - task will be executed once the preceding task completed, regardless to the completion status (failed/succeeded)
  </Step>
</Steps>

## Execute SQL task

<Steps>
  <Step>
    Select your DB connection.

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-6.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=22c5d15230281f8254b3a3417e1aca65" alt="Execute SQL task with database connection selector" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-6.png" />
    </Frame>
  </Step>

  <Step>
    Write the SQL query that should be executed, and select the query result type from **Result type** dropdown. You can test the query by clicking **Test Query**.

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-7.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=00d02883499924a108ec45b3738d04d9" alt="SQL query editor with result type dropdown and Test Query button" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-7.png" />
    </Frame>
  </Step>

  <Step>
    You can assign the return value to a workflow variable.
  </Step>

  <Step>
    If there are at least two preceding tasks, choose task execution condition:

    * all preceding conditions evaluate to true (AND)
    * one of the preceding conditions evaluate to true (OR)
  </Step>
</Steps>

<Note>
  **Note**:

  BigQuery connections uses standard SQL within Execute SQL task
</Note>

## Run package task

<Steps>
  <Step>
    Select the package to run

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-8.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=a6ecf5b8edd7101e8558a1ef8a9406c3" alt="Run package task with package selector dropdown" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-8.png" />
    </Frame>
  </Step>

  <Step>
    Optionally, you may edit the dataflow variables if you want to override the values with workflow variables.
  </Step>

  <Step>
    If there are at least two preceding tasks, choose task execution condition:

    * all preceding conditions evaluate to true (AND)
    * one of the preceding conditions evaluate to true (OR)
  </Step>
</Steps>

## File mover task

<Steps>
  <Step>
    Select a file storage connection as the source of the file(s)
  </Step>

  <Step>
    Input source bucket (if connection requires a bucket) as well as the source path

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-9.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=2e57063368106e4eb90877bd86cc7516" alt="File mover task source connection and path configuration" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-9.png" />
    </Frame>
  </Step>

  <Step>
    You can enable following options on the source connection:

    * Delete files from source
    * Fail if the source is empty
  </Step>

  <Step>
    Select a file storage connection as the destination of the file(s)
  </Step>

  <Step>
    Input destination bucket (if connection requires a bucket) as well as the destination path. Any directories/folders in the path must already exist. If you wish to change the name of the file, define the path all the way to the file name and extension - for example, directory\_name/new\_file\_name.csv

    <Frame>
      <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-10.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=19ca79a836233f0d8d6d38f86b21d1f6" alt="File mover task destination connection and path configuration" width="1200" height="828" data-path="images/creating-packages/creating-a-workflow/image-10.png" />
    </Frame>
  </Step>

  <Step>
    If there are at least two preceding tasks, choose task execution condition:

    * all preceding conditions evaluate to true (AND)
    * one of the preceding conditions evaluate to true (OR)
  </Step>
</Steps>

<Note>
  **Note**:

  Package variable (\`\$\`) and Wildcard file pattern (\`\*\`) can be applied as bucket or path name.
</Note>

## Using Variables in Workflows

User variables can be defined at the workflow package level and can be used for both the Execute SQL Task and the Run Package Task.

1. **Execute SQL Task**

   * Variables can also be assigned values by the Execute SQL task. This is useful if you want to have dynamic values on your variable and use it later on.
     * When using variables in a SQL query, enclose the variable within curly brackets (i.e: `'${var\_name}'`).
     * Example of using a variable within the Execute SQL task query:\
       DELETE FROM event\
       WHERE `date>='${START\_DATE}'`\
       AND `date<='${END\_DATE}'`

2. **Run Package Task**
   * Workflow package level variables can be used to override dataflow level variables. Values you do not override track the child dataflow's current variables and secrets, so edits made on the child are reflected in the workflow without re-saving. Here's an example:

     <Frame>
       <img src="https://mintcdn.com/integrateio/2ttHYDu3EKov-VoY/images/creating-packages/creating-a-workflow/image-11.png?fit=max&auto=format&n=2ttHYDu3EKov-VoY&q=85&s=cf33b9a6b58473155fbf7255ab4a0199" alt="Workflow variables overriding dataflow package variables" width="1200" height="639" data-path="images/creating-packages/creating-a-workflow/image-11.png" />
     </Frame>

     <Note>
       **Note:**

       Take note that we address package variables regularly as **\$variable\_name** which is different from the way we used it on Execute SQL Task above.
     </Note>

     * If both workflow and dataflow variables have the same name, you will still have to assign the workflow variable to the dataflow variable.
     * If a task dataflow uses a variable that isn't defined at the dataflow level but is assigned a value at the workflow level, the dataflow task will use the workflow variable value.
