Target bucket - the name of the target cloud storage bucket where the package output directory and files will be written. Only relevant in the case of object stores such as Amazon S3 and Google Cloud Storage object stores.
Target directory - the name of the target directory (within the bucket for object stores). One or more files will be created in the directory. By default, if the target directory already exists, the job will fail (see below).
Destination format define the target format to use.
Delimited values options - produce csv, tsv or any other delimited values format. The output files are utf-8 encoded.
Line delimited JSON - produce a JSON object per record in each line of the output files. The output files are utf-8 encoded.
Parquet - Apache Parquet is a columnar storage format popular with Impala, AWS Athena, Presto and other open source DWH solutions. You can control the Parquet output with the system variables: _PARQUET_COMPRESSION, _PARQUET_PAGE_SIZE, _PARQUET_BLOCK_SIZE. Note that datetime and complex data types are not supported with Parquet.
Fixed width - write records as fixed-width lines with no separators, one column per field. Configure the per-field layout in the Fixed width layout step.
Delimited values options
Delimiter - select or type a single character to separate values in the output file (tab by default).
String Qualifier - if double quote or single quote are selected, fields that contain the selected delimiter will be enclosed in single or double quotes.
Escape Character - Character used to escape string qualifiers on the field data in case they exist. Only selectable if string qualifier exists (as there’s nothing to escape if there is no string qualifier).
Line ending - Line ending for each line. Choose between Unix/Linux (CRLF) or Windows (CRLF).
Write field names in header - check to add a header line containing the field names in each output file. If you’d like to have spaces in the headers name-
It works by automatically by setting triple underscores (___) on the alias / column name in a select component as shown below.
If you would like to change the identifier other than underscore, you could add a package variable_FS_REPLACE_WITH_SPACE_IDENTIFIER and specify the character.
Output compression - Select the type of compression for your data (Gzip, Bzip2 or none). Using Gzip or Bzip2 compression adds a .gz or .bz2 suffix to the output directory name.
When Destination format is set to Fixed width, a Fixed width layout step appears in the editor. The layout is a table with one row per incoming field, in schema order. Every incoming field is written, with no separators between columns.To write only some fields, or to write them in a different order, add a Select component before this destination. The Select shapes the outgoing tuple; the layout describes how that tuple is laid out on the line.Each row has the following settings:
Field - the incoming field name. Read-only.
Width - number of characters this field occupies on the line. Whole number between 1 and 1000.
Align - Left or Right. Controls which side the padding is added to.
Pad - exactly one character used to fill unused width. A single space is allowed. Defaults to a space.
If too long - what to do when the value is wider than Width:
Fail the job - stop the job on the first row that does not fit. Numeric fields always fail on overflow, regardless of this setting.
Truncate - cut the value to fit. Applies to text fields only.
Decimals - number of implied decimal places for numeric fields. 0 writes the value as-is. With 2, the value 12.34 is written as 1234 (the last two digits are the fractional part, with no decimal point). Values are rounded HALF_UP. Whole number between 0 and 1000.
Sign - where to place the sign for numeric fields:
None - no sign character.
Leading - sign character at the start of the field.
Trailing - sign character at the end of the field.
The record width in characters is shown below the table and updates as you edit.
Click Preview with data to run this component through the previewer and draw the records the job would actually write. Use it to check that the layout produces the byte layout you expect before saving.Rows whose sample value will not fit in the configured width are marked with an advisory message. This is a cue only and does not block Save. Blocking errors (invalid width, pad, or decimals) appear in the layout’s errors box and prevent Save.
The layout persists as fixed_width_destination_type on the destination component. The field entries match the incoming schema by index at runtime. A sample entry:
record_delimiter and char_encoding round-trip through the API but are not editable in the UI. To write a layout with a different record delimiter (for example CRLF), edit the package through the API.
To encrypt output files with a trading partner’s OpenPGP public key, select an active entry from the PGP Key dropdown. The entry must hold a public key. Files are encrypted to that public key before they are written to the target path. Paths that contain a space, semicolon, or parenthesis are not supported when a PGP key is set. See PGP Keys for how to add and manage entries.
Write all files directly and fail the job if target directory already exists - Files will be written directly to the target directory. By default, the job will fail if the destination directory already exists. You can use variables to dynamically generate new directory names every time a job is executed (e.g. /output/${_JOB_ID}).
Write all files directly and delete target directory if already exists - Files will be written directly to the target directory. During execution, the job checks if the directory exists and deletes it.
Write all files directly and replace files in directory if they already exist - Files will be written directly to the target directory. When selected, the job will not fail when the destination directory exists and files will be written to the existing directory. If files with the same names exist in the directory, they will be overwritten. If you’d like to add files to the directory, make sure to change use a custom file pattern that is unique for every job execution (e.g. use the variable ${_JOB_ID} in the file prefix (see below).
Warning:Target directory will be deleted on job failure.
Use intermediate storage and copy files to an existing directory in destination - Files will be written to an intermediate storage and then will be copied to the target directory. Please make sure that the target directory exists.
Use intermediate storage and copy files to a new directory in destination - Files will be written to an intermediate storage and then will be copied to a new target directory. If the directory already exists, the job will fail.
Merge output to single file - check to make sure only a single file is written to destination directory. Limitations on file sizes in certain platforms may fail your job.
Specify the character encoding of the output file. Only available when Destination action is set to Use intermediate storage and copy files to an existing directory in destination.
Default file pattern is part-[mr]-[0-9]{5} (for example part-m-00000).
To change it select custom pattern:
File name prefix - leave empty to keep the default prefix (part-[mr]-) or change to your custom prefix. Use variables to set the prefix dynamically. A common use case is a time stamp generated dynamically with a variable (for more information, see Using and setting variables.)
File name suffix - Integrate.io ETL automatically suggests suffixes according to the file format and compression type you selected.
