Breadcrumbs

Pipelines and integrations

Use Pipelines and integrations to bring data into Yarken from cloud storage, APIs, and supported external applications.

You can configure reusable connections, map source fields, automate recurring ingestion, and monitor data processing from one application area.

Go to Admin > Pipelines to access:

  • Connections

  • Automate

  • Data Mapping Templates

  • Pipelines

What you can do

Use Pipelines and integrations to:

  • Connect Yarken to supported cloud storage providers, APIs, and external applications.

  • Create reusable connections and maintain authentication details.

  • Map source fields to Yarken fields with Data Mapping Templates.

  • Automate recurring file ingestion.

  • Retrieve supported billing, usage, licensing, and operational data through APIs.

  • Run data ingestion on a schedule or on demand.

  • Monitor pipeline and integration activity.

Choose what you need to do

Your goal

Go to

Connect Yarken to cloud storage or an API

Connections

Reuse source-to-destination field mappings

Data Mapping Templates

Set up recurring file ingestion

Automate

Create, run, or monitor an ingestion workflow

Pipelines

Use a Yarken-managed Advanced Integration

Advanced integrations

Resolve a connection, mapping, or ingestion issue

Troubleshoot Pipelines and integrations

Choose BASIC or ADVANCED

The Connections and Pipelines pages contain BASIC and ADVANCED tabs.

Choose the method that matches your data source and integration requirements.

Method

Use it when

BASIC

You need a native Yarken connection or pipeline for a supported cloud storage provider or API.

ADVANCED

You need a Yarken-managed integration or provider-specific workflow enabled for your environment.

BASIC integrations

Use BASIC integrations for supported recurring data ingestion.

A BASIC integration may use:

  • A cloud storage or API connection.

  • A Data Mapping Template.

  • An Automate configuration.

  • A scheduled or on-demand pipeline.

BASIC integrations are suitable for:

  • Files stored in Azure Blob Storage.

  • Files stored in AWS S3.

  • Files stored in Google Cloud Storage.

  • Microsoft Graph API data.

  • Power BI API data.

  • Supported Google API data.

ADVANCED integrations

Use ADVANCED integrations for more complex or provider-specific workflows.

Yarken manages the integration logic that connects supported external applications and services with Yarken.

An Advanced Integration may use:

  • An Advanced Connection.

  • A Yarken-managed Advanced Integration.

  • An Advanced Pipeline.

  • A supported Yarken destination.

Available Advanced Integrations depend on what Yarken has enabled in your environment.

How BASIC integrations work

A BASIC integration typically follows these steps.

Step 1: Configure a connection

A connection stores the credentials and configuration Yarken needs to access the source.

Depending on the source, you may configure:

  • Cloud storage access.

  • Custom API credentials.

  • Platform credentials.

  • A supported consent flow.

For setup instructions, see Connections.

Step 2: Configure a Data Mapping Template

A Data Mapping Template maps fields in a source file to Yarken fields.

File-based pipelines use the selected template each time they process a file. The selected Destination Table helps determine where the loaded data appears in Yarken.

Some API pipelines use predefined mappings and do not require a custom template.

For setup instructions, see Data Mapping Templates.

Step 3: Configure Automate or create a pipeline

Use Automate for a simplified recurring file-ingestion workflow.

Use Pipelines when you need to configure or manage the complete workflow.

A pipeline can define:

  • The source connection.

  • The source folder or API data.

  • The Data Mapping Template, where required.

  • The schedule.

  • The pipeline name and description.

See Automate or Create BASIC pipelines.

Step 4: Activate and run the pipeline

New BASIC pipelines are inactive by default.

Review the connection, mapping, and schedule before you activate the pipeline. An inactive pipeline does not run on schedule.

You can also select Run Once to start an immediate run.

Step 5: Monitor and validate the result

Use View Run History to review:

  • Run status.

  • Execution date and time.

  • Files or records processed.

  • Error details.

  • Processing duration.

After the run succeeds, verify the loaded data in its destination feature.

For more information, see Monitor pipeline runs.

Supported BASIC integrations

Cloud storage

  • Azure Blob Storage

  • AWS S3

  • Google Cloud Storage

APIs

  • Microsoft Graph API

  • Power BI API

  • Google API

  • Google Reseller API

The available connection and pipeline options may vary by environment.

How ADVANCED integrations work

Advanced Integrations use Yarken-managed workflows to exchange data with supported external systems.

A typical workflow may include:

  1. An Advanced Connection authorizes Yarken to access the external provider.

  2. The Yarken-managed Advanced Integration retrieves and processes the supported data.

  3. The Advanced Pipeline runs the integration and records execution activity.

  4. The processed data is made available in the supported Yarken destination.

Yarken deploys the required Advanced Integration components during implementation.

Your administrator may need to:

  • Prepare the provider credentials and permissions required by the integration.

  • Authorize the Advanced Connection.

  • Run, activate, or deactivate the Advanced Pipeline where supported.

  • Monitor execution history.

  • Validate the results.

Not every Advanced Integration uses every component.

For more information, see Advanced integrations.

Supported ADVANCED integrations

  • Google Cloud Storage

  • Azure Blob Storage

  • AWS S3

  • Azure AI Foundry

  • Anthropic

  • GitHub Copilot

  • Cursor

  • LiteLLM

  • AWS Recommendations

The available integrations depend on what Yarken has enabled for your environment.

After data is loaded

Where data appears depends on the selected Destination Table, the file or API data type, and the target Yarken feature.

After a pipeline completes successfully:

  1. Review the execution status.

  2. Confirm that Yarken processed the expected files or records.

  3. Open the feature associated with the destination data type.

  4. Select the applicable entity, year, and month.

  5. Confirm that the expected data is available.

  6. Refresh the affected cube when you need immediate reporting results.

  7. Review the data in the relevant report, dashboard, or Analytics data area.

For file-based loads, you can also review the UPLOADED FILES tab. Files loaded through pipelines are marked as Interface in the Uploaded By column.

Refresh the affected cube to review newly loaded data immediately. Otherwise, Yarken performs an automatic refresh within 24 hours of the last data change.

Cube refresh and best practices for Pipelines

Related content