Breadcrumbs

Pipelines

Use Pipelines to automate recurring data ingestion from supported cloud storage providers, APIs, and Advanced Integrations. A pipeline controls how Yarken retrieves, processes, schedules, and monitors incoming data.

Go to Admin > Pipelines > Pipelines to work with BASIC and ADVANCED pipelines.

A pipeline uses an existing data source configuration. BASIC pipelines use a BASIC Connection. File-based BASIC pipelines also use a Data Mapping Template. ADVANCED pipelines are deployed as part of an enabled Advanced Integration.

What you can do

Use Pipelines to:

  • Retrieve files from supported cloud storage providers.

  • Retrieve supported usage, licensing, and other data through APIs.

  • Apply reusable Data Mapping Templates to file-based ingestion.

  • Schedule recurring ingestion to run hourly, daily, or monthly where supported.

  • Run a BASIC pipeline on demand with Run Once.

  • Activate or deactivate scheduled ingestion without recreating the pipeline.

  • Edit supported pipeline configuration when source or scheduling requirements change.

  • Review run history, processed records, and error details.

  • Review Advanced Pipelines that Yarken has enabled for your environment.

  • Validate loaded data and refresh affected cubes for reporting.

Before you begin

The prerequisites depend on the pipeline type.

Pipeline type

Before you create or use it

BASIC cloud storage pipeline

Configure a BASIC cloud storage connection. Prepare the source folder and a Data Mapping Template for the destination data type.

BASIC API pipeline

Configure or authorize the supported API connection. Consent-based setup can create the associated pipeline automatically.

ADVANCED pipeline

Yarken must enable and deploy the Advanced Integration. Your administrator then authorizes the required Advanced Connections, manages the exposed Advanced pipeline controls, and validates the results.

The pipeline types and integrations available to you depend on your Yarken environment.

Choose BASIC or ADVANCED

Method

Use it when

Typical examples

BASIC

You need a native Yarken pipeline for a supported cloud storage provider or API.

Azure Blob Storage, AWS S3, Google Cloud Storage, Microsoft Graph API, Power BI API, and supported Google APIs.

ADVANCED

You need a Yarken-managed integration or provider-specific workflow enabled for your environment.

Cloud ingestion workflows, Azure AI Foundry metrics, OpenAI API Platform usage, ChatGPT Workspace cost and usage, Cursor usage, LiteLLM usage, and AWS Recommendations.

BASIC pipelines

Select the BASIC tab to create and manage native Yarken pipelines.

Cloud storage pipelines

Use a cloud storage pipeline when the source data is stored as files.

Supported storage integrations include:

  • Azure Blob

  • AWS S3

  • Google Cloud Storage

A cloud storage pipeline typically uses:

  • A BASIC cloud storage connection.

  • A source folder or directory.

  • A Data Mapping Template.

  • A recurring schedule.

Keep each automated folder aligned to the expected file type and mapping. A file with an unexpected structure can fail validation or load incomplete data.

API pipelines

Use an API pipeline to retrieve supported structured data without handling source files.

Supported API integrations include:

  • Microsoft Graph API

  • Power BI API

  • Google Cloud API

Supported API pipelines use predefined mappings. You do not select a custom Data Mapping Template during API pipeline setup.

For supported APIs, the consent flow can create the connection and associated pipeline automatically. Microsoft Graph API and Power BI API also support manual configuration when required. Google API connections use the supported consent flow.

Create a BASIC pipeline

Create a BASIC pipeline when you want Yarken to retrieve data from a supported cloud storage provider or API.

Before you start:

  • Configure the required Connection.

  • Prepare a Data Mapping Template for file-based pipelines.

  • Confirm the source folder or API data you want to retrieve.

  • Decide how often the pipeline should run.

For provider requirements and step-by-step setup for Azure Blob Storage, AWS S3, Google Cloud Storage, Microsoft Graph API, Power BI API, and Google Cloud API, see Create BASIC pipelines.

New BASIC pipelines are inactive by default. Activate the pipeline before you expect it to run on schedule.

Manage BASIC pipelines

After you create a pipeline, you can manage it from the BASIC tab.

You can:

  • Edit the pipeline configuration.

  • Run the pipeline immediately with Run Once.

  • Activate or deactivate scheduled ingestion.

  • Review pipeline execution.

  • Delete pipelines you no longer need.

For detailed management steps, see Manage BASIC pipelines.

Deactivating a pipeline stops scheduled ingestion without deleting its configuration.

Run Once starts an immediate execution and does not change the existing schedule.

ADVANCED pipelines

Select the ADVANCED tab to review pipelines delivered through Advanced Integrations.

Yarken deploys and maintains the Advanced Integration for your environment. Your administrator authorizes the required Advanced Connections, manages the exposed Advanced pipeline controls, monitors execution, and validates the results.

Advanced Pipelines may include workflows such as:

image-20260924-103338.png
Review Advanced Pipelines enabled for your environment

An Advanced Integration can include:

  • An Advanced Connection.

  • A Yarken-managed Advanced Integration.

  • An Advanced Pipeline.

  • A supported Yarken destination.

For Advanced file-based integrations, Yarken automatically applies the required template as part of the deployed Advanced Integration. You do not need to configure it, and you cannot edit the template.

For setup, authorization, and automation guidance, see Advanced integrations.

Monitor pipeline runs

Use pipeline monitoring to confirm that ingestion completed and investigate failed or long-running runs.

You can review:

  • View Run History for individual pipeline executions.

  • Background Processes for running and completed background activity.

  • Audit Logs for configuration changes.

For detailed monitoring guidance, see Monitor pipeline runs.

How Yarken handles cloud storage files

When a cloud storage pipeline processes a file, Yarken moves the file based on the processing result.

  • Processed — Contains files that Yarken ingested successfully.

  • Error — Contains files that Yarken could not process because of validation, format, or mapping issues.

Do not upload, modify, or reuse files in the Processed or Error folders. If a file moves to Error, correct the source issue and reprocess a corrected file.

After a pipeline runs

After the pipeline completes successfully:

  1. Review the run status and processed file or record count.

  2. For file-based pipelines, confirm that Yarken moved the processed files to the expected folder.

  3. Open the Yarken feature associated with the destination data type.

  4. Select the applicable entity, year, and month.

  5. Verify that the expected records are available.

  6. Refresh the affected cube when you need immediate reporting results.

  7. Review the data in the relevant report, dashboard, or Analytics data area.

A successful pipeline run confirms ingestion completed. It does not always mean the latest data is immediately visible in reporting. Refresh the affected cube when you need to validate reporting results without waiting for the automatic refresh.

For current refresh guidance, see Cube refresh and best practices for Pipelines.

Pipeline best practices

  • Use separate pipelines for different data domains to reduce overlap and simplify troubleshooting.

  • Keep file structures and source folders stable for recurring ingestion.

  • Validate the Data Mapping Template after the first file-based run.

  • Use Run Once to test a configuration before relying on the next scheduled run.

  • Review run history after changes to connections, mappings, folders, or schedules.

  • Monitor failed or long-running executions and resolve errors before the next scheduled run.

  • Refresh the affected cube when you need to validate newly ingested data immediately.

Troubleshoot pipelines

Issue

What to check

The pipeline did not run on schedule

Confirm that the pipeline is active, the schedule is correct, and the source connection is valid.

The pipeline is running but has not completed

Review Background Processes and the available run details for long-running activity or errors.

No file was processed

Confirm that a new file exists in the configured source folder and matches the expected file structure.

A file moved to the Error folder

Review required fields, the Data Mapping Template, file format, and schema consistency. Correct the source file before reprocessing it.

An API pipeline does not return data

Verify the API connection, consent or credentials, selected API data, and run history.

The run succeeded, but data is missing from reports

Verify the destination data first, then refresh the affected cube if you need immediate reporting results.

An ADVANCED pipeline is inactive or failing

Verify the required Advanced Connection and review the Advanced pipeline status and run history. Follow the guide for that Advanced Integration.

For additional troubleshooting, see Troubleshoot Pipelines and integrations.

Related content