Create BASIC pipelines

Use this guide to create BASIC pipelines for supported cloud storage providers and APIs.

A pipeline controls how Yarken retrieves and processes data from an existing source. The setup differs depending on whether you use cloud storage or an API.

Before you begin

Before you create a pipeline:

• Confirm that you have access to create and manage pipelines.

• Configure the required source connection, or create the connection during pipeline setup where supported. For connection setup, see Connections.

• For file-based pipelines, prepare a supported source file. Supported formats include CSV, GZ, and Parquet.

• For file-based pipelines, prepare the required Data Mapping Template. For mapping guidance, see Data Mapping Templates.

• Confirm the source folder, API data type, and schedule you want to use.

Choose the pipeline type

Create a cloud storage pipeline

Cloud storage pipelines retrieve files from supported cloud storage providers and load them into Yarken on a recurring schedule.

Before you create a pipeline, review the requirements for your storage provider. Then follow the common setup workflow.

Provider-specific requirements

Review the requirements for your cloud storage provider before you start the pipeline setup. The connection details, source location, and authentication requirements vary by provider.

Azure Blob Storage

AWS S3

Google Cloud Storage

Common setup workflow

Step 1: Open Pipelines

1. Go to Admin > Pipelines > Pipelines.

2. Select the BASIC tab.

3. Select + NEW PIPELINE.

   

Step 2: Select the storage provider

Select one of the supported storage integrations:

• Azure Blob

• AWS S3

• Google Cloud Storage

Step 3: Select the connection

Select an existing connection for the provider.

If the required connection does not exist, create one during setup where available. Use Create BASIC connections for provider-specific authentication fields and permissions.

Step 4: Enter the source directory

Enter the Directory Name that contains the files Yarken should retrieve.

Step 5: Select the Data Mapping Template

Select the template that maps the source file fields to the required Yarken destination fields.

• Review the template before using it for a new source format.

• For Expenses (spend) ingestion, review the Delete before load option when it is available for the source setup.

  

Step 6: Select the Date Format for Cloud Consumption mapping template

When you select a Cloud Consumption Data Mapping Template, Yarken adds a Date Format step.

Select the date format used in the FOCUS source file. Yarken saves the selected format with the Data Mapping Template and uses it when processing files with that template.

Step 7: Configure the schedule

On the Schedule tab, select the required frequency:

• Hourly — Configure the required hourly interval.

• Daily — Runs every day starting at 12:00 AM.

• Monthly — Select the day of the month and month interval.

Step 8: Enter the pipeline details

1. On the Details tab, enter a clear Pipeline Name.

2. Select SAVE.

Yarken creates the pipeline in an inactive state.

Step 9: Activate and validate the pipeline

1. Review the pipeline configuration.

2. Use the activation toggle to activate the pipeline.

3. Confirm the activation.

4. Use Run Once if you want to test the pipeline immediately.

5. Use View Run History to confirm that the first run completed successfully.

Set up an API pipeline using consent

For supported APIs, the consent workflow is the recommended setup path.

When the tenant administrator grants consent and the option to create the pipeline is selected, Yarken can create the connection and corresponding API pipeline automatically.

The consent workflow can:

• Authorize the required API access.

• Create the supported connection.

• Configure the predefined API mapping.

• Create the associated pipeline.

For consent-based connection setup, see Create an API connection using consent.

Recommended schedules for API pipelines

Some supported integrations have a recommended pipeline schedule. Use the recommended cadence unless your implementation requires a different configuration.

Create an API pipeline manually

Create an API pipeline manually only when the required connection already exists and the pipeline was not created through the consent workflow.

Common manual setup workflow

Step 1: Open Pipelines

1. Go to Admin > Pipelines > Pipelines.

2. Select the BASIC tab.

3. Select + NEW PIPELINE.

Step 2: Select the API integration

Select the required API:

• Microsoft Graph API

• Power BI API

• Google Cloud API

Step 3: Select the connection

Select the existing API connection for the integration.

Step 4: Select the API data

Select the data set that Yarken should retrieve. The available options depend on the selected API.

Step 5: Configure the schedule

Select the required schedule for that API data set.

Step 6: Enter the pipeline details

1. Enter a clear Pipeline Name.

2. Select SAVE.

Step 7: Activate and validate the pipeline

1. Activate the new pipeline.

2. Use Run Once when you want an immediate test run.

3. Use View Run History to verify the execution.

API-specific data to sync

Microsoft Graph API

Power BI API

Google Cloud API

After you create a pipeline

After you save and activate the pipeline:

1. Use Run Once when you want to validate the configuration immediately.

2. Use View Run History to confirm successful execution.

3. For file-based pipelines, confirm that Yarken processed the expected file.

4. Open the Yarken feature associated with the destination data type.

5. Refresh the affected cube when you need immediate reporting results.

For ongoing changes and lifecycle actions, see Manage BASIC pipelines.

For execution monitoring, see Monitor pipeline runs.

Related content

• Pipelines

• Connections

• Data Mapping Templates

• Automate

• Manage BASIC pipelines

• Monitor pipeline runs