Creating Pipelines

General prerequisites

These apply to all Cloud Storage Pipelines:

• You must have the admin rights within the application to create and manage pipelines.

• The data you plan to import must follow a supported format, such as CSV, GZ, or Parquet.

• The data mapping templates should be define. For more information about the data mapping template, refer Data Mapping Templates.

• Required Connections must be configured in advance for the selected cloud storage or API source. For more information on how to configure Cloud Storage connection, refer Creating a new Pipeline Connection | Create Cloud Storage Connection (manual only).

Create a Cloud Storage Pipeline

Cloud Storage Pipelines allow you to automatically fetch usage and billing files from cloud object storage and import them into the application. During setup, you select the storage provider, configure access, choose a data mapping template, define the schedule, and assign a pipeline name. Once created, the pipeline runs automatically based on the configured frequency.

Azure Blob Storage

Prerequisites

Before creating an Azure Cloud Storage Pipeline, ensure you have:

• A valid Azure Storage account with Blob Storage enabled.

• An Azure connection configured in YäRKEN using:

  • Storage account name

  • Access key

• Access to the container and folder path where usage files are stored.

• Azure usage files available in Blob Storage, such as:

  • Azure Usage Details or

  • Azure Cost Management Export

• A data mapping template to define how Azure file fields map to YäRKEN fields.

• File location details, including container name and file path or naming convention.

Procedure

To create,

1. Navigate to Admin > Pipelines > Pipelines.

2. On the Pipelines page, click + NEW PIPELINE.

3. On the Create New Pipeline dialog box, choose the storage type as Azure Blob and click NEXT.

4. Select an existing configured Connection or create a new. For more information on how to configure a connection prior to creating the pipeline, refer Creating a new Pipeline Connection | Create Cloud Storage Connection (manual only).

   

   1. (Optional) To create a new connection, click + ADD NEW and enter the below details on Create a new connection dialog box:

      1. Name: The name of the connection.

      2. Account Name: The Azure Storage Account Name for accessing the storage container.

      3. Access Key: The Azure Storage Account Access Key for accessing the storage account data.

      4. Container Name: The name of the container within the Azure Storage Account where the exported files are stored, for example, billing.

      5. Click SAVE.

5. Enter a Directory Name from which you want to retrieve files and select NEXT.

6. Select a Data Mapping Template from the dropdown list and click NEXT. For more information on Data Mapping Template, refer Data Mapping Templates.

7. On Schedule tab, select Frequency for the pipeline to run from the dropdown list and then click NEXT:

   1. Hourly: Select the hour(s) between 0 to 12 for the pipeline to start run in the field labelled. The run will occur at your defined intervals.

   2. Daily: Occurs every day, starting at 12:00 AM.

   3. Monthly: Specify the day of the month and select the month interval.

9. Enter the Pipeline Name and click SAVE.

The new pipeline is created in the application and remains deactivated by default. You must activate the pipeline to start scheduled or on-demand runs.

AWS S3

Prerequisites

Before creating an AWS Cloud Storage Pipeline, ensure you have:

• A valid AWS S3 account with access to the required bucket.

• An AWS connection configured in YäRKEN using:

  • Access key

  • Secret key

• Access to the S3 bucket and folder path where usage files are stored.

• AWS usage files exported to S3, such as:

  • AWS Cost and Usage Report (CUR)

• A data mapping template to map fields from AWS usage files to YäRKEN fields.

• File location details, including folder path and/or file naming patterns.

Procedure

To create,

1. Navigate to Admin > Pipelines > Pipelines.

2. On the Pipelines page, click + NEW PIPELINE.

3. On the Create New Pipeline dialog box, choose the storage type as AWS S3 and click NEXT.

4. Select an existing configured Connection or create a new. For more information on how to configure a connection prior to creating the pipeline, refer Creating a new Pipeline Connection | Create Cloud Storage Connection (manual only).

   

   1. (Optional) To create a new connection, click + ADD NEW and enter the below details on Create a new connection dialog box:

      1. Name: The name of the connection.

      2. Access Key: The Azure Storage Account Access Key for accessing the storage account data.

      3. Secret Key:  The AWS Secret Key used along with the Access Key to access the S3 storage bucket.

      4. Bucket Name: The name of the S3 bucket where the exported files are being stored, for example, yarken-billing.

      5. Click SAVE.

5. Enter a Directory Name from which you want to retrieve files and select NEXT.

6. Select a Data Mapping Template from the dropdown list and click NEXT. For more information on Data Mapping Template, refer Data Mapping Templates.

7. On Schedule tab, select Frequency for the pipeline to run from the dropdown list and then click NEXT:

   1. Hourly: Select the hour(s) between 0 to 12 for the pipeline to start run in the field labelled. The run will occur at your defined intervals.

   2. Daily: Occurs every day, starting at 12:00 AM.

   3. Monthly: Specify the day of the month and select the month interval.

9. Enter the Pipeline Name and click SAVE.

The new pipeline is created in the application and remains deactivated by default. You must activate the pipeline to start scheduled or on-demand runs.

Google Cloud Storage

Prerequisites

Before creating a Google Cloud Storage Pipeline, ensure you have:

• A valid Google Cloud Storage (GCS) bucket.

• A Google Cloud Storage connection configured in YäRKEN using:

  • Access key

  • Secret key

• Access to the GCS bucket and folder path containing usage files.

• GCP usage files exported to GCS, such as:

  • GCP Billing Export files

• A data mapping template to map GCP file fields to YäRKEN fields.

• File location details, including bucket path and file naming patterns.

Procedure

To create,

1. Navigate to Admin > Pipelines > Pipelines.

2. On the Pipelines page, click + NEW PIPELINE.

3. On the Create New Pipeline dialog box, choose the storage type as Google Cloud Storage and click NEXT.

4. Select an existing configured Connection or create a new. For more information on how to configure a connection prior to creating the pipeline, refer Creating a new Pipeline Connection | Create Cloud Storage Connection (manual only).

   

   1. (Optional) To create a new connection, click + ADD NEW and enter the below details on Create a new connection dialog box:

      1. Name: The name of the connection.

      2. Access Key: The access key used to authenticate and authorize the access to the storage service.

      3. Secret Key: A secret key associated with the access key used to authenticate and authorize secure access to the storage service.

      4. Bucket Name: The name of the GCS bucket where the exported files (objects) are being stored, for example, yarken-billing.

      5. Click SAVE.

5. Enter a Directory Name from which you want to retrieve files and select NEXT.

6. Select a Data Mapping Template from the dropdown list and click NEXT. For more information on Data Mapping Template, refer Data Mapping Templates.

7. On Schedule tab, select Frequency for the pipeline to run from the dropdown list and then click NEXT:

   1. Hourly: Select the hour(s) between 0 to 12 for the pipeline to start run in the field labelled. The run will occur at your defined intervals.

   2. Daily: Occurs every day, starting at 12:00 AM.

   3. Monthly: Specify the day of the month and select the month interval.

9. Enter the Pipeline Name and click SAVE.

The new pipeline is created in the application and remains deactivated by default. You must activate the pipeline to start scheduled or on-demand runs.

Create an API Pipeline manually

As part of the consent-link process, when a tenant accepts the consent links, the application automatically creates the API connections and their corresponding API pipelines. Use the below section to create an API Pipelines manually.

Microsoft Graph API

The Microsoft Graph API pipeline is used to collect:

• License details

• Product usage details

• Entra ID activity details

• Teams activity details

• Defender enrolled device details

Prerequisite

• A Microsoft Graph API connection must already be configured in the application.

Procedure

To configure,

1. Navigate to Admin > Pipelines > Pipelines.

2. On the Pipelines page, click + NEW PIPELINE.

3. On the Create New Pipeline dialog box, select Microsoft Graph API and click NEXT.

4. Under Select a connection, click the dropdown icon and select already configured API connection for Microsoft Graph API.

5. Select the API data to sync and click NEXT:

   1. License Details

   2. Product Usage Details

   3. Entra ID Activity Details

   4. Teams Activity Details

   5. Defender Enrolled Details

6. On the Schedule tab, choose Monthly frequency and select Day 7 of every 1 month (recommended).

7. Click NEXT.

8. On the Details tab, enter a Pipeline Name and click SAVE.

9. The new pipeline is created in a deactivated state. Activate it using the Activate toggle and click OK on confirmation popup.

10. Once activated, the pipeline runs on the scheduled timeframe. You may also use Run Once to trigger an immediate execution.

The YäRKEN application immediately starts fetching data. You can click View Run History icon to check the last running status.

Power BI API

For Power BI, two separate pipelines must be created using the same Power BI connection:

• One for daily usage details

• One for monthly workspace details.

Prerequisite

• A Power BI API connection must already be configured.

Procedure

Pipeline 1: Power BI Usage Details (Daily)

To configure,

1. Navigate to Admin > Pipelines > Pipelines.

2. On the Pipelines page, click + NEW PIPELINE.

3. On the Create New Pipeline dialog box, select Power BI API and click NEXT.

4. Under Select a connection, click the dropdown icon and select already configured API connection for Power BI API.

5. Select Power BI Usage Details to sync and click NEXT.

6. On the Schedule tab, select Daily frequency.

7. Click NEXT.

8. On the Details tab, enter a Pipeline Name and click SAVE.

9. The new pipeline is created in a deactivated state. Activate it using the Activate toggle and click OK on confirmation popup.

10. Once activated, the pipeline runs on the scheduled timeframe. You may also use Run Once to trigger an immediate execution.

The YäRKEN application immediately starts fetching data. You can click View Run History icon to check the last running status.

Pipeline 2: Power BI Workspace Details (Monthly)

Procedure

To configure,

1. Navigate to Admin > Pipelines > Pipelines.

2. On the Pipelines page, click + NEW PIPELINE.

3. On the Create New Pipeline dialog box, select Power BI API and click NEXT.

4. Under Select a connection, click the dropdown icon and select already configured API connection for Power BI API.

5. Select Power BI Workspace Details to sync and click NEXT.

6. On the Schedule tab, select Monthly frequency and choose Day 7 of every 1 month.

7. Click NEXT.

8. On the Details tab, enter a Pipeline Name and click SAVE.

9. The new pipeline is created in a deactivated state. Activate it using the Activate toggle and click OK on confirmation popup.

10. Once activated, the pipeline runs on the scheduled timeframe. You may also use Run Once to trigger an immediate execution.

The YäRKEN application immediately starts fetching data. You can click View Run History icon to check the last running status.

Google Cloud API

The Google Cloud API pipeline is used to collect:

• Sign-in details

• Product usage details

Prerequisite

• A Google API connection must be configured before creating the pipeline.

Procedure

To configure,

1. Navigate to Admin > Pipelines > Pipelines.

2. On the Pipelines page, click + NEW PIPELINE.

3. On the Create New Pipeline dialog box, select Google Cloud API and click NEXT.

4. Under Select a connection, click the dropdown icon and select already configured API connection for Google Cloud API.

5. Select the API data to sync and click NEXT:

   1. Sign In Details

   2. Product Usage Details

6. On the Schedule tab, select Monthly frequency and choose Day 7 of every 1 month (recommended).

7. Click NEXT.

8. On the Details tab, enter a Pipeline Name and click SAVE.

9. The new pipeline is created in a deactivated state. Activate it using the Activate toggle and click OK on confirmation popup.

10. Once activated, the pipeline runs on the scheduled timeframe. You may also use Run Once to trigger an immediate execution.

The YäRKEN application immediately starts fetching data. You can click View Run History icon to check the last running status.

Related content

Pipelines

Managing Pipelines

Creating a new Pipeline Connection | Create API Connection using Consent URL

Creating a new Pipeline Connection | Create API Connection manually

Creating a new Pipeline Connection | Create Cloud Storage Connection (manual only)