Integrations, Pipelines & Data Automation — FAQs and Troubleshooting

Use this page for customer questions and troubleshooting related to connecting Yarken to external systems, creating and managing connections, building pipelines, using data mapping templates, automating recurring file ingestion, monitoring pipeline execution, and resolving integration issues.

Understanding integrations

Q: What are integrations in Yarken?

A: Yarken uses integrations to bring in data from cloud platforms, storage locations, and APIs. Yarken can also send data and trigger actions back to connected systems where supported.

Once data enters Yarken through an integration, it can be mapped, allocated, reported, analyzed, governed, and used in planning and optimization workflows — including TBM allocation, FinOps analysis, Planning, Analytics, License Usage, Insights, Recommendations, Cost Explorer, dashboards, and executive reporting.

Q: What is the relationship between connections, pipelines, and recipes?

A: These four components work together to move data between Yarken and external systems:

A pipeline depends on a connection, but a connection can be reused across multiple pipelines.

Q: What role does Workato play in Yarken integrations?

A: Yarken uses Workato to ingest information and data from third-party tools and applications into Yarken. Workato handles connecting to source systems, managing supported triggers and actions, and moving data into the right Yarken workflow.

In practice, administrators work with Yarken connections, pipelines, mappings, and related setup inside the product. Workato powers the underlying integration flow where supported. The connection is the controlled access point between Yarken and the external system, while Workato helps orchestrate the ingestion and automation behind the scenes.

Q: Which roles can manage integrations, connections, and pipelines?

A: The Admin role has full access to create, configure, and manage connections, pipelines, and automation. The Cost Model Manager role also has access to pipeline and integration management as part of its broader admin-level access to data and model management. Non-admin users cannot access the Pipelines or Connections screens.

Connections

Q: What connection types does Yarken support?

A: Yarken supports two main connection categories:

Cloud storage connections must be configured manually. API connections can be created manually or automatically through a secure consent URL flow. Unlike some supported API flows, cloud storage connections do not automatically create pipelines — the related pipeline still needs to be configured separately.

Q: What are the prerequisites for API connections?

A: Important prerequisites for API connections:

• Google API: Requires Google Workspace Super Admin. Admin SDK API and Enterprise License Manager API must be enabled

• Microsoft Graph and Power BI API: Requires a Microsoft Entra (Azure AD) tenant with Global Administrator or Privileged Role Administrator consent

• Power BI: Requires a Microsoft Entra Security Group with the application added as a member, and service principal access (read-only) enabled in the Power BI Tenant Admin portal

Q: What is the difference between Custom Credentials and Platform Credentials for API connections?

A: For Microsoft Graph and Power BI API connections, you can choose how authentication is handled:

Q: Can a connection be reused across multiple pipelines?

A: Yes. A connection stores the credentials and configuration once. Multiple pipelines can reference the same connection. For example, a single AWS S3 connection can be used by AWS pipelines for cloud billing, spend, and budget file ingestion.

Pipelines

Q: What types of pipelines does Yarken support?

A: Yarken supports two pipeline types:

Cloud storage pipelines require a data mapping template. API pipelines use predefined mappings because the API structure is standardized.

Q: What data does each API pipeline collect?

A: Each API pipeline collects specific data types:

Q: Are API pipelines created automatically after consent?

A: Yes. When a user accepts the application via the consent link, the system automatically creates the required pipelines for Microsoft Graph API, Power BI API, and Google API (depending on which links are used). The pipelines are triggered to run immediately for initial ingestion without manual intervention.

By default, newly created API pipelines are scheduled to run monthly on the 7th. Existing daily and hourly schedules are left unchanged.

Q: What scheduling options are available for pipelines?

A: Pipelines support three scheduling frequencies:

You can also trigger a pipeline manually using the Run Once option for instant data refresh. On-demand runs do not affect the existing schedule.

Q: How do I identify files loaded via pipelines vs. manual uploads?

A: Files ingested through pipelines can be identified in the UPLOADED FILES tab. Check the Uploaded By column — files loaded via pipelines are marked as Interface.

Data mapping templates

Q: What are data mapping templates and why are they important?

A: Data mapping templates define how the fields (column headers) in your source files map to the destination fields in Yarken. They ensure consistent and accurate loading of data across all manual or automated uploads.

Once a mapping template is saved, it becomes the reference point for all future uploads for that file type. All automated cloud storage pipelines require a data mapping template — without one, the pipeline cannot process files. API pipelines use predefined mappings and do not require a custom template.

Q: Can I edit or delete a data mapping template?

A: Yes, with restrictions:

Editing:

• You can edit templates when file structures change

• Exception: Mapping fields for Cloud Consumption file types (Azure, AWS, FOCUS) cannot be added or modified manually — these mappings are system-defined due to their standardized structure

• All other file types can be edited, even if they are already in use by active pipelines

Deleting:

• A data mapping template that is currently used by a pipeline (active or inactive) cannot be deleted

• To delete it, you must first remove or update the pipeline that references the template

Automate

Q: What is Automate and how does it differ from Pipelines?

A: Automate simplifies setting up recurring file ingestion from cloud storage. It provides a centralized view of all file types that support automated uploads — such as actual spend, budget allocation, cost pools, assets, metrics, cloud consumption files (Azure, AWS, FOCUS), and other file types.

Automate creates a pipeline behind the scenes. You manage, deactivate, or delete the automation from the Pipelines screen.

Q: What file naming convention should I follow for automated uploads?

A: For monthly upload-supported data (such as Actual Spend, Budget, Cloud Consumption, Assets), the file name must end with the _YYYYMM format (for example, ActualSpend_202512.csv). This helps Yarken automatically associate the file with the correct month and year during upload.

For non-monthly uploads (such as Master Data), you do not need to follow the year-month naming format.

Q: What happens if I delete an uploaded file — does it stop automation?

A: No. Deleting an uploaded file from the UPLOADED FILES tab does not impact automation. Automation continues to run on schedule. However, if the data mapping template associated with that file type is deleted, automation cannot run until a new mapping is created.

Q: How do I deactivate automation for a file type?

A: You cannot disable automation directly from the Automate screen. To deactivate:

1. Navigate to Admin → Pipelines → Pipelines

2. Find the pipeline associated with the automation

3. Disable or delete the pipeline

The Automate icon turns grey only after the linked pipeline is revoked.

Monitoring pipelines

Q: How do I monitor pipeline execution?

A: Yarken provides three monitoring tools:

View Run History and Background Processes focus on execution behavior. Audit Logs focus on configuration changes.

Q: How does Yarken handle files in cloud storage after processing?

A: After each pipeline run, Yarken automatically organizes files in the configured storage location:

Important guidelines:

• Do not upload, modify, or reuse files in the processed or error folders

• If files appear in the Error folder, review the pipeline logs, correct the issue, and reprocess using corrected files placed in the source folder (not the error folder)

• The Processed and Error folders act as an audit trail for file-level ingestion

Cube refresh after pipeline ingestion

Q: Do I need to refresh cubes after a pipeline run?

A: Yes. After a pipeline successfully completes data ingestion, the corresponding cubes must be refreshed to ensure the latest data is available in dashboards and reports.

If no manual action is taken, data will appear in dashboards after the next automatic refresh cycle. Cubes that have pending data changes move to Pending status on the Cube Refresh page.

Best practices

Q: What are the recommended best practices for integrations and pipelines?

A: Follow these practices for reliable integration outcomes:

Troubleshooting Integrations, Pipelines & Data Automation

Pipeline did not ingest data — no new records after scheduled run

Cause: No new files exist in the configured folder, the pipeline is inactive, or the schedule has not triggered.

Resolution:

1. Confirm new files exist in the configured folder — the pipeline only processes files it has not already ingested

2. Verify the pipeline is active — newly created pipelines are deactivated by default. Check the status indicator on the Pipelines page

3. Check the pipeline schedule — confirm the frequency and next run time

4. Check Background Processes for active or stalled runs

5. Verify the connection is valid — expired credentials or revoked access will prevent the pipeline from reaching the source

6. If using Automate, confirm the Automate icon is blue (active) — grey means inactive

Files moved to the Error folder after pipeline run

Cause: The files failed validation due to unsupported file extensions, invalid file structure, corrupted files, or mapping mismatches.

Resolution:

1. Navigate to the error folder in your cloud storage and identify the rejected files

2. Review the pipeline View Run History for error details

3. Check the data mapping template — verify required fields are present and correctly mapped

4. Validate the file format and schema consistency — ensure headers match the expected template

5. For monthly data files, verify the file name ends with _YYYYMM format

6. Correct the file and place the corrected version in the source folder (not the error folder) for reprocessing

7. Do not modify or reuse files in the error folder directly

Pipeline is running but not completing — appears stuck

Cause: The pipeline is processing a large volume of data, or an error occurred during processing that did not trigger a failure status.

Resolution:

1. Review Background Processes for the pipeline's execution status

2. Check the pipeline logs for long-running steps or errors

3. If the pipeline has been running for an unusually long time, check the source file size — very large files may take longer to process

4. Verify the connection is still valid — a mid-run authentication failure can cause the pipeline to stall

5. If the pipeline remains stuck, deactivate it, wait for the current run to complete or time out, then reactivate and trigger a manual run

Missing data in dashboards or reports after successful pipeline run

Cause: The pipeline completed successfully but the affected cube has not been refreshed.

Resolution:

1. Confirm the pipeline run completed successfully in View Run History

2. Navigate to the Cube Refresh page and check for pending cubes

3. Trigger a manual cube refresh for the affected cube to see updates immediately

4. If no manual action is taken, Yarken performs an automatic cube refresh within 24 hours

5. After the cube refresh, verify the data appears in the expected dashboards and reports

6. If data is still missing after refresh, check the data mapping template — fields may not be mapped to the correct Yarken fields

Connection authentication failing — pipeline cannot reach source system

Cause: Credentials have expired, been revoked, or the source system permissions have changed.

Resolution:

1. Navigate to Admin → Pipelines → Connections

2. Select the affected connection and verify the credentials:

   • Cloud storage: Check Access Key, Secret Key, Account Name, Container/Bucket Name, Region (for AWS S3)

   • API (Custom Credentials): Check Tenant ID, Client ID, Client Secret

   • API (Platform Credentials): Re-authorize via consent URL

3. Click TEST CONNECTION to validate the updated credentials

4. For API connections, verify that the consent has not been revoked in the source identity platform (e.g., Microsoft Entra ID → Enterprise Applications)

5. For Power BI API connections, verify that service principal support is enabled in the tenant settings and the application is a member of the required security group

6. Check the Consent Available column on the Connections page — if it shows no valid consent, re-run the consent flow

7. Important: Editing credentials immediately affects scheduled runs

Data mapping mismatch — fields not aligning after pipeline run

Cause: The source file schema has changed, or the data mapping template does not match the current file structure.

Resolution:

1. Open the data mapping template associated with the pipeline

2. Compare the template's source fields with the actual file headers

3. If the source file schema has changed (new columns, renamed columns, removed columns), update the mapping template to reflect the changes

4. Note: Cloud Consumption file type mappings (Azure, AWS, FOCUS) cannot be modified manually — they are system-defined

5. Re-run the pipeline after updating the mapping

6. Validate the first run's output to confirm fields align correctly with Yarken's data model

Pipeline not triggering on schedule

Cause: The pipeline is deactivated, the schedule is misconfigured, or the system did not detect new files.

Resolution:

1. Verify the pipeline is active — check the status indicator on the Pipelines page

2. Review the schedule configuration — confirm the frequency (Hourly, Daily, Monthly) and the specific timing

3. For cloud storage pipelines, confirm new files exist in the configured folder — the pipeline does not run if no new files are detected

4. Check Audit Logs to see if the pipeline was recently disabled or modified

5. If the pipeline was recently created via consent URL, verify the default schedule (monthly on the 7th) matches your expectations — update if needed

6. Try triggering the pipeline manually using Run Once to confirm it can execute successfully

Automation cannot run — data mapping template deleted

Cause: The data mapping template associated with the automated file type was deleted. Without a mapping template, the pipeline cannot process files.

Resolution:

1. Navigate to Admin → Pipelines → Automate

2. Locate the affected file type — the Automate icon may still show as blue, but the pipeline will fail on execution

3. Create a new data mapping template for the file type (from the Data Mapping Templates screen, during a manual upload, or via the Automate setup)

4. Update the pipeline to reference the new mapping template

5. Trigger a manual run to verify the pipeline processes files correctly with the new mapping

6. To prevent this in future, note that a data mapping template used by a pipeline (active or inactive) cannot be deleted — the pipeline must be removed first

Cloud storage pipeline not detecting files in the configured folder

Cause: Files are placed outside the configured directory, the folder path has changed, or the file naming convention does not match expectations.

Resolution:

1. Verify the Directory Name configured in the pipeline matches the actual folder path in cloud storage

2. Confirm files are placed inside the configured folder — files stored outside the directory will not be ingested

3. Check the file naming convention — for monthly data, files must end with _YYYYMM format

4. Verify the cloud storage connection has read access to the folder

5. Check if files were already moved to the processed or error folders from a previous run — the pipeline does not reprocess files in these folders

6. If the folder path changed, update the pipeline configuration to point to the new path

Pipeline created via consent URL but data is not appearing

Cause: The consent was granted but the pipeline has not run yet, or the pipeline ran but the cube has not been refreshed.

Resolution:

1. Navigate to Admin → Pipelines → Pipelines and locate the auto-created pipeline

2. Check the pipeline status — it should be active and show a last run time

3. If the pipeline has not run, trigger a manual run using Run Once

4. After the run completes, check View Run History for success or failure

5. If the run succeeded, trigger a cube refresh for the affected cube

6. If the run failed, check the error details — common causes include insufficient permissions in the source system or missing required API permissions

7. For Microsoft Graph API, verify the required API permissions were granted during consent

8. For Google API, verify that Admin SDK API and Enterprise License Manager API are enabled in the tenant

Cannot delete a connection — deletion blocked

Cause: The connection is used by one or more active or inactive pipelines.

Resolution:

1. Navigate to Admin → Pipelines → Pipelines

2. Identify all pipelines that reference the connection

3. Either delete or update those pipelines to use a different connection

4. Once no pipelines reference the connection, return to Connections and delete it

5. Note: Only unused connections can be deleted

Duplicate records appearing after pipeline re-run

Cause: The same file was processed multiple times, or files with overlapping data were placed in the source folder.

Resolution:

1. Check the processed folder — verify the same file was not placed back in the source folder after being processed

2. Review the pipeline run history — look for multiple runs that processed the same file

3. Verify that only new, unique files are placed in the source folder for each pipeline run

4. If duplicates were created, identify the affected records and remove or correct them

5. Follow the best practice of using dedicated folders per file type and consistent naming conventions to prevent overlap

Cannot edit an active pipeline

Cause: The edit capability is disabled for active pipeline connections by design.

Resolution:

1. Navigate to Admin → Pipelines → Pipelines

2. Deactivate the pipeline using the toggle button

3. Click the Edit icon to update the schedule, mapping template, API data, name, or description

4. Click SAVE

5. Reactivate the pipeline

6. Changes take effect from the next scheduled or manual run

Related content

• Connected Data Sources (Integrations)

• Connections

• Pipelines and Data Automation

• Creating Pipelines

• Creating a new Pipeline Connection

• Data Mapping Templates

• Automate

• Understand S3 Folder Structure

• Create AWS Billing Data Export