Use this article to understand how Yarken automates recurring data ingestion through pipelines, schedules, and reusable mapping templates.
Automate recurring data ingestion
Pipelines and Data Automation help teams bring data into Yarken on a recurring schedule.
Instead of downloading files from source systems and uploading them manually each time, teams can use pipelines to retrieve data from cloud storage or APIs, apply a data mapping template, and make the data available for downstream reporting and analysis.
This keeps data ingestion more consistent and reduces manual effort across recurring processes such as cloud usage, license usage, spend, budget, and other scheduled data loads.
What you can do
-
Automate recurring file ingestion from cloud storage.
-
Create pipelines using data mapping templates.
-
Map source fields to Yarken fields consistently each time the pipeline runs.
-
Schedule recurring uploads for supported data domains.
-
Monitor pipeline execution and running history.
-
Update pipeline schedules, mappings, names, and descriptions as requirements change.
-
Use separate pipelines for different data domains such as spend, budget, cloud consumption, tenant licensing, and individual license usage.
-
Reduce the need for repeated manual uploads.
Data mapping templates
Data mapping templates define how source file fields map to Yarken fields.
They help standardize the upload process and avoid manual field mapping each time data is loaded.
All automated pipelines require a data mapping template, so the system can apply the same mapping logic whenever the pipeline runs.
Pipelines
Pipelines automate data retrieval and upload.
A cloud storage pipeline is created by selecting the storage provider and assigning a data mapping template. Once configured, the pipeline can retrieve files and apply the mapping needed to load data into Yarken.
Pipelines are useful when the same type of data needs to be loaded repeatedly, especially from structured source locations.
Automate
Automate focuses on scheduled and recurring file ingestion.
It is designed for cases where files are added to cloud storage and need to be picked up by Yarken without a user manually uploading them each time.
Automate focuses on file ingestion only. The quality of the output still depends on the source file, mapping template, and downstream processing.
How teams use it
Teams use Pipelines and Automate to make recurring data loads more reliable.
Cloud usage files, license usage data, spend files, budget files, and other recurring datasets can be handled through separate pipelines. This reduces overlap, keeps data domains cleaner, and makes monitoring easier.
When a requirement changes, teams can update the pipeline schedule, mapping template, selected API data where supported, name, or description.
What makes it different
Pipelines and Data Automation help Yarken operate as a repeatable data system, not a manual upload workspace.
The same mapping logic can be reused, files can be ingested on schedule, and teams can monitor execution to identify issues early.
This gives Finance, IT, FinOps, and administrators more confidence that reporting and analysis are based on a consistent data intake process.
When to use Pipelines and Data Automation
Use Pipelines when you need recurring data ingestion from cloud storage or supported APIs.
Use data mapping templates when a source file needs consistent field mapping into Yarken.
Use Automate when files are placed in cloud storage and should be picked up on a recurring schedule.
For detailed step by step instructions on creating data mapping templates, creating pipelines, managing pipelines, monitoring execution, troubleshooting failures, and refreshing cubes, refer to the relevant user guides.
Next step
Connected Data Sources (Integrations)
Related articles
Connected Data Sources (Integrations)