Pipeline and integration best practices

Use these practices to keep Yarken pipelines and automated ingestion reliable, predictable, and easier to maintain.

Plan your ingestion

Use separate pipelines for different data domains

Use a separate pipeline for each data domain or ingestion purpose, such as spend, budget, Cloud Consumption, tenant licensing, or individual license usage.

This separation helps reduce data overlap and makes each ingestion workflow easier to monitor and troubleshoot.

Reuse supported connections where appropriate

A connection can be reused by multiple supported pipelines. Reuse an existing valid connection instead of creating duplicate connections for the same source when the access requirements are the same.

For connection setup and maintenance, see Connections.

Keep source files predictable

Maintain a consistent file structure

Keep the source schema consistent for recurring file-based ingestion. Changes to column names, required fields, or file structure can affect how the selected Data Mapping Template processes future files.

If the source structure changes:

• Review the Data Mapping Template before the next scheduled run.

• Update the template where editing is supported.

• Use Run Once to validate the change before relying on the next scheduled execution.

For mapping behavior and restrictions, see Data Mapping Templates.

Follow the required file naming convention

Use the naming convention required for the data type you are automating. This is especially important for monthly data where Yarken uses the file name to identify the applicable period.

For the current naming and directory requirements, see Automate.

Keep automated source folders stable

For recurring cloud-storage ingestion, use dedicated and stable source folders. Avoid changing directory paths unless the pipeline configuration is updated to match.

Validate before relying on scheduled runs

Validate the first pipeline run

After you create a pipeline, confirm that the first execution:

• Connects to the expected source.

• Processes the expected files or API data.

• Uses the expected Data Mapping Template for file-based ingestion.

• Loads data into the intended Yarken destination.

Test configuration changes

After you change a connection, source directory, mapping, selected API data, or other supported pipeline setting, use Run Once when you need to validate the configuration immediately.

For pipeline lifecycle actions, see Manage BASIC pipelines.

Monitor pipeline activity regularly

Review pipeline execution regularly so you can identify failed or unexpected runs before they affect downstream reporting.

Use:

• View Run History to review individual pipeline executions.

• Background Processes to check active and completed processing.

• Audit Logs when you need to understand configuration or activation changes.

For monitoring details, see Monitor pipeline runs.

Validate data after ingestion

A successful pipeline run confirms that the ingestion process completed. You should also verify that the expected data is available in the destination Yarken feature.

After an important first run, mapping change, or source change:

1. Review the pipeline execution result.

2. Confirm the expected records were processed.

3. Open the applicable destination in Yarken and verify the data.

4. Refresh the affected cube when you need the latest data immediately in dashboards, reports, or Analytics.

Respond to failures without duplicating work

When a run fails, start with the execution details before recreating connections, mappings, or pipelines.

1. Review View Run History and Background Processes.

2. Identify whether the issue is related to the connection, source file, mapping, schedule, or downstream refresh.

3. Correct the affected configuration only.

4. Use Run Once to verify the fix when immediate validation is required.

For issue-specific resolution steps, see Troubleshoot Pipelines and integrations.

Related content

• Pipelines and integrations

• Connections

• Data Mapping Templates

• Automate

• Pipelines

• Manage BASIC pipelines

• Monitor pipeline runs

• Troubleshoot Pipelines and integrations

• Cube Refresh