Use pipeline monitoring to confirm that data ingestion completed as expected and identify runs that require attention. Yarken provides View Run History, Background Processes, and Audit Logs for execution monitoring and traceability.
This page focuses on monitoring pipeline execution. For BASIC pipeline configuration, activation, Run Once, or deletion, see Manage BASIC pipelines. For Advanced pipeline status, run history, and job details, see Manage Advanced pipelines.
What you can monitor
Use the available monitoring views to:
-
Confirm whether a pipeline run completed successfully.
-
Review the execution date and time.
-
Review processed file or record counts where available.
-
Review error details for failed runs.
-
Track pipelines that are currently running.
-
Confirm when long-running background activity completes.
-
Review pipeline-related configuration changes.
-
Confirm how Yarken handled files after a cloud storage run.
Choose the monitoring view
|
View |
Use it to |
Best for |
|---|---|---|
|
View Run History |
Review individual pipeline executions and available run details. |
Confirming whether a specific pipeline run succeeded or failed. |
|
Background Processes |
Review running and completed background execution activity. |
Tracking a pipeline that is still processing or taking longer than expected. |
|
Audit Logs |
Review pipeline-related system actions and configuration changes. |
Identifying who changed a pipeline and when the change occurred. |
View Run History and Background Processes focus on execution. Audit Logs focus on configuration activity and traceability.
Review View Run History
Each pipeline execution is recorded in View Run History.
The run history can include:
-
Execution date and time.
-
Run result, including whether the execution completed successfully or failed.
-
The number of processed files or records, where available.
-
The source integration or interface.
-
Error details when a run fails.
Step 1: Open the pipeline
-
Go to Admin > Pipelines > Pipelines.
-
Select the applicable BASIC or ADVANCED tab.
-
Find the pipeline you want to review.
Step 2: Open the run history
Open the pipeline execution history using the available run-history action.
Step 3: Review the execution details
Review the latest execution and expand the row when additional details are available.
Confirm:
-
The pipeline ran at the expected time.
-
The execution completed successfully.
-
The expected files or records were processed.
-
No error details require action.
A successful execution confirms that the pipeline run completed. If expected data is still missing from Reports or Analytics, verify the destination data and refresh the affected cube before treating the issue as an ingestion failure.
Monitor Background Processes
Background Processes shows real-time and completed background activity for pipeline executions.
Use it to:
-
Monitor a pipeline that is currently running.
-
Confirm when a background execution completes.
-
Review individual pipeline execution activity.
-
Track a long-running pipeline before investigating it as a failure.
See Background Processes for the complete feature guidance.
Review Audit Logs
Audit Logs record pipeline-related system actions and configuration changes.
Use Audit Logs to:
-
Track when pipelines are created or updated.
-
Track when pipelines are activated or deactivated.
-
Review the user associated with a configuration change.
-
Filter relevant events by time, user, or available execution information.
Open Audit Logs for the complete feature guidance.
Use Audit Logs when you need to understand what changed, who changed it, and when. Use Run History or Background Processes when you need to understand what happened during execution.
Monitor cloud storage file processing
When a cloud storage pipeline processes files, Yarken automatically organizes the files in the configured storage location.
Yarken creates two folders:
|
Folder |
Contains |
|---|---|
|
Processed |
Files that Yarken ingested successfully. |
|
Error |
Files Yarken could not process because of validation, format, or mapping issues. |
After each run, Yarken moves processed files to the applicable folder. This prevents the same source file from being processed repeatedly and provides file-level traceability.
Do not upload, modify, or reuse files in the Processed or Error folders.
If a file moves to Error
A file in the Error folder requires attention before you reprocess it.
-
Review the corresponding pipeline run.
-
Review the available error details.
-
Correct the source file, mapping, or other confirmed issue.
-
Reprocess a corrected file through the supported ingestion workflow.
The Processed and Error folders provide a file-level audit trail. Use them together with Run History when investigating a file-based pipeline execution.
Validate the result after a successful run
After a successful pipeline run:
-
Confirm that Yarken processed the expected file or records.
-
Open the Yarken feature associated with the destination data type.
-
Select the applicable entity, year, and month.
-
Verify that the expected records are available.
-
Refresh the affected cube when you need immediate reporting results.
-
Review the data in the relevant report, dashboard, or Analytics data area.
For current refresh guidance, see Cube refresh and best practices for Pipelines.
When a run needs attention
Use monitoring to identify the symptom before you change the pipeline configuration.
|
What you observe |
Next check |
|---|---|
|
No run appears at the expected time |
Confirm the pipeline is active and review its configured schedule. |
|
The pipeline is still running |
Review Background Processes before treating the run as failed. |
|
The run status is Failed |
Review the available run error details before changing the connection, mapping, or source file. |
|
A file appears in Error |
Review the run details, source file format, required fields, and mapping. |
|
The run succeeded, but reporting data is not current |
Verify the destination data and refresh the affected cube. |
|
The configuration changed unexpectedly |
Review Audit Logs to identify the change and the user associated with it. |
Troubleshoot Pipelines and integrations
Use this guide for detailed diagnosis and recovery steps for connection, mapping, automation, and BASIC pipeline issues.
Related content