Cloud storage Advanced Integrations

Use Cloud storage Advanced Integrations to automate file ingestion from supported cloud storage providers into Yarken.

Yarken deploys and maintains the integration workflow for your environment. Your organization provides the required cloud credentials, authorizes the Advanced Connection, and operates the exposed Advanced pipeline controls.

Supported cloud storage providers

Cloud storage Advanced Integrations can be enabled for the following supported providers:

Provider	Connection	Use it to
Amazon S3	Amazon S3 Advanced Connection	Retrieve supported files from the configured S3 bucket and source path.
Azure Blob Storage	Azure Blob Storage Advanced Connection	Retrieve supported files from the configured Azure Storage account and source location.
Google Cloud Storage	Google Cloud Storage Advanced Connection	Retrieve supported files from the configured Google Cloud Storage bucket.

The integrations available to your organization depend on what Yarken has enabled for your environment.

How cloud storage Advanced Integrations work

The exact source configuration depends on the provider, but the ingestion flow follows this pattern:

Cloud storage source — Your organization places supported files in the source location configured for the integration.
Advanced Connection — Yarken uses the authorized provider connection to access the configured storage location.
Yarken-managed integration workflow — The deployed integration retrieves and processes matching files.
Data Mapping Template, where required — Yarken automatically applies the mapping configured for the Advanced Integration.
Advanced pipeline — The pipeline runs the integration and records the execution result.
Yarken destination — Yarken loads the processed data into the destination configured for the integration.

You do not select or edit the Data Mapping Template for an Advanced Integration. Yarken applies and maintains the required mapping as part of the deployed integration workflow.

Responsibilities

Activity	Responsibility
Enable and deploy the integration workflow	Yarken
Configure and maintain the integration logic and mapping	Yarken
Prepare the cloud storage account, bucket or container, and provider access	Your cloud administrator
Authorize the Advanced Connection	Your Yarken administrator
Run or activate the Advanced pipeline	Admin or Cost Model Manager
Validate the imported data	Your organization

Before you begin

Before using a cloud storage Advanced Integration:

Confirm that Yarken has enabled the required integration for your environment.
Prepare the cloud credentials required for the deployed connection.
Confirm the source bucket, container, or storage location agreed for the integration.
Follow the required source-file structure and naming conventions for the data being ingested.

Provider-side setup requirements can vary. The sections below document only the connection fields currently exposed in Yarken.

Configure your cloud storage provider

Required fields are marked with an asterisk (*) in Yarken. Fields without an asterisk are optional or conditional based on the selected connection configuration.

Amazon S3

Click to view Advanced connection fields

Open the Amazon S3 Advanced Connection and complete the fields shown for your environment.

Field	Requirement	Description
Connection type	Required	Set to Cloud for the cloud-hosted connection.
Authorization type	Required	Select the authorization type configured for the deployed connection. Access key (Deprecated) can appear for existing configurations.
Access key ID	Required	Enter the access key ID for the AWS identity used by the connection.
Secret access key	Required	Enter the corresponding secret access key.
Restrict to bucket	Optional	Restricts the connection to the specified S3 bucket. Use it when the AWS identity has limited `s3:ListBucket` access.
Restrict to path	Optional	Restricts the connection to a specified bucket path or object path.
Region	Required	Enter the AWS Region for the S3 source.
Download threads	Optional	Controls concurrent downloads. The default is 1, and the maximum is 20.

The Access key authorization option is deprecated. Do not change the configured authorization approach unless Yarken provides an approved replacement for your integration.

Azure Blob Storage

Click to view Advanced connection fields

Open the Azure Blob Storage Advanced Connection and complete the fields shown for your environment.

Field	Requirement	Description
Connection type	Required	Set to Cloud for the cloud-hosted connection.
Storage account	Required	Enter the Azure Storage account name.
Connection account type	Optional	Select the applicable account type: Common, Organization, or Tenant-specific. Common is the default.
OAuth 2.0 authorization code scopes	Optional	Configure additional scopes when required. If you do not select scopes, the connection uses Storage, Offline_access, and Management by default.
Client ID	Conditional	Required when the authentication type is set to Client credentials.
Client secret	Conditional	Required when the authentication type is set to Client credentials.
Access key	Optional	Can be provided for actions that use the storage account access key.

The Azure connection screen includes additional settings that depend on the selected account and authentication configuration. Keep the deployed values unchanged unless Yarken instructs you to update them.

Google Cloud Storage

Click to view Advanced connection fields

Open the Google Cloud Storage Advanced Connection and complete the fields shown for your environment.

Field	Requirement	Description
Connection type	Required	Set to Cloud for the cloud-hosted connection.
Project identifier	Required	Enter the Google Cloud project identifier.
GCS Project service account email	Required	Enter the email address of the Google Cloud service account used by the connection.
Private key	Required	Paste the private key from the downloaded service-account JSON file.
Restrict to bucket	Optional	Restricts the connection to one or more specified buckets. The UI supports a comma-separated list.
Requested permissions (OAuth scopes)	Optional	Controls the requested Google Cloud Storage permissions. `devstorage.read_only` is the minimum permission and is always requested in addition to any selected permissions.

Treat the service-account private key as a secret. Do not expose it in screenshots, documentation, email, chat, or support tickets.

Authorize the cloud storage connection

Go to Admin > Pipelines > Connections.
Select the ADVANCED tab.
Open the cloud storage connection that Yarken deployed for the integration.
Enter the provider-specific credentials and required connection details.
Select Connect.
Confirm that the connection status changes to Connected.

For the general authorization workflow, see Authorize Advanced Connections.

Run the Advanced pipeline

After the connection shows Connected:

Go to Admin > Pipelines > Pipelines.
Select the ADVANCED tab.
Locate the cloud storage pipeline deployed for your integration.
Select Run to start an immediate ingestion, or activate recurring execution when required.
Open the pipeline to review the run result.

See Manage Advanced pipelines for manual runs, activation, and run-history guidance.

Validate the ingestion

After a successful run, confirm that:

The Advanced pipeline shows a successful execution.
The expected source file was available in the configured cloud storage location.
The expected data is available in the destination configured for the integration.
The imported period, records, and key values match the source data.

If the latest data is required immediately in reporting, refresh the affected cube when applicable.

File and source-location best practices

Keep the configured source location dedicated to the files intended for the integration.
Use a consistent file structure and naming convention.
Do not change the configured bucket, container, or source path without coordinating the integration change.
Validate a representative file after the integration is first enabled or after a source-format change.
Monitor failed runs before adding additional files to a source location with an unresolved issue.

Troubleshoot cloud storage ingestion

Issue	What to check
The connection remains Access requested	Verify the provider credentials and required connection fields, then select Connect.
The pipeline runs but does not ingest the expected file	Confirm the file is in the configured bucket, container, and source location and follows the expected file requirements.
The pipeline fails	Review the run error and job details. Confirm the Advanced Connection still shows Connected.
Data is loaded incorrectly	Do not edit a Data Mapping Template. Confirm the source file matches the agreed integration format and contact Yarken if the deployed mapping requires an update.
The pipeline succeeds but data is not visible in reporting	Validate the destination data first, then refresh the affected cube if an immediate reporting refresh is required.

For general Advanced Integration issues, see Troubleshoot Advanced Integrations.

Related content