What is a Pipeline Connection?
A Connection is a saved set of credentials and configuration that allows a pipeline to connect to an external data source, for example, AWS S3, Azure Storage, Google Cloud Storage, Power BI API, or Microsoft Graph API. Once created and saved, pipelines use the connection to run scheduled or instant ingestion without re-entering credentials. The connections are reused by many pipelines and are managed centrally in the Pipelines > Connections page.
Connection vs Pipeline
-
Connection: how to connect to the data source (endpoints + credentials)
-
Pipeline: Ingest data from cloud storage or APIs (select connection, data mapping, and schedule)
Supported Connection types
Typical connection types you can create:
-
Cloud storage: AWS S3, Azure Blob, and Google Cloud Storage
-
API: Microsoft Graph API, Power BI API, Google API, and Google Reseller API
Pipeline Connection creation methods
The API connections can be configured manually or provisioned automatically via consent flow, while the cloud storage connection requires explicit manual configuration.
API - manual or automatic
API connections can be created manually or automatically using a consent URL.
-
In manual setup, the user adds the credentials (Tenant ID, Client ID, Secret, etc.), validates, and saves.
-
The automatic connection is initiated through the consent URL. You can either authenticate within the app or opt to copy the consent URL and share the URL to authorize via email or another device.
-
After accepting required permissions, the application automatically creates connections. The consent URL is unique to Microsoft Graph, Power BI, and Google APIs. Clicking an individual URL will create the corresponding API connections (if required permissions are granted by the tenant admin). It also auto-creates the pipelines linked to those connections.
Note:
-
Automatic API setup usually requires global tenant admin consent in the identity provider and still needs verification (app registrations, service principal).
-
Power BI requires an explicit service principal model for app-only access and tenant admin settings to allow service principals.
Cloud storage - manual Only
The cloud storage connections (Azure Blob, AWS S3, and Google Cloud Storage) must be manually configured. Storage connections require provider-specific details (access key, secret key, container/bucket name).
Note: Unlike API connections, the application does not create pipelines automatically for cloud storage; users must set up these pipelines manually.
Fields required to configure a connection manually
When creating a connection manually, the exact fields you must provide depend on the connection type.
-
API connections: Tenant Id, Client Id, and Client Secret
-
Cloud storage: Account Name, Access Key, Secret Key, Container Name, and Bucket Name
The API connections fields are auto-captured when configured through the consent URL.
Authentication types (available only for Microsoft Graph and Power BI API)
The authentication type allows you to choose how the API connections will authenticate to the external system.
-
Custom Credentials: The auth details, such as Tenant ID, Client ID, or Client Secret, are required to be added manually.
-
Platform Credentials: Use this when you want to auto-create the connections using a consent URL. The credentials are auto-captured by the application.
Note: The Authentication Type is not available for the cloud storage and Google API connections.
Next steps
Related content