Azure Data Factory (ADF)

Version Supported: Version 2

Tool Permissions Prerequisites

Warning: Missing permissions could end up in broken lineages.

• A dedicated application registered with the 'Reader' role assigned to the relevant Data Factories.
• Valid 'Client Secret' for authentication credentials.

How to set up the permissions

 

Step 1:

Application Setup:  Quickstart: Register an app in the Microsoft identity platform

Guidelines for application setup:

1. select ‘Accounts in this organizational directory only ’When creating a new application (registration), under ‘Who can use this application or access this API?’.

2. On the same page, leave 'Redirect URI empty'.

3. Credentials - On the Application page, under Manage > Certificates & Secrets, use the Client Secret option for Credentials. Copy it immediately, as it won't be fully visible afterward.

Step 2:

Assign the dedicated application a 'Reader' role to the relevant Factory/ies by following the below steps:

1 - Under your DataFactory, go to the 'Access control (IAM)' tab and click on 'ADD > Add role assignment'.

 
2 - Look for the 'Reader' role and click it.

 
3 - Under the 'Members' tab, choose 'User, group, or service principal' and click on '+ Select members', then search for your application.

 
4 - The last step will be to review your configuration and assign the role by clicking on 'Review + assign'.

 
5 - After completing the previous steps, go back to your DataFactory's 'Access control (IAM)' tab > 'Role assignments'. Your application should be there.

 

Setting up ADF Metadata Source

Metadata Sources are set on the Octopai Client (OC) 

Legend:

1. Connection Name - Give a meaningful name, as it will be displayed to the Octopai platform users.
2. Subscription ID - Found in the 'Subscriptions' section of the Azure portal.
3. Tenant ID - Available in the 'App registrations' section under the application you created.
4. Application (Client) ID - Available in the 'App registrations' section under the application you created.
5. Client Secret - Generated in 'App registrations > Certificates & secrets'.
6. Resource Group - Found in the 'Resource groups' section where you created or assigned resources for your Data Factory.
7. Factory Name - Listed in the 'Data Factory' section under your specific factory instance.
8. API Version - Usually specified in the Azure documentation or the REST API version section related to Data Factory.

After completing all the mandatory fields, click on 'Next' > 'Finish' > and 'Run' to extract the metadata from your source.

Video - Extraction Guide

Watch the following video to learn about the Azure Data Factory Extraction process