Google BigQuery

Version Supported: up to 2023

Tool Permissions Prerequisites

Warning: Missing permissions could end up in broken lineages.

Google Service Account (SA) integrated with Octopai.

How to establish a Google Cloud Service Account

Creating a Google Cloud Service Account (SA)

1. Open the Google Cloud Console and navigate to your desired project.
2. Access the IAM & Admin menu and select 'Service Accounts'.
3. Click on 'Create Service Account' to initiate the creation process.
   
4. Assign a unique, identifiable name to your Service Account. Upon clicking 'Create and Continue', an Identity and Access Management (IAM) principal will be instantiated automatically.
   

   Important: Do not click 'Done' at this point.

   

5. From the 'roles' dropdown menu, select both 'BigQuery Data Viewer' and 'BigQuery Job User' to assign necessary permissions to your Service Account.
   
   
6. Complete the Service Account creation process by clicking 'Done'.
7. Access the newly created Service Account and navigate to the 'KEYS' tab.
   
   
   

Generating a Service Account Key

• Click on 'ADD KEY', then 'Create new key'. Make sure to select the JSON format. The key will automatically be downloaded to your local system.
  

Configuring Octopai with BigQuery

1. Open Octopai and start the creation of a new metadata source, choosing 'BigQuery' as your source.
2. Assign a descriptive name to this metadata source for easy reference in the future.
3. Input the Project ID associated with your Google Cloud project. This can be found within the downloaded JSON file (under the 'project_id' field) or in the project selector on the Google Cloud Console.
4. Specify the file path where your downloaded JSON key file is stored in the 'Key Path' field.
5. Save your settings and initiate the connection by clicking 'Save and Run'.

Setting up Google BigQuery Metadata Source

Metadata Sources are set on the Octopai Client (OC) 

How to verify the extracted Metadata File

Access the Octopai Target Folder (TGT)

1. Go to the TGT Folder located on the Server where the Octopai Client is installed.
   By default: C:\Program Files (x86)\Octopai\Service\TGT
2. Open the zip file having your Connector Name
   Example:
3. Verify its content
   Quantity & Quality of inner files

Troubleshoot

Error during the extraction:

• Check the permissions
• Send the log with the connector number and name to Octopai Support - C:\Program Files (x86)\Octopai\Service\log