Octopai Extraction APIs

Overview

The following API endpoints are designed to facilitate the extraction of lineage data from Octopai's system.

For optimal results, we suggest initiating the process with a call to /assets/query in order to obtain a comprehensive list of assets. Subsequently, you should make a secondary call to /lineage. This will allow you to retrieve detailed lineage information for each individual asset identified in the initial query. This two-step approach ensures a more structured and efficient retrieval of data, providing you with a clearer understanding of the asset lineage within Octopai's ecosystem.

Assets API

Description

This API focuses on retrieving data about various assets like columns, tables, etc. It's primarily used to fetch assets and their properties.

This API required when searching for asset lineage by using the asset _key in order to call /lineage.

Key Function

Search Assets

Endpoint

GET /api/v2.0/assets/query

Request Body

{
"ConnectionIds": {list},
"AssetNames": {list},
"ToolNames": {list},
"ToolTypes": {list},
"DatabaseName": {string},
"SchemaName": {string},
"LayerName": {string},
"TableName": {string},
"assetType": {number},
"batchSize": {number},
"nextID": {string}
}

Request Body Parameters

Name	Required	Type	Description
ConnectionIds	No	list	Search by ConnectionIds - Get all assets for this connection IDs - Example: ["101","102","103"]
AssetNames	No	list	Search by AssetNames - Get assets that contains one of the AssetNames values in assetName
ToolNames	No	list	Search by ToolNames
ToolTypes	No	list	Search by ToolTypes - For Cross System Lineage (assetType=1) [DATABASE / ETL / REPORT] - For Column Level Lineage (assetType=2) [DB / ETL / REPORT]
DatabaseName	No	string	Search by DatabaseName
SchemaName	No	string	Search by DatabaseName
LayerName	No	string	Search by LayerName
TableName	No	string	Search by TableName
assetType	Yes	number	1 - For Cross System Lineage 2 - For Column Level Lineage
batchSize	Yes	number	Number of returned results per page (max:10,000)
nextID	Yes (From the second call)	string	Get the next result batch

Response Body

[
"hasMore": {bool},
"id": {string},
"columns":[
{
"_key": {string},
"connectionId": {string},
"connLogicName": {string},
"toolName": {string},
"toolType": {string},
"displayConnectionId": {string},
"containerObjectName": {string},
"containerObjectPath": {string},
"containerObjectType": {string},
"controlFlowPath": {string},
"controlFlowName": {string},
"objectID": {string},
"objectGUID": {string},
"objectType": {string},
"assetName": {string},
"dataType": {string},
"precision": {string},
"scale": {string},
"layerName": {string},
"schemaName": {string},
"databaseName": {string},
"tableName": {string},
"isObjectData": {bool},
"isMap": {bool},
"isVisible": {bool},
"lvl5": {string},
"lvl4": {string},
"lvl3": {string},
"lvl2": {string},
"lvl1": {string},
"updatedDate": {string},
"createDate": {string},
"serverName": {string},
"isSrcColumnOrphan": {bool}
}
]
]

Pagination

Use GET /api/v2.0/assets/query/scroll/{nextID} with the nextID from the first call to get the next results chunk.

Request Sample

{
"assetNames": ["storeid"],
"schemaName": "SALES",
"tableName": "Customer", 
"databaseName": "AdventureWorks2014",
"toolTypes": ["DB"],
"assetType": 2,
"limit": 1000
}

 

Response Sample

{
"items": [
{
"_key": "8D1A993B6739DAB65392B54D45F7052E",
"connectionId": "251",
"connLogicName": "sqlserverdbwa03testETLTRUE",
"toolName": "SQLS",
"toolType": "DB",
"displayConnectionId": "251",
"containerObjectName": "sp_built_MrrPerson_MrrCustomer",
"containerObjectPath": "AdventureWorks2014.dbo",
"containerObjectType": "SQL_STORED_PROCEDURE",
"controlFlowPath": "sp_built_MrrPerson_MrrCustomer",
"controlFlowName": "sp_built_MrrPerson_MrrCustomer",
"objectID": "ADVENTUREWORKS2014SALESCUSTOMERTABLESTOREID",
"objectGUID": "8D1A993B6739DAB65392B54D45F7052E",
"objectType": "TABLE",
"assetName": "StoreID",
"dataType": "",
"precision": "",
"scale": "",
"layerName": "Customer",
"schemaName": "SALES",
"databaseName": "AdventureWorks2014",
"tableName": "Customer",
"isObjectData": true,
"isMap": false,
"isVisible": true,
"lvl5": "",
"lvl4": "",
"lvl3": "",
"lvl2": "",
"lvl1": "",
"updatedDate": "2024-05-26T10:21:51.224Z",
"createDate": "0001-01-01T00:00:00Z",
"serverName": "",
"isSrcColumnOrphan": null
},
{
"_key": "1D8EF9FCE4205A4EA88894BEB5E47956",
"connectionId": "251",
"connLogicName": "sqlserverdbwa03testETLTRUE",
"toolName": "SQLS",
"toolType": "DB",
"displayConnectionId": "251",
"containerObjectName": "sp_built_MrrPerson_MrrCustomer",
"containerObjectPath": "AdventureWorks2014.dbo",
"containerObjectType": "SQL_STORED_PROCEDURE",
"controlFlowPath": "sp_built_MrrPerson_MrrCustomer",
"controlFlowName": "sp_built_MrrPerson_MrrCustomer",
"objectID": "SP_BUILT_MRRPERSON_MRRCUSTOMERADVENTUREWORKS2014.DBOSP_BUILT_MRRPERSON_MRRCUSTOMERCUSTOMERTABLESTOREID",
"objectGUID": "1D8EF9FCE4205A4EA88894BEB5E47956",
"objectType": "Table",
"assetName": "StoreID",
"dataType": "",
"precision": "",
"scale": "",
"layerName": "Customer",
"schemaName": "SALES",
"databaseName": "AdventureWorks2014",
"tableName": "Customer",
"isObjectData": true,
"isMap": true,
"isVisible": true,
"lvl5": "",
"lvl4": "",
"lvl3": "",
"lvl2": "",
"lvl1": "",
"updatedDate": "2024-05-26T10:21:51.224Z",
"createDate": "2024-05-26T10:21:51.224Z",
"serverName": "",
"isSrcColumnOrphan": null
},
{
"_key": "0E0766C0D1D88D9572502C46F389D6E0",
"connectionId": "251",
"connLogicName": "sqlserverdbwa03testETLTRUE",
"toolName": "SQLS",
"toolType": "DB",
"displayConnectionId": "251",
"containerObjectName": "ufnGetContactInformation",
"containerObjectPath": "AdventureWorks2014.dbo",
"containerObjectType": "SQL_TABLE_VALUED_FUNCTION",
"controlFlowPath": "ufnGetContactInformation",
"controlFlowName": "ufnGetContactInformation",
"objectID": "UFNGETCONTACTINFORMATIONADVENTUREWORKS2014.DBOUFNGETCONTACTINFORMATIONCUSTOMERTABLESTOREID",
"objectGUID": "0E0766C0D1D88D9572502C46F389D6E0",
"objectType": "Table",
"assetName": "StoreID",
"dataType": "",
"precision": "",
"scale": "",
"layerName": "Customer",
"schemaName": "SALES",
"databaseName": "AdventureWorks2014",
"tableName": "Customer",
"isObjectData": false,
"isMap": true,
"isVisible": true,
"lvl5": "",
"lvl4": "",
"lvl3": "",
"lvl2": "",
"lvl1": "",
"updatedDate": "2024-05-26T10:21:51.224Z",
"createDate": "2024-05-26T10:21:51.224Z",
"serverName": "",
"isSrcColumnOrphan": null
}
],
"hasMore": false,
"cursorId": null
}

Lineage API

Description

This API is designed to pull lineage data, which is crucial for understanding the relationships and dependencies between different assets.

Key Function

Get Lineage Data for Asset

Endpoint

GET /api/v2.0/lineage

 

Request Body

{
"assetKey": {string},
"depth": {number},
"direction": {number},
"limit": {number},
"assetType": {number}
}

 

Request Body Parameters

Name	Required	Type	Description
assetKey	Yes	string	Asset key to start from.
depth	Yes	number	Number of hops for lineage from the start asset.
direction	No	number	Default: 3. 1 - Get inputs relations only for this asset. 2 - Get outputs relations only for this asset. 3 - Any (Inputs and Outputs).
limit	Yes	number	Limit number of total assets in result.
assetType	Yes	number	1 - For Cross System Lineage. 2 - For Column Level Lineage.

Response Body

[
{
"nodes": [<list-of-assets>],
"edges": [<list-of-links>],
"startNode" {<start-node>}
}
]

 

Request Sample

{
"assetKey": "8D1A993B6739DAB65392B54D45F7052E",
"depth": "6",
"limit": "1000",
"assetType": 2,
"direction": null
}

Response Sample

{
"nodes": [
{
"_key": "8D1A993B6739DAB65392B54D45F7052E",
"connectionId": "251",
"connLogicName": "sqlserverdbwa03testETLTRUE",
"toolName": "SQLS",
"toolType": "DB",
"displayConnectionId": "251",
"containerObjectName": "sp_built_MrrPerson_MrrCustomer",
"containerObjectPath": "AdventureWorks2014.dbo",
"containerObjectType": "SQL_STORED_PROCEDURE",
"controlFlowPath": "sp_built_MrrPerson_MrrCustomer",
"controlFlowName": "sp_built_MrrPerson_MrrCustomer",
"objectID": "ADVENTUREWORKS2014SALESCUSTOMERTABLESTOREID",
"objectGUID": "8D1A993B6739DAB65392B54D45F7052E",
"objectType": "TABLE",
"assetName": "StoreID",
"dataType": "",
"precision": "",
"scale": "",
"layerName": "Customer",
"schemaName": "SALES",
"databaseName": "AdventureWorks2014",
"tableName": "Customer",
"isObjectData": true,
"isMap": false,
"isVisible": true,
"lvl5": "",
"lvl4": "",
"lvl3": "",
"lvl2": "",
"lvl1": "",
"updatedDate": "2024-05-26T10:21:51.224Z",
"createDate": "0001-01-01T00:00:00Z",
"serverName": "",
"isSrcColumnOrphan": null
},
{
"_key": "0E0766C0D1D88D9572502C46F389D6E0",
"connectionId": "251",
"connLogicName": "sqlserverdbwa03testETLTRUE",
"toolName": "SQLS",
"toolType": "DB",
"displayConnectionId": "251",
"containerObjectName": "ufnGetContactInformation",
"containerObjectPath": "AdventureWorks2014.dbo",
"containerObjectType": "SQL_TABLE_VALUED_FUNCTION",
"controlFlowPath": "ufnGetContactInformation",
"controlFlowName": "ufnGetContactInformation",
"objectID": "UFNGETCONTACTINFORMATIONADVENTUREWORKS2014.DBOUFNGETCONTACTINFORMATIONCUSTOMERTABLESTOREID",
"objectGUID": "0E0766C0D1D88D9572502C46F389D6E0",
"objectType": "Table",
"assetName": "StoreID",
"dataType": "",
"precision": "",
"scale": "",
"layerName": "Customer",
"schemaName": "SALES",
"databaseName": "AdventureWorks2014",
"tableName": "Customer",
"isObjectData": false,
"isMap": true,
"isVisible": true,
"lvl5": "",
"lvl4": "",
"lvl3": "",
"lvl2": "",
"lvl1": "",
"updatedDate": "2024-05-26T10:21:51.224Z",
"createDate": "2024-05-26T10:21:51.224Z",
"serverName": "",
"isSrcColumnOrphan": null
},
{
"_key": "6C3F321F06624C447A6CBA9A1AE4DC2E",
"connectionId": "251",
"connLogicName": "sqlserverdbwa03testETLTRUE",
"toolName": "SQLS",
"toolType": "DB",
"displayConnectionId": "251",
"containerObjectName": "ufnGetContactInformation",
"containerObjectPath": "AdventureWorks2014.dbo",
"containerObjectType": "SQL_TABLE_VALUED_FUNCTION",
"controlFlowPath": "ufnGetContactInformation",
"controlFlowName": "ufnGetContactInformation",
"objectID": "UFNGETCONTACTINFORMATIONADVENTUREWORKS2014.DBOUFNGETCONTACTINFORMATIONUFNGETCONTACTINFORMATIONTABLESTOREID",
"objectGUID": "6C3F321F06624C447A6CBA9A1AE4DC2E",
"objectType": "Table",
"assetName": "STOREID",
"dataType": "",
"precision": "",
"scale": "",
"layerName": "ufnGetContactInformation",
"schemaName": "DBO",
"databaseName": "AdventureWorks2014",
"tableName": "ufnGetContactInformation",
"isObjectData": false,
"isMap": true,
"isVisible": true,
"lvl5": "",
"lvl4": "",
"lvl3": "",
"lvl2": "",
"lvl1": "",
"updatedDate": "2024-05-26T10:21:51.224Z",
"createDate": "2024-05-26T10:21:51.224Z",
"serverName": "",
"isSrcColumnOrphan": null
},
{
"_key": "A3853ADFE6CBBA5F6E72A83FBAA8FD53",
"connectionId": "251",
"connLogicName": "sqlserverdbwa03testETLTRUE",
"toolName": "SQLS",
"toolType": "DB",
"displayConnectionId": "251",
"containerObjectName": "ufnGetContactInformation",
"containerObjectPath": "AdventureWorks2014.dbo",
"containerObjectType": "SQL_TABLE_VALUED_FUNCTION",
"controlFlowPath": "ufnGetContactInformation",
"controlFlowName": "ufnGetContactInformation",
"objectID": "ADVENTUREWORKS2014DBOUFNGETCONTACTINFORMATIONSQL_TABLE_VALUED_FUNCTIONSTOREID",
"objectGUID": "A3853ADFE6CBBA5F6E72A83FBAA8FD53",
"objectType": "SQL_TABLE_VALUED_FUNCTION",
"assetName": "STOREID",
"dataType": "",
"precision": "",
"scale": "",
"layerName": "ufnGetContactInformation",
"schemaName": "DBO",
"databaseName": "AdventureWorks2014",
"tableName": "ufnGetContactInformation",
"isObjectData": true,
"isMap": false,
"isVisible": true,
"lvl5": "",
"lvl4": "",
"lvl3": "",
"lvl2": "",
"lvl1": "",
"updatedDate": "2024-05-26T10:21:51.224Z",
"createDate": "2024-05-26T10:21:51.224Z",
"serverName": "",
"isSrcColumnOrphan": null
},
{
"_key": "07B1F48DA85A593EE4C298B5189277B9",
"connectionId": "251",
"connLogicName": "sqlserverdbwa03testETLTRUE",
"toolName": "SQLS",
"toolType": "DB",
"displayConnectionId": "251",
"containerObjectName": "ufnGetContactInformation",
"containerObjectPath": "AdventureWorks2014.dbo",
"containerObjectType": "SQL_TABLE_VALUED_FUNCTION",
"controlFlowPath": "ufnGetContactInformation",
"controlFlowName": "ufnGetContactInformation",
"objectID": "UFNGETCONTACTINFORMATIONADVENTUREWORKS2014.DBOUFNGETCONTACTINFORMATIONUFNGETCONTACTINFORMATIONTABLE*",
"objectGUID": "07B1F48DA85A593EE4C298B5189277B9",
"objectType": "Table",
"assetName": "*",
"dataType": "",
"precision": "",
"scale": "",
"layerName": "ufnGetContactInformation",
"schemaName": "DBO",
"databaseName": "AdventureWorks2014",
"tableName": "ufnGetContactInformation",
"isObjectData": false,
"isMap": true,
"isVisible": true,
"lvl5": "",
"lvl4": "",
"lvl3": "",
"lvl2": "",
"lvl1": "",
"updatedDate": "2024-05-26T10:21:51.224Z",
"createDate": "2024-05-26T10:21:51.224Z",
"serverName": "",
"isSrcColumnOrphan": null
}
...
],
"edges": [
{
"_from": "05E69AF36C41C5FA59208CD75C937AA7",
"_to": "F6DC4A4D0BA1C88011F843A31BB6BFD4",
"sourceObjectGUID": "05E69AF36C41C5FA59208CD75C937AA7",
"sourceIsVisible": true,
"targetObjectGUID": "F6DC4A4D0BA1C88011F843A31BB6BFD4",
"targetIsVisible": true,
"isCompressed": true
},
{
"_from": "07B1F48DA85A593EE4C298B5189277B9",
"_to": "56DEB310C9AE639E2103AC0D47825172",
"sourceObjectGUID": "07B1F48DA85A593EE4C298B5189277B9",
"sourceIsVisible": true,
"targetObjectGUID": "56DEB310C9AE639E2103AC0D47825172",
"targetIsVisible": true,
"isCompressed": true
},
{
"_from": "0E0766C0D1D88D9572502C46F389D6E0",
"_to": "07B1F48DA85A593EE4C298B5189277B9",
"sourceObjectGUID": "0E0766C0D1D88D9572502C46F389D6E0",
"sourceIsVisible": true,
"targetObjectGUID": "07B1F48DA85A593EE4C298B5189277B9",
"targetIsVisible": true,
"isCompressed": true
},
{
"_from": "0E0766C0D1D88D9572502C46F389D6E0",
"_to": "6C3F321F06624C447A6CBA9A1AE4DC2E",
"sourceObjectGUID": "0E0766C0D1D88D9572502C46F389D6E0",
"sourceIsVisible": true,
"targetObjectGUID": "6C3F321F06624C447A6CBA9A1AE4DC2E",
"targetIsVisible": true,
"isCompressed": true
},
{
"_from": "0FB0ED21690CCC1CA4ADEFE3AC458B09",
"_to": "FEDEBF9494CBC56B779101A0990E52D5",
"sourceObjectGUID": "0FB0ED21690CCC1CA4ADEFE3AC458B09",
"sourceIsVisible": true,
"targetObjectGUID": "FEDEBF9494CBC56B779101A0990E52D5",
"targetIsVisible": true,
"isCompressed": true
}
...
],
"startNode": {
"_key": "8D1A993B6739DAB65392B54D45F7052E",
"connectionId": "251",
"connLogicName": "sqlserverdbwa03testETLTRUE",
"toolName": "SQLS",
"toolType": "DB",
"displayConnectionId": "251",
"containerObjectName": "sp_built_MrrPerson_MrrCustomer",
"containerObjectPath": "AdventureWorks2014.dbo",
"containerObjectType": "SQL_STORED_PROCEDURE",
"controlFlowPath": "sp_built_MrrPerson_MrrCustomer",
"controlFlowName": "sp_built_MrrPerson_MrrCustomer",
"objectID": "ADVENTUREWORKS2014SALESCUSTOMERTABLESTOREID",
"objectGUID": "8D1A993B6739DAB65392B54D45F7052E",
"objectType": "TABLE",
"assetName": "StoreID",
"dataType": "",
"precision": "",
"scale": "",
"layerName": "Customer",
"schemaName": "SALES",
"databaseName": "AdventureWorks2014",
"tableName": "Customer",
"isObjectData": true,
"isMap": false,
"isVisible": true,
"lvl5": "",
"lvl4": "",
"lvl3": "",
"lvl2": "",
"lvl1": "",
"updatedDate": "2024-05-26T10:21:51.224Z",
"createDate": "0001-01-01T00:00:00Z",
"serverName": "",
"isSrcColumnOrphan": null
},
"depth": 12,
"direction": "Both"
}