Enhancing Data Connectivity: Octopai's Universal Connector for Databases & ETLs Tools Guide

As data demands are ever-evolving, data teams are always on a search for better understanding of their data ecosystem. The need for analysis and visualization of additional systems is growing and as such Octopai is constantly expanding its extensive coverage for out-of-the-box supported technologies in our Data Intelligence Platform.
However, as your needs progress, it is crucial to provide an overview of the complete data landscape with various systems and data flows.

We know you’re using some of the newest data systems that nobody is supporting with automation yet, and you might have built your own homegrown and customized data processes. Your lineage tool must be able to provide coverage for these as well so you can see a full, complete picture.
Therefore, Octopai has developed the Universal Connector, empowering you to add your metadata from these types of systems into Octopai’s Data Intelligence platform to get the full picture - complete lineage, data discovery and a data catalog.
You get unlimited ingestion capabilities to enrich the platform with additional lineage, allowing you to add the final piece of the puzzle and get full visibility of your data ecosystem.
This flexibility allows you to adapt quickly to your changing data landscape, and consistently get a complete view regardless of what data systems you’re using.

How it is done

Use the Octopai templates to ingest your metadata into the platform. The rest is fully automated.

What we do for you

This metadata, along with the metadata automatically ingested from out-of-the-box supported systems, is analyzed using machine learning. In turn Octopai provides you with end-to-end column-level lineage, inner system lineage, cross system lineage, data discovery and a data catalog of your entire data landscape accessible to all data users in the organization.

The Benefits

No blind spots - perform changes with confidence.
Get a clear picture of data transformations.
Increased visibility of the organization's complete data ecosystem.
Future proofing your expanding data landscape by providing access to unlimited data systems.
Add Link to our out-of-the-box technologies.

Process Description

Download the Template and fill in the required information from the bottom of this article.

• Universal Connector Links.csv
• Universal Connector Objects.csv

Universal Connector Link

Column Name	Description	Required
Process Name	Name of the process that wraps the task, for example “Workflow” in Informatica or “Package” in SSIS	No
Process Path	path of the process - for example, path of where the SSIS package is stored, including the package name and suffix. (aaa\bbb\ccc\Package Name.dtsx)	No
Process Type	The type of process - Job, Map, package...	Yes
Process Description	Short process description to be identified clearly in the lineages	No
Task Name	The task name - the atomic unit that holds the data flow within the process	Yes
Task Path	The path of the task - the location of the atomic unit that runs the process. (aaa\bbb\ccc\Package Name\container\Task Name) (like path of Data Flow)	No
Source Component	Name of the logic component in the ETL tool Example: For Informatica- name of Aggregator in the map / In case there is no component- fill in the Table name	No
Source Provider Name	Provider of source object - Ex: Oracle, SQL Server	No
Source Server	Server name of the source Object	No
Source Database	Database name of the source Object	Yes
Source Schema	Schema Name of the source Object	Yes
Source Object	Name of the source object	Yes
Source Column	Column Name in the source object	Yes
Source Data Type	Data Type of Column	No
Source Precision	Precision of Column	No
Source Scale	Scale of Column	No
Source Object Type	Type of Object - Table, View, File	Yes
Target Provider Name	Provider of target object - Ex: Oracle, SQL Server	No
Target Component	Name of the logic component in the ETL tool Example: For Informatica- name of Aggregator in the map / In case there is no component- fill in the Table name	No
Target Server	Server name of the target object	No
Target Database	Database name of the target object	Yes
Target Schema	Schema Name of the source object	Yes
Target Object	Name of the target object	Yes
Target Column	Column Name in the target object	Yes
Target Data Type	Data Type of Column	No
Target Precision	Precision of Column	No
Target Scale	Scale of Column	No
Target Object Type	Type of Object - Table, View, File	Yes
Expression	Formula/transformation between source column and target column	No
LinkType	DataFlow or ImpactAnalysis	No (Default = DataFlow)
Link Description	Documentation about the link	No (Default = Empty String)

Note: All parameters are string values. Template structure should not be changed, all parameter’s names (Column Names) are case sensitive, all values are NOT case sensitive.

Example for ETL process on Cross Lineage

The Universal Connector will link the Source and the Target for the Task Name as a main Object:

Universal Connector Objects

Column Name	Description	Required
Provider Name	Provider of object - Ex: Oracle, SQL Server	No
Server Name	Server name of the object	No
Database Name	Database name of the object	Yes
Schema Name	Schema Name of the object	Yes
Object Name	Name of the source object	Yes
Object Description	Documentation about the object	No (Default = Empty String)
Column Name	Column Name in the source object	Yes
Column Description	Short Column description	No (Default = Empty String)
Data Type	Data Type of Column	No
Is Nullable	If column accept no value	No
Precision	Precision of Column	No
Scale	Scale of Column	No
Object Type	Type of Object - Table, View, File,…	Yes

Note: All parameters are string values. Template structure should not be changed, all parameter’s names (Column Names) are case sensitive, all values are NOT case sensitive.

How to set up the Universal Connector?

Read this Article