End-to-End Column Lineage

End-to-End Column Lineage is a powerful feature within Octopai, a comprehensive data management platform, that enables organizations to track and understand the complete lifecycle of specific data columns or attributes. This functionality provides users with a holistic view of the data lineage, allowing them to trace the path of a data element from its source to its final destinations, across various data processing stages.

With Octopai's End-to-End Column Lineage feature, users can easily identify the origin of a specific data column or attribute, including the data source, such as a database table or file, from which it originates. They can then follow the lineage to discover the intermediate steps involved in data transformations, such as data cleansing, aggregations, or calculations, as well as the data repositories, reports, dashboards, or downstream systems that utilize the data.

This comprehensive understanding of the data lineage is essential for several data management tasks. Firstly, it facilitates data governance by providing transparency and accountability. Users can track and document the flow of data, ensuring compliance with regulatory requirements and internal data policies. Additionally, it enables organizations to identify potential data lineage issues or inconsistencies that may impact data quality, allowing for timely resolution and improved data accuracy.

The End-to-End Column Lineage feature also simplifies the process of impact analysis. Users can easily assess the potential consequences of making changes to a specific data column by tracing its impact across the entire data landscape. This knowledge helps organizations make informed decisions regarding data transformations, system migrations, or updates, minimizing the risk of unintended consequences and ensuring smooth data operations.

Moreover, the End-to-End Column Lineage functionality enhances collaboration among data teams. Business analysts, data engineers, and data scientists can easily access and share the complete lineage information, promoting effective communication and alignment across different roles and departments. This collaborative approach streamlines data-related projects and reduces the time and effort spent on manually tracing data dependencies.

Overall, Octopai's End-to-End Column Lineage empowers organizations with a comprehensive and visual representation of data lineage, allowing users to trace the complete lifecycle of specific data columns or attributes. By leveraging this feature, organizations can strengthen data governance, improve data quality, streamline impact analysis, and foster collaboration across their data teams. With a deeper understanding of their data lineage, organizations can make informed decisions, optimize data operations, and unlock the full value of their data assets.

Top use cases include:

• Impact analysis of a change to a column in the source system
• Root cause analysis to uncover the source of reporting errors
• Column-level visualization of data flow
• Regulatory compliance audit preparation

Access the End to End Lineage dashboard from the Space Menu

Search Box (Fuzzy Search) - From reports to processes, part of the names.

(Only for columns or the component having the column)

Data Objects List by Type & Color

BLUE AREA - ETL, SP, SQL Files   

RED AREA - DB Objects, Analysis Services, Physical Objects 7)

GREEN AREA - Reports

Hop to End to End Column Lineage

E2E Column Lineage

Collapse/Expand the View from the bar under the Navigation panel

E2E Column Lineage Functionalities over each Column

Click on the three dots next to the column to open the drop menu with the following options:

• Properties - Column Properties will pops-up on the left side of the screen
• Search in Discovery - Hop to Discovery Module
• Expand Right E2E Column Lineage - Impact Analysis
• Expand Left E2E Column Lineage - Root Cause Analysis
• Override Column Lineage - Start the E2E Lineage from this Column

E2E Column Lineage Functionalities over each Table

Click on the three dots next to the table to open the drop menu with the following options:

• Properties - Table Properties will pops-up on the left side of the screen
• Search in Discovery - Hop to Discovery Module
• Show Additional Columns - Will show all the columns related to the table
  
  Same options can be selected like the E2E functionalities over each column.

E2E Column Lineage Functionalities over each Inner Square

Click on the three dots next to the inner square name to open the drop menu with the following options:

• Properties - Table Properties will pops-up on the left side of the screen
• Search in Discovery - Hop to Discovery Module
• Inner System Lineage - Hop to the Inner System Lineage

E2E Column Lineage Functionalities over each Outer Square

Click on the three dots next to the table to open the drop menu with the following options:

• Properties - Table Properties will pops-up on the left side of the screen
• Search in Discovery - Hop to Discovery Module
• Inner System Lineage - Hop to the Inner System Lineage
• Cross System Lineage - Hop to the Cross System Lineage
• View in Data Catalog - Hop to Catalog Module