Data Lineage and Data lifecycle

[niilahti]

Requirements to be considered:

Case: The retention period of the data must be defined: whether the data (results of data product) is deleted immediately after the service / end process is provided, whether it is subject to a time-limited retention right or obligation, or whether it is stored permanently.

[niilahti]

Possible solution:

dataLifecycle element under Data Licensing -component. It refers to the entire period of time that data / or data product exists in your system.
dataLineage element under DataOps -component. It captures the origin of data, the sequence of operations associated with data, and status after operation. Possible data lineage analysis in machine readable format, e.g. JSON or XML. Then datalineage tool needs to be explained, e.g. data catalog tools.

[niilahti]

I will create the pull requests.

[niilahti]

TCS 15.11.2022 decision, Open: Needs more details and discussion/research. Anyone interested can comment and commit.

[manfred-sorg-basf]

IMHO we need schemas for lineage at least if you understand lineage as transformation from sources:

(source) -> (schema) -> (lineage) -> (schema) -> (data access)

We are using YARRRML Sources, JSON Schemas and most probably we will be using SDTL (Structured Data Transformation Language) ... until ODPS contains equivalent abilities.