Skip to content

Latest commit

 

History

History
151 lines (128 loc) · 9.4 KB

metadata-change-log-event.md

File metadata and controls

151 lines (128 loc) · 9.4 KB
Raw

Metadata Change Log Event V1

Event Type

MetadataChangeLogEvent_v1

Overview

This event is emitted when any aspect on DataHub Metadata Graph is changed. This includes creates, updates, and removals of both "versioned" aspects and "time-series" aspects.

Disclaimer: This event is quite powerful, but also quite low-level. Because it exposes the underlying metadata model directly, it is subject to more frequent structural and semantic changes than the higher level Entity Change Event. We recommend using that event instead to achieve your use case when possible.

Event Structure

The fields include

Name Type Description Optional
entityUrn String The unique identifier for the Entity being changed. For example, a Dataset's urn. False
entityType String The type of the entity being changed. Supported values include dataset, chart, dashboard, dataFlow (Pipeline), dataJob (Task), domain, tag, glossaryTerm, corpGroup, & corpUser. False
entityKeyAspect Object The key struct of the entity that was changed. Only present if the Metadata Change Proposal contained the raw key struct. True
changeType String The change type. UPSERT or DELETE are currently supported. False
aspectName String The entity aspect which was changed. False
aspect Object The new aspect value. Null if the aspect was deleted. True
aspect.contentType String The serialization type of the aspect itself. The only supported value is application/json. False
aspect.value String The serialized aspect. This is a JSON-serialized representing the aspect document originally defined in PDL. See https://github.com/datahub-project/datahub/tree/master/metadata-models/src/main/pegasus/com/linkedin for more. False
previousAspectValue Object The previous aspect value. Null if the aspect did not exist previously. True
previousAspectValue.contentType String The serialization type of the aspect itself. The only supported value is application/json False
previousAspectValue.value String The serialized aspect. This is a JSON-serialized representing the aspect document originally defined in PDL. See https://github.com/datahub-project/datahub/tree/master/metadata-models/src/main/pegasus/com/linkedin for more. False
systemMetadata Object The new system metadata. This includes the the ingestion run-id, model registry and more. For the full structure, see https://github.com/datahub-project/datahub/blob/master/metadata-models/src/main/pegasus/com/linkedin/mxe/SystemMetadata.pdl True
previousSystemMetadata Object The previous system metadata. This includes the the ingestion run-id, model registry and more. For the full structure, see https://github.com/datahub-project/datahub/blob/master/metadata-models/src/main/pegasus/com/linkedin/mxe/SystemMetadata.pdl True
created Object Audit stamp about who triggered the Metadata Change and when. False
created.time Number The timestamp in milliseconds when the aspect change occurred. False
created.actor String The URN of the actor (e.g. corpuser) that triggered the change.

Sample Events

Tag Change Event

{
    "entityType": "container",
    "entityUrn": "urn:li:container:DATABASE",
    "entityKeyAspect": null,
    "changeType": "UPSERT",
    "aspectName": "globalTags",
    "aspect": {
        "value": "{\"tags\":[{\"tag\":\"urn:li:tag:pii\"}]}",
        "contentType": "application/json"
    },
    "systemMetadata": {
        "lastObserved": 1651516475595,
        "runId": "no-run-id-provided",
        "registryName": "unknownRegistry",
        "registryVersion": "0.0.0.0-dev",
        "properties": null
    },
    "previousAspectValue": null,
    "previousSystemMetadata": null,
    "created": {
        "time": 1651516475594,
        "actor": "urn:li:corpuser:datahub",
        "impersonator": null
    }
}

Glossary Term Change Event

{
    "entityType": "dataset",
    "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:hdfs,SampleHdfsDataset,PROD)",
    "entityKeyAspect": null,
    "changeType": "UPSERT",
    "aspectName": "glossaryTerms",
    "aspect": {
        "value": "{\"auditStamp\":{\"actor\":\"urn:li:corpuser:datahub\",\"time\":1651516599479},\"terms\":[{\"urn\":\"urn:li:glossaryTerm:CustomerAccount\"}]}",
        "contentType": "application/json"
    },
    "systemMetadata": {
        "lastObserved": 1651516599486,
        "runId": "no-run-id-provided",
        "registryName": "unknownRegistry",
        "registryVersion": "0.0.0.0-dev",
        "properties": null
    },
    "previousAspectValue": null,
    "previousSystemMetadata": null,
    "created": {
        "time": 1651516599480,
        "actor": "urn:li:corpuser:datahub",
        "impersonator": null
    }
}

Owner Change Event

{
    "auditHeader": null,
    "entityType": "dataset",
    "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:hdfs,SampleHdfsDataset,PROD)",
    "entityKeyAspect": null,
    "changeType": "UPSERT",
    "aspectName": "ownership",
    "aspect": {
        "value": "{\"owners\":[{\"type\":\"DATAOWNER\",\"owner\":\"urn:li:corpuser:datahub\"}],\"lastModified\":{\"actor\":\"urn:li:corpuser:datahub\",\"time\":1651516640488}}",
        "contentType": "application/json"
    },
    "systemMetadata": {
        "lastObserved": 1651516640493,
        "runId": "no-run-id-provided",
        "registryName": "unknownRegistry",
        "registryVersion": "0.0.0.0-dev",
        "properties": null
    },
    "previousAspectValue": {
        "value": "{\"owners\":[{\"owner\":\"urn:li:corpuser:jdoe\",\"type\":\"DATAOWNER\"},{\"owner\":\"urn:li:corpuser:datahub\",\"type\":\"DATAOWNER\"}],\"lastModified\":{\"actor\":\"urn:li:corpuser:jdoe\",\"time\":1581407189000}}",
        "contentType": "application/json"
    },
    "previousSystemMetadata": {
        "lastObserved": 1651516415088,
        "runId": "file-2022_05_02-11_33_35",
        "registryName": null,
        "registryVersion": null,
        "properties": null
    },
    "created": {
        "time": 1651516640490,
        "actor": "urn:li:corpuser:datahub",
        "impersonator": null
    }
}

FAQ

Where can I find all the aspects and their schemas?

Great Question! All MetadataChangeLog events are based on the Metadata Model which is comprised of Entities, Aspects, and Relationships which make up an enterprise Metadata Graph. We recommend checking out the following resources to learn more about this:

You can also find a comprehensive list of Entities + Aspects of the Metadata Model under the Metadata Modeling > Entities section of the official DataHub docs.