Data Ingestion Schema Tables¶
The DataIngestion
schema is a core part of the Sidra metadata model. It stores metadata about the entire data ingestion lifecycle—from source configuration and pipeline orchestration to the structure and lineage of ingested assets.
This schema is organized into several categories, each supporting a different functional aspect of data ingestion in Sidra.
Data Ingestion Metadata¶
Tracks the ingestion of individual assets into the DSUs. This includes:
- The structure and attributes of raw data files or extracts
- The data sources they originated from
- Their current status in the ingestion process
Metadata is organized hierarchically to reflect the relationships between sources, providers, and assets.
Azure Data Factory Integration¶
Stores metadata describing the Azure Data Factory (ADF) artifacts used by Sidra. This includes:
- Pipelines orchestrating the movement and transformation of data
- Dataset definitions
- Activities and execution logic
- Trigger configurations
These tables support Sidra’s dynamic pipeline generation and monitoring capabilities.
Data Storage Unit Configuration¶
Contains metadata for all configured Data Storage Units (DSUs) in the Sidra platform. This includes:
- Logical and physical storage details
- Region-specific configurations
- Roles assigned to each storage resource
This metadata powers DSU-level automation, isolation, and governance.