Skip to content

Data Ingestion Schema Tables

The DataIngestion schema is a core part of the Sidra metadata model. It stores metadata about the entire data ingestion lifecycle—from source configuration and pipeline orchestration to the structure and lineage of ingested assets.

This schema is organized into several categories, each supporting a different functional aspect of data ingestion in Sidra.

Data Ingestion Metadata

Tracks the ingestion of individual assets into the DSUs. This includes:

  • The structure and attributes of raw data files or extracts
  • The data sources they originated from
  • Their current status in the ingestion process

Metadata is organized hierarchically to reflect the relationships between sources, providers, and assets.

Azure Data Factory Integration

Stores metadata describing the Azure Data Factory (ADF) artifacts used by Sidra. This includes:

  • Pipelines orchestrating the movement and transformation of data
  • Dataset definitions
  • Activities and execution logic
  • Trigger configurations

These tables support Sidra’s dynamic pipeline generation and monitoring capabilities.

Data Storage Unit Configuration

Contains metadata for all configured Data Storage Units (DSUs) in the Sidra platform. This includes:

  • Logical and physical storage details
  • Region-specific configurations
  • Roles assigned to each storage resource

This metadata powers DSU-level automation, isolation, and governance.