Sidra Service Metadata¶
This section describes the metadata model used by the Sidra Service. This metadata is stored in the Sidra database and is foundational to many of the platform’s services, including data ingestion, cataloging, pipeline automation, and data product governance.
The metadata is organized into schemas that correspond to functional areas of the platform.
DataIngestion¶
The DataIngestion schema contains metadata related to:
- Assets ingested into the platform
- Azure Data Factory pipelines used for orchestration
- Configuration of Data Storage Units (DSUs), including their logical and physical organization
These tables support Sidra’s pipeline generation, ingestion tracking, and metadata-driven processing.
Apps¶
The Apps schema contains metadata related to Sidra’s Data Products—solutions that consume and transform data from the platform.
Legacy naming under active review
Historically, Sidra referred to Data Products as “Apps” or “Client Applications.” The current metadata schema still reflects this original naming. While the platform has fully embraced the term Data Product, a migration of schema and table names is pending, and will be carefully managed for backward compatibility.
Management¶
The Management schema contains system-level metadata for configuring and managing the platform. This includes:
- Global configuration settings
- Platform deployment details
- Operational and administrative references
Additional Schemas¶
These auxiliary schemas support other cross-functional capabilities in Sidra:
- DataPreview: Stores data samples captured during DSU ingestion. These previews are surfaced in the Sidra Web Data Catalog to help users understand the structure and content of loaded entities.
- DataCatalog: Contains metadata about attribute popularity and platform-wide roles and permissions used in Sidra’s built-in data catalog.
- Hangfire: Manages background job execution across the platform using Hangfire for task scheduling and reliability.
- Notifications: Stores system-generated notifications triggered by operational events within the platform.
These schemas extend the core metadata model with support for automation, observability, and usability across platform services.