Skip to content

Data Ingestion Hierarchy in Sidra

When content is ingested into the Sidra Data & AI Platform, metadata is automatically captured and stored to support tracking, data lineage, governance, and data product automation.

This metadata is organized as a set of interrelated elements that form the metadata model in Sidra. These elements are structured hierarchically to reflect both data lineage and organizational ownership.

Metadata Model Elements

  • Asset
    Represents an individual piece of content ingested into the platform. Each Asset includes metadata such as a unique identifier, ingestion timestamp, and is always linked to an Entity.

  • Entity
    Defines the common structural characteristics of a group of Assets. Each content ingestion generates an Asset, and Assets with the same structure are grouped under the same Entity. An Entity includes metadata about formatting, separators, encoding, and null handling. Entities are associated with a set of Attributes and belong to a Provider.

  • Attribute
    Describes the structure of the content at the column level—data type, position, and rules for ingestion. Attributes are defined under an Entity.

  • Provider
    A logical group of related Entities, usually based on data source or ownership. Each Provider is assigned to a Data Storage Unit (DSU).

  • Data Storage Unit (DSU)
    A logical and physical isolation boundary in the data platform. Each DSU acts as a container for Providers and their associated metadata and data assets.

Hierarchical Structure

The metadata model follows this hierarchy:

Data Intake Metadata Hierarchy

This structure ensures a clear lineage from ingested data down to column-level attributes and supports scalable governance.

Adding a New Data Source

When onboarding a new type of content (e.g., a new data source), it must be integrated into the metadata model through an Entity. The process may require:

  • Entity: Always required—defines the schema for the new data.
  • Attributes: One per column or field in the content.
  • Provider: Only required if the Entity doesn't logically fit under an existing Provider.
  • DSU: Already created during Sidra deployment. Only create a new DSU if strict isolation is required.

Optional: - AttributeFormats: Used if the Attributes need transformation or normalization during ingestion.

Shared Metadata Columns

Some columns are common across multiple tables in the metadata model:

  • SecurityPath
    A hierarchical path that defines access control. Format:
{DataStorageUnitId}/{ProviderId}/{EntityId}

Example: 1/10/100 identifies an Entity with ID 100, under Provider 10, in DSU 1.

  • ParentSecurityPath
    The SecurityPath of the parent element in the metadata hierarchy.

Example: For the Entity above, its ParentSecurityPath would be 1/10.

These paths allow Sidra to enforce fine-grained access control throughout the metadata model.