How Does Sidra Work?¶

Sidra Data & AI Platform is designed to streamline the journey from raw data to business-ready outputs. It orchestrates the entire lifecycle of data ingestion, transformation, and delivery using scalable, cloud-native architecture. Sidra combines automation with modularity, making it suitable for organizations embracing data governance, domain ownership, and modern product-oriented data practices.

This page introduces Sidra’s key architectural components and how they work together to support real-time and batch data use cases.

Core Concepts¶

Sidra is built around three foundational building blocks: Supervisor, Data Storage Units (DSUs), and Data Products.

Supervisor¶

Supervisor is the central entry point for managing Sidra. It enables:

• Deployment and updating of services
• Configuration of core platform settings

This service guarantees that Sidra deployments remain reproducible, auditable, and seamlessly evolve with ongoing platform updates and as new services become available.

Learn more about Supervisor

Data Storage Units (DSUs)¶

A DSU is the central orchestrated component where data lands automatically and gets prepared for downstream usage by the business with the Data Products. DSUs offer:

• Landing zones for incoming data (structured and unstructured), alongside its metadata and related AI models
• Optimized Delta Lake storage (partitioned, compressed, query-ready)
• Isolation of infrastructure and compute for compliance and performance
• Support for optional services such as Azure Search, Cognitive Services, or ML model serving

Each DSU is region-deployable and logically separated, making it ideal for multi-tenant or geo-distributed architectures.

Data Products¶

Data Products are downstream consumers of DSU data that implement business-specific logic and serve domain needs. They can:

• Be deployed independently
• Use prebuilt or custom templates (BI dashboards, ML pipelines, APIs, apps)
• Implement governance, lineage, and access control at the product level
• Use Sidra APIs and SDKs to access metadata and DSU content

Sidra encourages ownership and autonomy through these Data Products—embracing principles from the Data Mesh movement while offering flexibility to adopt at your own pace.

How Data Flows in Sidra¶

Sidra enables a fully automated, metadata-driven orchestration process that collects and prepares data from diverse systems.

Step-by-Step Overview¶

1. Data arrives at the landing zone
   Sidra supports multiple ingestion mechanisms:

   • Structured connectors (SQL Server, Oracle, MySQL, etc.)
   • File-based ingestion from blob storage or SFTP
   • API-based sources via custom-built connectors

2. Metadata extraction and registration
   Upon arrival, Sidra reads schema metadata and registers new datasets and structures automatically.

3. Storage and optimization in the DSU
   Data is stored in an optimized Delta Lake format with partitioning, versioning, and compression. It is now available to all authorized Data Products using Sidra's reverse ETL Engine, the Sync processes.

4. Data Product consumption and transformation
   Data Products can query, transform, and expose the data as needed—whether through dashboards, ML models, or external-facing APIs.

5. Shared services enable governance and observability
   Features like auditing, monitoring, anomaly detection, and the Data Catalog provide full visibility and control across the platform. Other more specific services, such as the API Builder and FHIR services, can accelerate the development of data products for Sidra users.