How does Sidra work?¶
Main concepts¶
Some of the first concepts to learn about Sidra are:
- Sidra Data Platform uses Supervisor, an interface allowing you to install Sidra services and update them.
- DSUs are groups of resources for processing data in Sidra. They enable data orchestration, storage, processing, and services like Azure Cognitive services, Model Serving or Azure Search for knowledge store.
- Sidra services and data in the DSU are made available to Data Products. These are separate Azure resources for performing business logic on data (exploratory analysis, Power BI reports or data transformations). Data Products synchronize transparently with the DSU and Sidra Shared Services for common security, authorization and authentication schemas.
Process¶
Data orchestration in Sidra begins by defining which data is collected and how. We provide a solution that collects data points from different sources and processes them in virtually real-time.
Sidra orchestration uses data pipelines, a set of processing elements sequentially connected. This approach reduces manual intervention and enables smooth and automated end-to-end data flow.
The steps for a data ingestion in Sidra are:
-
Structured or unstructured data arrives at the landing zone (an Azure storage container) or through Sidra database connector plugins (Azure SQL, SQL Server, MySQL, Oracle...).
-
Data is registered based on its metadata.
-
The raw data gets saved in the data lake (optimized storage), readily available to be accessed by Data Products.