Learn about Sidra Data Products¶

Data Products, also known as Apps, is another main component of Sidra Data Platform.

For each one who needs to interact with the data, its access the Data Lake will be implemented as a Data Product. They will all leverage the common Data Lake to retrieve the necessary data sets, as well as a common set of services, but might use different storage/query engines depending on the workload that they will need to execute.

Also, Data Product can be restricted to access a subset of the data stored in Data Lake using the integrated security that Azure Data Lake Gen2 provides or using Sidra Query API for fine-grained level of access restrictions.

While some applications might hold little more than a SQL Database and Power BI model against it, others might require the use of a broader set of products such as Spark clusters and messaging mechanisms.

It is in the stage of moving the data from the storage account into the specific storage of the application when specific processing and transformations might take place; this way the change to the data is more localized and the raw data is still always available for other systems or future audit/reference.

Data Products¶

To understand better what is a Data Product, you can refer to the detailed Data Products documentation pages.

You can also dive into learning more about Sidra's provided Data Products:

Data Labs¶

Data Labs aims at offering a Databricks sandbox for enabling exploratory data analysis on synchronized data from the data lake.

Data Labs comes pre-packaged via ready-to-use Data Product template. A specific instance of that template can be deployed and configured in your specific environment.

Basic SQL template¶

Sidra offers a Data Product template (vanilla version) for a basic Data Product with an SQL database for hosting relational models.

This template leverages the accelerators to build Data Products in Sidra, like authentication, authorization management, as well as the metadata synchronization with Sidra Service and pipelines to orchestrate the movement of copies of the data from the data lake to the Data Product storage and staging tables.

The Data Product deployment from this template will automatically create the staging tables for starting the transformation logic into the relational models that will be stored in the SQL Data Product database.

Business Intelligence¶

Set of tools which access and analyze the data to offers to its users a detailed, comprehensive, and intelligent overview over the state of the business using reports, dashboards, graphs, charts, summaries, and maps.

Data Quality¶

Sidra Data Platform aims to offer a unified place where the data is stored for anyone who wants to interact with it. So, as the data volume is constantly growing the data consistency is an important factor to be maintained.

Data Quality can be created and installed as a Sidra Data Product to ensure the quality of the data in the Data Lake using optimized procedures, and custom business data quality rules.