Learn about Sidra Service¶

Sidra Service (called before Sidra Core) is one of the main components of Sidra Data Platform.

In this section we will delve into the following concepts:

• Orchestration - how data movement is orchestrated in Sidra Data Platform
• Computation - how data and computation management is performed in Sidra Data Platform

• Metadata - metadata definition used by the platform:

  • Generic artifacts - a list of triggers, datasets and pipelines that are predefined in Sidra Data Platform

• Data Ingestion - follow the intake of the data from the data source until the Data Lake with the different Data Intake Processes that take part in Sidra

Common Services¶

The Sidra Service infrastructure will provide a set of common services to be used – and extended if needed – by the data lake clients.

These services are:

Operations Management
A centric view of the performance and reliability metrics is a must on a system like the one proposed here. A mechanism that allow to analyze the performance of each of the components and react to anomalies with auto-scaling or remediation actions will be implemented on the platform.

Security and Identity
A common UI and base security model will be provided to all applications siting on the platform. This model will be based on AD identities and will be extensible by each client.

API Management
Some of the applications will provide a set of APIs to interact with them. These should be handled by the API Management, which will allow to scale, secure, audit and monitor the different services provided by the data platform through its APIs.

Data Catalog
Some datasets are going to be potentially used across the complete platform. This data will be exposed from a common interface (API and SQL/Redis) to the Data Products, allowing to minimize the potential errors if every application needs to do the same tasks, as well as opening optimization opportunities.

Data Ingestion Anomaly Detection
A mechanism for the management of the enterprise data assets, including the data discovery and tagging, is something that a system which aims to promote the cohabitation of many data sets needs to provide.

Management UI
A contemporary, web-based management UI for developers and administrators aimed meet their requirements. It offers a visual user widget to ingest new data, an operational status tracking dashboard and Sidra log management.