Learn about Sidra Core¶
Core is one of the main components of Sidra Data Platform.
In this section we will delve into the following concepts:
- Orchestration - how data movement is orchestrated in Sidra Data Platform
- Computation - how data and computation management is performed in Sidra Data Platform
- Data flow process - follow the intake of the data from the data source until the Data Lake
- Metadata - metadata definition used by the platform
- Custom activities - custom activities available in Sidra Data Platform
- Generic artifacts - a list of triggers, datasets and pipelines that are predefined in Sidra Data Platform
The core infrastructure will provide a set of common services to be used – and extended if needed – by the data lake clients.
These services are:
A centric view of the performance and reliability metrics is a must on a system like the one proposed here. A mechanism that allow to analyse the performance of each of the components and react to anomalies with auto-scaling or remediation actions will be implemented on the platform.
Security and Identity
A common UI and base security model will be provided to all applications siting on the platform. This model will be based on AD identities and will be extensible by each client.
Some of the applications will provide a set of APIs to interact with them. These should be handled by the API Management, which will allow to scale, secure, audit and monitor the different services provided by the data platform through its APIs.
Some datasets are going to be potentially used across the complete platform. This data will be exposed from a common interface (API and SQL/Redis) to the client applications, allowing to minimize the potential errors if every application needs to do the same tasks, as well as opening optimization opportunities.
Data Ingestion Anomaly Detection
A mechanism for the management of the enterprise data assets, including the data discovery and tagging, is something that a system which aims to promote the cohabitation of many data sets needs to provide.
A contemporary, web-based management UI for developers and administrators aimed meet their requirements. It offers a visual user widget to ingest new data, an operational status tracking dashboard and core log management.