Skip to content

Sidra Service Overview

Sidra Service (formerly known as Sidra Core) is a foundational component of the Sidra Data & AI Platform. It delivers the metadata infrastructure, ingestion pipeline orchestration, and shared services needed to operate and manage data at scale.

common-services

Sidra Service plays a critical role in centralizing platform logic that is used across Data Products, pipelines, APIs, and orchestration processes.

From Monolith to Service-Oriented Architecture

Originally, all platform functionality in Sidra was delivered as a single monolithic application known as Sidra Core. As the platform matured, we adopted a service-oriented architecture. This transformation led to the development of the Supervisor, Sidra's service management layer, and the extraction of discrete responsibilities into dedicated services.

Today, Sidra Service continues to evolve as part of this migration. Several components that once lived within the monolith have already been moved out to standalone services, including:

  • Authentication Service: Manages identity and login processes.
  • Authorization Service: Manages roles and access control.

Other components still housed within Sidra Service will eventually follow this same path. These include:

  • The Data Catalog, which provides AI-assisted search and metadata discovery.
  • The Web-based Management UI, used to administer Data Intake Processes, Data Products, and view operational dashboards.

Key Responsibilities of Sidra Service

Sidra Service currently delivers a core set of features essential to the operation of the platform:

Metadata System

A robust metadata model that defines the structure of data sources, pipelines, attributes, and entities. This model supports all ingestion and transformation processes and is accessible via API.

Data Intake Process (DIP) Management and Pipeline Auto-Generation

Sidra Service provides tools and APIs to create and configure DIPs. These automate the ingestion of data from structured and unstructured sources into the platform using predefined templates and pipelines. These DIPs generate extraction and transformation pipelines using Azure Data Factory and Databricks notebooks. These are customized per connector and based on metadata definitions.

Sidra Core API

The Sidra API exposes endpoints for interacting with metadata, data ingestion, pipeline control, and platform services. It forms the foundation for integration with custom tooling, automation workflows, and operational management.

Learn More

Sidra Service provides the backbone for orchestration, metadata management, and platform-wide services. As the platform continues to evolve, its modular design ensures long-term scalability, composability, and maintainability.