Sidra Data Platform Resource Groups infrastructure

All the infrastructure needed in Sidra Data Platform is deployed using ARM templates. Azure Resource Manager, also known as ARM, is the deployment and management service for Azure. It provides a consistent management layer that enables to create, update, and delete resources in Azure subscription.

All the underlying cloud resources are grouped in resource groups. In Azure, a Resource Group is a container in Azure Resource Manager that holds related resources for an appplication. For every Resource Group in Sidra, there is a set of applications needed by Core and the Resource Groups with its related roles.

ARM Templates are a way to declare the Azure resources used by the platform, with all its properties in a JSON file. The orchestration script contains the parametrization and deployment order of all the resources to be deployed.

Resource Groups in Sidra

Sidra Core contains at least two different resource groups in Azure, which end up being codified in the form of ARM templates after the initialization and orchestration and parametrization process.

  • Core Resource Group
  • DSU Resource Group (minimum 1 DSU Resource Group, but there can be as many as DSUs in the Sidra Core installation)

The deployment scripts in Sidra also ensure that the defined resource naming convention, as well as the resource tags naming convention, is satisfied when creating these resources. The specifications of such resources depend on a set of pre-defined characteristics as per the Sidra installation size (see Sidra Core Deployment Installation sizes), as well as the specific data installation variables.

Besides the Sidra Core Resource Groups, the resources associated to each Sidra Client Application resources will be logically included in a different Resource Group.

Sidra Core Resource Group ARM template diagrams

Sidra Core Resource Group includes the group of Azure resources required for key transversal as well as common services across Sidra, namely:

  • App Service plan and App Services for Sidra Core API, Sidra Web, authentication and authorization services.
  • Commmon services, some of which are also transversal across all the Resource groups: App Insights, SignalR, LogAnalytics.
  • Persistence Layer through SQL Elastic Pool and SQL databases for the normal functioning of Core Sidra services: Core, Log, Operational DW, Identity Server.

Below is a logical high-level diagram depicting the key Azure resources inside the Core Resource group:

core-resource-group

Sidra DSU Resource Group includes the group of Azure resources required for the storage, orchestration, computing, cognitive and machine learning services of a deployable Data Storage Unit (DSU).

Data Storage Units (DSU) provide logical and physical isolation of data, to help with data compliance and regulations. Each DSU isolates not just the data storage, but also the compute, orchestration, intake and ML models, so they can be colocated in specific geographical regions.

Even though a Sidra implementation can have multiple DSUs, they all form part of the same Data Lake, sharing the resources in Sidra Core for modules and functionalities like the Data Catalog, Metadata Management, Security model, etc.

A Sidra DSU Resource group contains fundamentally the following types of resources:

  • Orchestration (mainly through Data Factory)
  • Storage accounts (for intake, raw as well as optimized storage)
  • Compute services (e.g. Databricks)
  • Model deployment and model management services (e.g. Azure Machine Learning Service, Search Service)

Below is a logical high-level diagram depicting the key Azure resources inside the DSU Resource group:

dsu-resource-group

Sidra Client Application Resource Group ARM template diagrams

In the case of Sidra Client Applications, there is high variability according to the limitless possibilities that can constitute a Client Application in Sidra. See section on Sidra Client Applications for more details about what constitutes a Client Application in Sidra and underlying resources.

As a general rule, most Client Applications will contain a set of common types of resources, such as:

  • Storage and persistence services
  • Compute services
  • Orchestration services
  • Common services (App Services for API, Web, KeyVault, etc)

Each Client Application template will therefore be depicted by a different ARM template diagram.