Skip to content

Installing Sidra

Information

Ensure you have a Sidra license before starting the process. For assistance or to obtain a license, please contact us at [email protected].

This document provides information for deploying Sidra in your Azure subscription by deploying first Sidra's Supervisor, which will bootstrap the rest of the services in turn.

Prerequisites

The requirements for a proper deployment process are:

1. Supervisor's deployment

The deployment consists of two main steps:

  1. Deployment of Azure Active Directory (AAD) App registrations by executing the PowerShell script: Pre-deploy.ps1.
  2. Deployment of Sidra's Supervisor ARM template by using "Deploy a custom template" feature in Azure Portal.

Step 1. Pre-deployment script

A PowerShell script must be run before the Supervisor is deployed. This script will create some necessary resources to start Sidra's installation like:

  • Managed Identities(stored within the Azure Resource Group):

    • Identity with "Directory-readers" privileges
    • Identity with "Application-administrator" privileges
  • With the appRegistrationName assigned on the Pre-deploy.ps1 (recommendable name: <<CompanyName>>.Supervisor.<<Environment>> ), the script will create two Azure Active Directory (AAD) App registrations:

    • <<CompanyName>>.Supervisor.<<Environment>>.Manager. This app registration is used to log in to Azure when the DSU is deployed. These values secrets are stored in Supervisor keyVault:
      • AADApplications--Manager--Password
      • AADApplications--Manager--ApplicationId
    • <<CompanyName>>.Supervisor.<<Environment>>.IdentityServerAdAccess. This app registration is used by Authentication Service. When this app registration is created, the values secrets are also included in the Supervisor keyVault for authentication proposals. These secrets are:
      • AADApplications--IdentityServer--ApplicationId
      • AADApplications--IdentityServer--Password
      • AAD--0--ClientSecret
  • Azure Resource Group

Script parameters

For the PowerShell script to execute successfully, these are the necessary parameters:

Parameter name Description Example
appRegistrationName Name used in the creation of the ADD app registrations Sidra.Supervisor.Dev
subscriptionId The subscription's ID in which the resources are going to be deployed 000000-0000-0000-0000-000000000000
tenantId The directory ID of the used subscription 000000-0000-0000-0000-000000000000
resourceGroupName Name of the resource group which will have the created managed identities. This name is going to be used later for Sidra's resource group name Sidra.Supervisor.Dev
resourceGroupLocation Location where the resource group is created northeurope
Subscriptions and directories info in Azure Portal

These parameters can be retrieved from Azure Portal: subscription tenantId

Execute Command

The steps for deploying the script are:

  1. Open a PowerShell 7 cmd.
  2. In the cmd, go to the location where the Pre-deploy.ps1 script is.
  3. Execute this command in the cmd:

    Remember to replace the parameter placeholders for the correct value that you prepared in the previous step.

    .\Pre-deploy.ps1 -appRegistrationName [appRegistrationName] -tenantId [tenantId] -subscriptionId [subscriptionId] -resourceGroupName [resourceGroupName] -resourceGroupLocation [resourceGroupLocation]
    

    Example:

    .\Pre-deploy.ps1 -appRegistrationName "Sidra.Supervisor.Dev" -tenantId "000000-0000-0000-0000-000000000000" -subscriptionId "000000-0000-0000-0000-000000000000" -resourceGroupName "Sidra.Supervisor.Dev" -resourceGroupLocation "northeurope"
    
    • During the execution, a webpage from Azure is going to pop up in your default web browser asking you to log in. Log in with a user that satisfies the prerequisites mentioned before.
  4. Now, the command will output some values. This is an example:

    AAD applications
    ----------------------------------------------------------------
    App Registration Identity Server Id: 00000000-0000-0000-0000-000000000000
    App Registration Manager Id: 00000000-0000-0000-0000-000000000000
    App Registration Manager Object Id: 00000000-0000-0000-0000-000000000000
    ----------------------------------------------------------------
    

    From this script output, take note of those parameters values: App Registration Identity Server Id and App Registration Manager Id.

Step 2. Enable Unity Catalog

Starting on version 2024.06, Sidra now supports Databricks Unity Catalog as metastore backend.

More info about Unity Catalog in Databricks Documentation.

Data on old Sidra deployments need to be moved to Unity Catalog to keep it available in Sidra 2024.06. Click here to see how to do it.

Unfortunately, this step cannot be automated yet and has to be done manually between the pre-deployment script execution and the actual Sidra deployment.

You need to go to Databricks Account Console using a Databricks Account Administrator.

  • If you don´t have one, for example, if this is the first time you use Databricks, you need to use the Entra Id Global Administrator. With it, you can grant Account Administrator to other users.
  • If you cannot access to the Account Console maybe is because you have never deployed a Databricks Workspace in your Azure Tenant. You can deploy one using the Portal and remove it afterwards, the console will be available since then.

Once you have access to the console, there are three things to do:

  • Take note of the Databricks Account Id. You can find it in the upper right menu as in the figure below.

    Account Id

  • Grant Account Admin permissions to the Sidra Manager Service Principal you have got from the pre-deployment script, it is the App Registration Manager Id value. To do so, go to Users and Groups...

    Users and Groups

    Look for the Sidra Manager Service Principal

    Service Principals

    On the roles section, choose Account Admin

    Account Admin

    Finally, on the Principal Information section, generate a Client Secret and take note of it, as you will not be able to see it later.

    Client Secret

With that, you have finished the manual step. The next step is the Supervisor's ARM deployment, in which you will find some parameters that corresponds with the AccountId and Client Secret you have noted.

Step 3. Supervisor's ARM template deployment

Now that we have executed the pre-deployment script, we are ready to deploy the Supervisor:

  1. Go to Azure Portal and log in using the same user used for the execution of the pre-deployment script.
  2. Go to "Deploy a custom template" feature. You can use the top search bar in Azure Portal to search for it.
  3. Now click in "Build your own template in the editor".
  4. Replace the content on the template with Supervisor's ARM template (Supervisor.json file).

  5. Click "Save" button.

  6. Now, the following parameter's form of the Supervisor's ARM template will be displayed:

    Parameter name Description Example Required
    Subscription Selects the same subscription in which you executed the pre-deployment script Sidra Yes
    Resource Group Sets the same name used in the pre-deploy script Sidra Yes
    Region Selects the region in which you want to deploy Sidra's resources North Europe Yes
    Deployment Prefix Sets a prefix value for the name of the deployed resources. Maximum length of four characters. Beginning with a numeric character is not allowed sds Yes
    Environment Sets the environment name. Maximum length of four characters dev Yes
    Supervisor Admin Email Sidra's administrator email [email protected] Yes
    App Registration Identity Server Id Sets the value of "app_registration_identity_server/ApplicationId" from the output of the pre-deployment script 000000-0000-0000-0000-000000000000 Yes
    App Registration Manager Id Sets the value of "manager/ApplicationId" from the output of the pre-deployment script 000000-0000-0000-0000-000000000000 Yes
    App Registration Manager Object Id Sets the value of "manager/ApplicationId" from the output of the pre-deployment script 000000-0000-0000-0000-000000000000 Yes
    Company Name Sets the company name Sidra Yes
    Vnet Name Name of the existing VNet SidraVNet
    Vnet Resource Group Name of the existing VNet resource group Sidra.VNet
    Database Subnet Name Name of the existing database subnet SidraDB
    Container App Subnet Name Name of the existing container app subnet SidraContainerApp
    App Subnet Name Name of the existing app subnet SidraAppSubnet
    Storage Subnet Name Name of the existing storage subnet SidraStorage
    Default Subnet Name Name of the default subnet SidraDefSubnet
    Default Dsu Databricks Host Subnet Name Name of the default Databricks host subnet SidraHost
    Default Dsu Databricks Cluster Subnet Name Name of the default Databricks cluster subnet SidraCluster
    Databricks Account Id Sets the value of "Account Id" used to enable Unity Catalog 000000-0000-0000-0000-000000000000 Yes
    Databricks Admin Client Secret Sets the value of "Databricks Admin Client Secret" got when enable Unity Catalog doseXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Yes
    Default DSU Type Type of DSU to deploy. It can be 'Databricks' or 'Fabric'. The default value is Databricks. Databricks
    Fabric User Fabric user, this value will be mandatory if you selected dsuType = Fabric [email protected]
    Fabric Password Fabric password, this value will be mandatory if you selected dsuType = Fabric 000000-0000-0000-0000-000000000000
    Fabric User Object Id Fabric user object id, this value will be mandatory if you selected dsuType = Fabric 000000-0000-0000-0000-000000000000
    Sidra Admin Object Id Admin Sidra object id, this value will be mandatory if you selected dsuType = Fabric 000000-0000-0000-0000-000000000000

    The optional parameters are those related with the integration in the Supervisor deployment of an existing VNet from the customer. By default, a new VNet is created with the Supervisor deployment.

    Prerequisites for existing VNet integration

    In the resources group where the existing VNet is located, verify the following:

    • The existing VNet is in the same location where Sidra is going to be deployed
    • The VNet contains the next subnets:
    Subnets Service Endpoints Subnet Delegation Network Security Group
    container_app - Microsoft.KeyVault
    - Microsoft.Sql
    - Microsoft.Storage
    Microsoft.App/environments -
    database - Microsoft.KeyVault
    - Microsoft.Sql
    - Microsoft.Storage
    - -
    app - Microsoft.KeyVault
    - Microsoft.Web
    - Microsoft.Storage
    - Microsoft.CognitiveServices
    - -
    Storage - Microsoft.KeyVault
    - Microsoft.Storage
    - -
    databricks_cluster - Microsoft.KeyVault
    - Microsoft.Storage
    Microsoft.Databricks/ workspaces *Example: sds-t009-vnet-ngs-databricks-cluster
    databricks_host - Microsoft.KeyVault
    - Microsoft.Storage
    Microsoft.Databricks/ workspaces *Example: sds-t009-vnet-ngs-databricks-host

    *For the subnets associated with Network Security Groups and the creation of these NSG, in the same resource group where is the VNet, create the NSG and name it as: <name> + <select region>. Later, select the created NSG and associate it to the subnet.

    • There must be two NGSs: ngs-databricks-cluster and ngs-databricks-host.

    • In the case of deploying an additional DSU, the NSGs and the databricks_cluster and databricks_host subnets (Databricks requires only one Databricks for the deployment) must be duplicated. The Storage and Default subnets necessary to create an additional DSU can be shared between DSUs.

    For more information about how to deploy an additional DSU, check this documentation.

    For more information about how to deploy a DSU using Fabric, check this documentation.

  7. Click "Review + Create" button.

  8. Azure will start to validate the parameters. If everything is in order, the user will receive a notification stating Validation Passed.
  9. Click "Create" button.
  10. The deployment process will start and you will be redirect to the deployment page overview. Here you can check if the deployment is working well. When everything has finished you will see the message "Your deployment is complete.".
  11. Click to "Go to resource" and you will be redirect to the resource group with all the created resources.

2. Sidra's deployment

With the steps above, Supervisor's deployment must be completed. The Supervisor will deploy the services that compound Sidra (Sidra, Authorization and Authentication services) and enable the management of updates for each service. After Supervisor deployment, access to Supervisor web interface:

  1. Go to Supervisor Resource Group.
  2. Within the Container Application for Supervisor Web, click on the URL. supervisor-url
  3. Verify credentials and the automatic installation of Sidra will start along with the Authorization and Authentication services.
  4. Once installed, a screen will display the installed and additional services, enabling the update of each of them. supervisor

FAQ

  1. I am having an error executing the pre-deployment script. What am I doing wrong?

    • If the error message states something like "Forbidden - Authorization_RequestDenied" or "The client <client> with <user-id> does not have authorization or an ABAC condition not fulfilled to perform action", check that the user you are using to execute the script has all the permissions explained in the prerequisites above.
    • If the error is something related with one o more of the parameters of the script, check that the values you are using are correct and similar to the examples this document provides.
  2. Azure Portal is displaying the message Validation failed after clicking the "Review + Create" button. What is happening?

    • Check that you are filling all the parameters of the form and that the values you are using are correct. Compare them with the examples this document provides.
  3. In the deployment overview page, I am having the message Deployment has failed or the variant Your deployment failed. Error image

    • Check which resource has failed from the resource list and read the error message (it must have a red exclamation with a link to "Error details"). Most of the times it will point what is wrong. Likely, the error will be related with one of the values used in the parameter's form. Check that the values used satisfy the conditions explained and are similar to the examples this document provides.