Deploy an additional Data Storage Unit¶

This manual explains the process for deploying an additional Data Storage Unit (DSU) within Sidra. The initial step involves setting up multiple subnets, which is essential for supplying the API endpoint with the details required for the new DSU deployment.

Step 1. Configure subnets¶

Go to the Azure portal, and, in the Sidra resource group, click on the Virtual Network resource. By clicking on + Subnet, you can find the configuration menu.

Databricks private subnet¶

This is the subnet of the Databricks cluster of the new DSU (it must be the new subnet in the VNET that Core/Sidra Service uses).

The subnet configuration must take into account:

1. Selection of the next service endpoints:

   • Microsoft.KeyVault
   • Microsoft.Storage

2. Subnet Delegation: Microsoft.Databricks/workspaces.

3. The Network Security Group in the resources group must be associated to the subnet. For the creation of the NSG (for example, sds-t009-vnet-ngs-databricks-cluster), in the same resource group where is the VNET, create the Network security group and name it as: <name> + <select region>. Later, select the created NSG and associate it to the subnet.

   

Databricks public subnet¶

This is the subnet of the Databricks host of the new DSU (it must be the new subnet in the VNET that Core/Sidra Service uses).

The subnet configuration must take into account:

1. Selection of the next service endpoints:

   • Microsoft.KeyVault
   • Microsoft.Storage

2. Subnet Delegation: Microsoft.Databricks/workspaces.

3. The Network Security Group in the resources group must be associated to the subnet. For the creation of the NSG (for example, sds-t009-vnet-ngs-databricks-host), in the same resource group where is the VNET, create the Network security group and name it as: <name> + Region: <select region>. Later, select the created NSG and associate it to the subnet.

Default subnet¶

This is the default subnet used by the additional DSU, it might be the one already used in the first DSU or a new one.

The subnet configuration must take into account the selection of the next service endpoints:

• Microsoft.KeyVault
• Microsoft.Web
• Microsoft.Storage
• Microsoft.CognitiveServices

Storage subnet¶

This is the storage subnet of the new DSU, it might be the one already used in the first DSU or a new one.

The subnet configuration must take into account the selection of the next service endpoints:

• Microsoft.KeyVault
• Microsoft.Storage

Step 2. Configure the API Endpoint¶

Deploy the additional DSU through the next endpoint, filling the body as it follows.

POST/api/Plugin/dsu

{
    "resource_name_suffix" : "dsu2",
    "databricks_private_subnet" : "databricks_clusters2",
    "databricks_public_subnet" : "databricks_hosts2",
    "default_subnet" : "apps2",
    "storage_subnet" : "storages2"
}

Where:

• resource_name_suffix:

  • This is the suffix used by the new resource group (Core/Sidra Service resource group name).
  • As maximum, 4 characters are allowed.
  • It is also used as suffix for the resources within the resource group.

• databricks_private_subnet: subnet name of the Databricks cluster of the new DSU (previously configured).

• databricks_public_subnet: subnet name of the Databricks host of the new DSU (previously configured).
• default_subnet: default subnet name used by the additional DSU (previously configured).
• storage_subnet: storage subnet name of the new DSU (previously configured).