Setting up a Sidra installation using the CLI tool

The following guide covers how to install a whole Sidra environment from scracth. To do so, a CLI tool is provided to ease the process. The purpose of this document is to guide through the needed actions that must be taken with the CLI to deploy a new and totally functional Sidra platform.

Requirements

To be able to setup the complete Sidra environment we must own - An Azure subscription - An Azure DevOps account

Process overview

In order to setup a working development environment, the following steps are needed

  • Creation of AAD applications on Azure on each environment (development, test and production)with the needed permissions to allow the Sidra platform to access the required resources
  • Setup of Azure DevOps
    • Creation of Sidra solution source code repository
    • Creation of CI/CD pipelines
    • Setup of Azure DevOps variable groups for the CI/CD pipelines

In the following sections we are going to see how these steps are achieved and drastically simplified with the CLI tool

CLI overview

When the CLI is opened, it displays the available commands that can be used. To open it, just go to the directory where the executable is located, open a new CMD window, type sidra and press enter.

Sidra CLI welcome screen

All commands contains a description and can be asked for help in order to know what options that command can be passed. To request help for any command, just type sidra <COMMAND> --help

For example, to know what parameters are needed to be passed to the sidra install aadapplications command, type sidra install aadapplications --help and the information will be displayed:

Command help

Setup with the CLI

Following are the details of each command to use with the CLI to setup a Sidra installation

Creating the Azure DevOps profile

Several commands through the process need to be passed the same information. To avoid this, there is an specific command that creates a profile with that common shared information between them, which is the profile command.

So, to start the process, create a new profile. You can display the information of the parameters that need to be passed using sidra profile create --help.

Execute the command, filling the needed information:

sidra profile create --devOpsProjectUrl "<DEV_OPS_PROJECT_URL>" --devOpsPAT "<DEV_OPS_PAT>" --devOpsProject "<DEV_OPS_PROJECT>" --environment "<ENVIRONMENT>"

Installing the Active Directory applications

In order to work properly, Sidra needs to create several applications in the Azure Active Directory to access different resources, like access to the DataFactory APIs or Identity Server federation among others.

The applications are created per environment, so they are not shared between the different created environments.

The command will create the following:

  • The AAD applications.
  • An Azure Keyvault resource in which the sensitive information about the created applications will be stored.
  • The Core Resource group, for the mentioned KeyVault.

Execute the command, filling the needed information. Remember that you can use the CLI help to know what each parameter is for.

sidra install aadapplications --tenant "<TENANT>" --subscription "<SUBSCRIPTION>" --location "<LOCATION>" --keyvaultName "<KEYVAULT_NAME>" --identityServerUrl "<IDENTITY_SERVER_URL>" --customerClientId "<CUSTOMER_CLIENT_ID>" --customerClientSecret "<CUSTOMER_CLIENT_SECRET>" --resourcegroup "<RESOURCEGROUP>"

The detail of the created applications can be found on the Managed Applications blade on Azure portal. The name of the applications will follow the pattern <DEV_OPS_PROJECT>.<ENVIRONMENT>.<APP_SCOPE>.

Currently 3 applications are created

  • <Project>.<Env>.IdentityServerAdAccess for Identity Server federation
  • <Project>.<Env>.Manager to manage operations on Azure resources, primarily needed for the Data Factory
  • <Project>.<Env>.VSTS to manage Azure DevOps and Azure resource deployments

Deploying the source code

The platform is deployed using different PowerShell scripts and ARM templates that contain the definition of all the Azure resources used by Sidra and the orchestration to deploy them. Apart from this, there are webjobs, an API and custom activities used in DataFactory.

All of them are dispatched as Nuget packages. These Nuget packages are then installed in the Sidra solution.

Finally, to perform the deployment of the mentioned above, a multi-stage CI/CD pipeline is deployed and configured in Azure DevOps.

The command sidra deploy source will be in charge of everything. It will:

  • Create the Sidra solution and push all the code to the Azure DevOps default repository.
  • Create the variable group for the environment which is being deployed in the Azure DevOps library, with the definition of all the parameters and default values for some of them.
  • Create the multi-stage pipeline.

Execute the command, filling the needed information.

sidra deploy source

Alternatively, if the installation is going to use to the preview version of Sidra, the used command must be:

sidra deploy source --enablePrerelease

Custom naming convention

Before proceeding, if the organization uses a custom naming convention for Azure resources, it is needed to create two scripts and copy them to the Functions folder of the deployment project in the created solution. If there's no custom naming convention this step can be skipped.

These scripts are:

  • GetCustomResourceName.ps1, which defines the naming that the deployed resources will have.
  • GetCustomResourceTags.ps1, which defines the tags that the deployed resources will have.

They need to receive the same parameters as the GetDefaultResourceName.ps1 and GetDefaultResourceTags.ps1 scripts.

These files have to be placed on the following path on the code repository .\src\PlainConcepts.Sidra.Deployment\Scripts\Functions

To make these changes effective, these files have to be included in the generated binaries. To do so we can do it either with Visual Studio or editing the project file. After editing, commit and push the changes into the remote git repository.

Visual Studio

Open the Sidra solution file on the root folder PlainConcepts.Sidra.sln and follow these steps

  • On the Solution Explorer window, unfold the PlainConcepts.Sidra.Deployment project under src\Deployment folder
  • Select GetCustomResourceName.ps1 and GetCustomResourceTags.ps1 files under Scripts\Functions
  • Right click and select Properties
  • On the Properties panel, on the property Copy to Output Directory select Copy always
  • Save the project changes

Set copy always on Visual Studio

Project file edit

Open deployment project file .\src\PlainConcepts.Sidra.Deployment\PlainConcepts.Sidra.Deployment.csproj with the text editor of your choice and add the following xml tags and values to the file.

1
2
3
4
5
6
7
8
<ItemGroup>
  <None Update="Scripts\Functions\GetCustomResourceName.ps1">
    <CopyToOutputDirectory>Always</CopyToOutputDirectory>
  </None>
  <None Update="Scripts\Functions\GetCustomResourceTags.ps1">
    <CopyToOutputDirectory>Always</CopyToOutputDirectory>
  </None>
</ItemGroup>

Settings creation

In the previous step, the command created several variable groups with the definition of the parameters that are used to deploy the platform.

This last command will fill the parameters to finally deploy the solution. Many of them contain default values that can be overwritten.

You can see all the configurable parameters that can be passed to the command with sidra deploy settings --help.

The bare minimum are the indicated by the following command template. Fill the necessary information and execute it.

sidra deploy configure --deploymentOwnerName "<DEPLOYMENT_OWNER_NAME>" --deploymentOwnerEmail "<DEPLOYMENT_OWNER_EMAIL>" --infrastructureAlertEmail "<INFRASTRUCTURE_ALERT_EMAIL>" --companyName "<COMPANY_NAME>" --sidraAdminSubject "<SIDRA_ADMIN_SUBJECT>" --subscriptionId "<SUBSCRIPTION_ID>" --coreKeyVaultName "<CORE_KEY_VAULT_NAME>" --developmentTeam "<DEVELOPMENT_TEAM>" --resourceGroupName "<RESOURCE_GROUP_NAME>" --defaultDSUResourceGroupName "<DEFAULT_DSU_RESOURCE_GROUP_NAME>" --resourceNamePrefix "<RESOURCE_NAME_PREFIX>" --resourceNameMiddle "<RESOURCE_NAME_MIDDLE>" --installationSize "<INSTALLATION_SIZE>"

Deploy!

Now, everything is ready to deploy all the resources.

On Azure DevOps go to the Pipelines blade, and select the multi-stage pipeline "SIDRA Dev CD". This pipeline will create the build artifacts and deploy the infrastructure and binaries on Azure.

Pipelines Blade

After running the pipeline Sidra will be ready to be used!

Summing up

The process of setting up Sidra with the CLI is as simple as 4 commands. Each of these commands will be performed for each of the environments.

1
2
3
4
5
6
For each deployed environment
do
    sidra profile create
    sidra install aadapplications
    sidra deploy source
    sidra deploy settings (this can be used all the times needed to overwrite settings in Azure DevOps)

Update a Sidra installation

Apart from creating a Sidra project out of the box, the CLI allows to update the version of Sidra easily to the latest available, applying all needed updates to the Nuget packages, source code, DevOps settings and other possible tasks that are part of the newer versions.

The update command is available starting in the 2020.R3 version, and will work from that point forward, so it is not possible to update previous versions using this command.

Depending on the current version installed, a series of tasks will be applied.

For example, let's say that the current installed version is 2020.R3, and after that the versions 2020.R4 and 2021.R1 were release.

If we upgrade, tasks will be applied for 2020.R4 and then for 2021.R1.

However, if the current version was 2020.R4, only tasks for 2021.R1 would be applied.

To perform the update, execute the command sidra deploy update to update to the latest stable version.

Alternatively, if the installation is going to be updated to the preview version of Sidra, the used command must be:

sidra deploy update --enablePrerelease