Setting up a Sidra installation using the Sidra CLI tool

The following guide covers how to install a whole Sidra environment from scratch using the Sidra CLI tool.

Sidra CLI tool has been designed to ease the process of installing and updating Sidra Core.

This document details the different steps that must be taken with the CLI tool to deploy a new and totally functional Sidra Core platform environment.

Pre-requirements

To be able to setup the complete Sidra Core environment the following Azure resources must be owned:

  • An Azure subscription
  • An Azure DevOps account

Additionally, Sidra CLI includes some commands in order to check for pre-requisites for launching and executing the different installation commmands. These are detailed in the CLI overview section.

General Sidra Core installation process overview

The process of seting a working development environment for Sidra Core traditionally involves the following steps:

  • Creation of AAD applications on Azure on each environment (e.g. development, test and production), with the needed permissions to allow the Sidra platform to access the required resources.
  • Setup of Azure DevOps:
    • Creation of the Sidra solution source code repository.
    • Creation of the CI/CD pipelines for the code build and release steps.
    • Setup of Azure DevOps variable groups for the CI/CD pipelines. Variable groups are a collection of variables which reside in the releases.

In the following sections we are going to see how these steps are achieved and drastically simplified with the CLI tool.

CLI overview

The CLI tool is a solution that needs to be compiled in the local environment from where it will be executed.

To open it, just go to the directory where the executable is located, open a new CMD window, type sidra and press Enter.

When the CLI is opened, it displays the available commands that can be used:

Sidra CLI welcome screen

All commands contain a description. Commmands can be executed with sidra <COMMAND> --help to ask for help on the options available and needed parameters.

To request help for any command, just type sidra <COMMAND> --help

For example, to know what parameters are needed to be passed to the sidra install aadapplications command, type sidra install aadapplications --help and the information will be displayed:

Command help

Sidra CLI tool requires to have some pre-requisites setup to run some commands.

There are some specific command options in the sidra CLI tool that allow to setup and validate prerequisites needed for executing commands.

On the sidra CLI tool, in the appropriate path where the executable is, type the following command to verify the pre-requisites for executing Sidra CLI:

sidra prerequisites verify

The following command will launch the pre-requisites installation:

sidra prerequisites setup

Seting up Sidra Core with the Sidra CLI installation tool

Following are the different steps involved in setting up a Sidra Core installation with the tool:

Step 1: Creating the Azure DevOps profile

The process of installing Sidra through the CLI tool requires to pass over the same information at different steps. To avoid duplicating typing efforts, there is a specific command that creates a profile with the common shared information for a series of steps. This is the profile command.

The information on the parameters that need to be passed to this command is provided by typing sidra profile create --help.

It is recommended create a new profile before starting an installation process. Execute the command, filling in the needed information:

sidra profile create --devOpsProjectUrl "<DEV_OPS_PROJECT_URL>" --devOpsPAT "<DEV_OPS_PAT>" --devOpsProject "<DEV_OPS_PROJECT>" --environment "<ENVIRONMENT>"

Step 2: Installing the Active Directory applications

In order to work properly, Sidra needs to create several applications in the Azure Active Directory to access different resources, like access to the Data Factory APIs or Identity Server federation, among others.

The applications are created per environment, so they are not shared between the different created environments.

The command to create the AAD applications will create the following:

  • The AAD applications.
  • An Azure KeyVault resource, which will hold the sensitive information about the created applications.
  • The Core Resource group, for the mentioned KeyVault.

Execute the command, filling the needed information. Remember that you can use the CLI help to know what each parameter is for.

sidra install aadapplications --tenant "<TENANT>" --subscription "<SUBSCRIPTION>" --location "<LOCATION>" --keyvaultName "<KEYVAULT_NAME>" --identityServerUrl "<IDENTITY_SERVER_URL>" --customerClientId "<CUSTOMER_CLIENT_ID>" --customerClientSecret "<CUSTOMER_CLIENT_SECRET>" --resourcegroup "<RESOURCEGROUP>"

The detail of the created applications can be found on the Managed Applications blade on Azure portal. The name of the applications will follow the pattern <DEV_OPS_PROJECT>.<ENVIRONMENT>.<APP_SCOPE>.

Currently 3 applications are created by this command:

  • <Project>.<Env>.IdentityServerAdAccess for Identity Server federation.
  • <Project>.<Env>.Manager to manage operations on Azure resources, primarily needed for the Data Factory services.
  • <Project>.<Env>.VSTS to manage Azure DevOps and Azure resource deployments.

Step 3: Deploying the source code

The platform is deployed using different PowerShell scripts and ARM templates that contain the definition of all the Azure resources used by Sidra and the orchestration to deploy them. Apart from this, there are Webjobs, an API and custom activities used in DataFactory.

All of them are provided as Nuget packages. These Nuget packages are then installed in the Sidra solution.

Finally, to perform the deployment of all the above-mentioned resources, a multi-stage CI/CD pipeline is deployed and configured in Azure DevOps.

The command sidra deploy source will take care of all these actions. It will:

  • Create the Sidra solution and push all the code to the Azure DevOps default repository.
  • Create the variable group for the environment which is being deployed in the Azure DevOps library. The environment variable groups include the definition of all the parameters and default values for some of them.
  • Create the Azure DevOps multi-stage pipeline.

Execute the command to deploy the source code by filling in the needed information:

sidra deploy source

Alternatively, if the installation is going to use to the preview version of Sidra, the used command must contain the --enablePrerelease flag, as follows:

sidra deploy source --enablePrerelease

Step 4: Creating a custom naming convention (optional)

Before proceeding, if the organization uses a custom naming convention for Azure resources, two scripts need to be created to specifiy the naming conventions to use:

  • GetCustomResourceName.ps1, which defines the naming that the deployed resources will have.
  • GetCustomResourceTags.ps1, which defines the tags that the deployed resources will have.

They need to receive the same parameters as the GetDefaultResourceName.ps1 and GetDefaultResourceTags.ps1 scripts.

These files have to be placed on the following path on the code repository:

.\src\PlainConcepts.Sidra.Deployment\Scripts\Functions

To make these changes effective, these files must be included in the generated binaries. This can be done by two alternative methods: with Visual Studio or by editing the project file. Both methods are detailed below.

After editing, commit and push the changes into the remote git repository.

OPTION A: Include scripts from Visual Studio properties panel

Open the Sidra solution file on the root folder PlainConcepts.Sidra.sln and follow the following steps:

  • On the Solution Explorer window, unfold the PlainConcepts.Sidra.Deployment project under src\Deployment folder.
  • Select GetCustomResourceName.ps1 and GetCustomResourceTags.ps1 files under Scripts\Functions.
  • Right click and select Properties.
  • On the Properties panel, on the property Copy to Output Directory select Copy always.
  • Save the project changes.

Set copy always on Visual Studio

OPTION B: Edit the Project file Project

Open deployment project file .\src\PlainConcepts.Sidra.Deployment\PlainConcepts.Sidra.Deployment.csproj with the text editor of your choice and add the following xml tags and values to the file:

1
2
3
4
5
6
7
8
<ItemGroup>
  <None Update="Scripts\Functions\GetCustomResourceName.ps1">
    <CopyToOutputDirectory>Always</CopyToOutputDirectory>
  </None>
  <None Update="Scripts\Functions\GetCustomResourceTags.ps1">
    <CopyToOutputDirectory>Always</CopyToOutputDirectory>
  </None>
</ItemGroup>

Step 5: Creating the final settings for the the deployment

In Step 3, the command created several variable groups with the definition of the parameters that are used to deploy the platform.

The command in this step will fill the parameters to finally deploy the solution. Many of them contain default values that can be overwritten later.

You can see all the configurable parameters that can be passed to the command by typing sidra deploy settings --help.

The minimum set of mandatory parameters are the indicated by the following command template. Fill in the necessary information and execute it:

sidra deploy configure --deploymentOwnerName "<DEPLOYMENT_OWNER_NAME>" --deploymentOwnerEmail "<DEPLOYMENT_OWNER_EMAIL>" --infrastructureAlertEmail "<INFRASTRUCTURE_ALERT_EMAIL>" --companyName "<COMPANY_NAME>" --sidraAdminSubject "<SIDRA_ADMIN_SUBJECT>" --subscriptionId "<SUBSCRIPTION_ID>" --coreKeyVaultName "<CORE_KEY_VAULT_NAME>" --developmentTeam "<DEVELOPMENT_TEAM>" --resourceGroupName "<RESOURCE_GROUP_NAME>" --defaultDSUResourceGroupName "<DEFAULT_DSU_RESOURCE_GROUP_NAME>" --resourceNamePrefix "<RESOURCE_NAME_PREFIX>" --resourceNameMiddle "<RESOURCE_NAME_MIDDLE>" --installationSize "<INSTALLATION_SIZE>"

Step 6: Deploying Sidra Core!

At this step, all the parameters should be ready to deploy all the resources.

On Azure DevOps go to the Pipelines blade, and select the multi-stage pipeline "SIDRA Dev CD".

This pipeline will create the build artifacts and deploy the infrastructure and binaries on Azure.

Pipelines Blade

After running the pipeline Sidra will be ready to be used!

Summing up

The process of setting up Sidra with the CLI tool is as simple as typing 4 commands. Each of these commands will need to be performed for each of the environments.

1
2
3
4
5
6
For each deployed environment
do
    sidra profile create
    sidra install aadapplications
    sidra deploy source
    sidra deploy configure (this can be used all the times when itis required to overwrite settings in Azure DevOps)

Update a Sidra installation with the Sidra CLI tool

Apart from creating a Sidra project for the first time, the CLI allows to update the version of Sidra easily to the latest version availabl. In order to execute an update, the Sidra CLI tool is prepared to apply, just with one command, all the needed updates. This extends to: upgrading the Nuget packages, updating source code, DevOps settings and other possible tasks that are part of the newer versions.

Important notice: The update command is available starting in the 2020.R3 version, and will work from that point forward, so it is not possible to update previous versions using this command.

Depending on the current version installed, different sets of of tasks will be applied.

Let's consider some example scenarios:

If the installed version is 2020.R3, and after that the versions 2020.R4 and 2021.R1 were released:

Then, when upgrading with the CLI tool, tasks would be applied for 2020.R4 and then for 2021.R1.

However, if the current version was 2020.R4, only tasks for 2021.R1 would be applied.

To perform the update, execute the command sidra deploy update to update to the latest stable version.

Alternatively, if the installation is going to be updated to the preview version of Sidra, use the following command:

sidra deploy update --enablePrerelease