How to create a new Sidra solution using Visual Studio templates

The solutions based on Sidra Data Platform are developed using Microsoft technologies, so they use Visual Studio as the IDE -Integrated Development Environment- to implement the solution. Visual Studio uses an object called the Visual Studio (VS) solution as a container to organize the related projects that implement the solution.

Sidra provides a set of templates that can be used to create VS solutions based on Sidra. To use the templates it is required to have:

  • Visual Studio 2019.
  • .NET Core SDK 3.0 (at the moment of writing the documentation, this is the current version used)

These templates are going to create a solution in Visual Studio 2019 with all the required components. This solution will be compiled using a build definition and then deployed to Azure using a release pipeline. By default, the templates generates:

  • A solution in Visual Studio 2019 which can be compiled without errors.
  • A build definition in YAML to compile this solution.
  • A release definition in JSON to deploy the solution into Azure. The JSON can be imported into Azure pipelines, but it is required to populate some values before saving it.

The solution is open to be extended by adding new projects or custom code. The purpose of this document is to show how the templates can be used. Further information related to the deployment and the Sidra's components can be found in the corresponding documentation.

Installing the templates

The templates are located within the PlainConcepts.Sidra.DotNetCore Git repository and released as NuGet packages.

The following templates are available:

Name NuGet Package Id Description Short Name
Core PlainConcepts.Sidra.DotNetCore.CoreTemplate This is the basic Core template. It includes the webjobs (RetryImport, DataFactoryManager, DatabaseBuider), custom activities (CreateTable, GenerateTransferQuery, ImportAsset), the API and the deployment project sidra
Consumer App PlainConcepts.Sidra.DotNetCore.ConsumerAppTemplate This is the basic template for a Client app. It includes the webjobs (Sync, DataFactoryManager, DatabaseBuilder), Extract custom activity and the deployment project. sidra-app
Consumer App with PowerBI PlainConcepts.Sidra.DotNetCore.ConsumerAppPowerBITemplate This template is used to build a Client app with the artifacts required to support PowerBI operations (such as deploying DataFlows, Reports and processing them) plus the components provided by the base Client app template. sidra-app-powerbi
Custom Activity PlainConcepts.Sidra.DotNetCore.CustomActivityTemplate This template is used to create a new Azure Data Factory custom activity. sidra-ca

The short name indicates the name which the template will be picked once it is installed. They can be installed in two different ways by using the dotnet commands: (1) from a file system directory or (2) from a NuGet package.

Installing from a local directory

Assuming the Sidra git repository has been cloned somewhere in the local file system and the desired branch checked out (e.g: Release-1.4), a template can be installed by running the following command:

1
dotnet new -i [Template Config Directory Path]

where:

  • [Template Config Directory Path] should be the full path to the .template.config directory for the desired template (e.g: C:\..\PlainConcepts.SIDRA\templates\ConsumerApp\Content).

The install command will look for the folder .template.config in that path and expect a template.json that contains the short name used to identify the newly installed template (e.g: sidra-app)

Installing from a local NuGet package

Assuming the NuGet package (.nupkg file) for the template is available in the local file system, a template can be installed by running the following command:

1
dotnet new -i [NuGet Package File Path]

where:

  • [NuGet Package File Path] should be the file path to the NuGet package that contains the template (e.g: C:\...\PlainConcepts.Sidra.DotNetCore.CoreTemplate.1.5.0-preview.80.nupkg).

Installing from a NuGet feed

Assuming that the NuGet feed contains the NuGet package with the template, it can be installed by running the following command:

1
dotnet new -i [NuGet Package Id] --nuget-source [NuGet Feed Url]

where:

  • [NuGet Package Id] should be the NuGet Package Id for a specific template (e.g: PlainConcepts.Sidra.DotNetCore.CoreTemplate)
  • [NuGet Feed Url] should be the URL of the NuGet feed (e.g. https://www.myget.org/<<redacted>>/index.json)

IMPORTANT

Note that dotnet will try to install the latest stable version available.

If the template to be installed is a different version or a pre-release version, you need to specify the version in the format <package-name>::<package-version> (e.g: dotnet new -i PlainConcepts.Sidra.DotNetCore.Core::1.5.0-pre)

.NET Core SDK version

The installation of a new VS template will be done for a specific .NET Core SDK version. The template will be installed in the .NET Core SDK version configured for the path where the installation command is executed which can be checked with this command:

1
dotnet --version

For example, the following set of commands shows how the installation is done for the 2.2.401 version of the SDK but it is not available in the 3.1.101:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
C:\>cd usingOldSdkVersion

C:\usingOldSdkVersion>dotnet --version
2.2.401

C:\usingOldSdkVersion>dotnet new -i C:\repo\PlainConcepts.SIDRA\templates\Core\Content

C:\usingOldSdkVersion>dotnet new sidra -l
Usage: new [options]

Options:
  -h, --help          Displays help for this command.
  -l, --list          Lists templates containing the specified name. If no name is specified, lists all templates.
  -n, --name          The name for the output being created. If no name is specified, the name of the current directory is used.
  -o, --output        Location to place the generated output.
  -i, --install       Installs a source or a template pack.
  -u, --uninstall     Uninstalls a source or a template pack.
  --nuget-source      Specifies a NuGet source to use during install.
  --type              Filters templates based on available types. Predefined values are "project", "item" or "other".
  --dry-run           Displays a summary of what would happen if the given command line were run if it would result in a template creation.
  --force             Forces content to be generated even if it would change existing files.
  -lang, --language   Filters templates based on language and specifies the language of the template to create.
  --update-check      Check the currently installed template packs for updates.
  --update-apply      Check the currently installed template packs for update, and install the updates.


Templates                                  Short Name      Language      Tags
------------------------------------------------------------------------------
Plain Concepts Sidra Core Application      sidra           [C#]          Sidra

C:\usingOldSdkVersion>cd ..

C:\>dotnet --version
3.1.101

C:\>dotnet new sidra -l
Usage: new [options]

Options:
  -h, --help          Displays help for this command.
  -l, --list          Lists templates containing the specified name. If no name is specified, lists all templates.
  -n, --name          The name for the output being created. If no name is specified, the name of the current directory is used.
  -o, --output        Location to place the generated output.
  -i, --install       Installs a source or a template pack.
  -u, --uninstall     Uninstalls a source or a template pack.
  --nuget-source      Specifies a NuGet source to use during install.
  --type              Filters templates based on available types. Predefined values are "project", "item" or "other".
  --dry-run           Displays a summary of what would happen if the given command line were run if it would result in a template creation.
  --force             Forces content to be generated even if it would change existing files.
  -lang, --language   Filters templates based on language and specifies the language of the template to create.
  --update-check      Check the currently installed template packs for updates.
  --update-apply      Check the currently installed template packs for update, and install the updates.


No templates matched the input template name: sidra.

More information about SDK versions can be found in Microsoft Docs .

Creating a project based on templates

Once the templates are installed, a project can be created based on them. To list the Sidra templates the following command could be executed:

1
dotnet new sidra -l

The output of this command (supposing that two templates have been installed) will be:

1
2
3
4
Templates                                                   Short Name             Language      Tags
------------------------------------------------------------------------------------------------------
Plain Concepts Sidra Consumer Application                   sidra-app              [C#]         Sidra
Plain Concepts Sidra Core Application                       sidra                  [C#]         Sidra

The short name will be used to create the templates. A .NET Core template could include parameters to populate the solution created. There is a set of default parameters available that can be listed using the command:

1
dotnet new -h

Additionally every template can define its own list of additional parameters which can be showed using the following command:

1
dotnet new <<template short name>> -h

In the case of Sidra Core template this is the list of parameters:

Parameter name Description
-n | --name [Optional] Name of the solution to create. If not present, it will use the name of the folder.
-lm | --log-minimumlevel [Required] The minimun log level that appear in the configuration. This value can be changed later in the appsettings.json files for each project into the solution.
-D | --DevOpsARMServiceConnection [Optional] This is the ID of the service connection used to deploy the solution into Azure using the Release pipeline. Once the release is created, this value can be changed in every step used with Azure.

For example, to create a solution based on one of the templates installed before, the command will be:

1
dotnet new sidra -n Sidra.Tutorial.Core --log-minimumlevel "Warning" 

And if the command have been succesfully executed, the result should be:

1
The template "Plain Concepts Sidra Core Application" was created successfully

The solution has been created in the same directory where the command has been executed and it constains the following:

Item Description
src The folder where all the projects are inside the solution. All the projects contain some type of startup class (Program.cs as an example) which is going to invoke the code appearing in the Sidra packages. No more code is on the solution, as all belongs to Sidra.
[CD] Dev 'Solution name' deployment.json This is the release definition in JSON format. This file should be imported into the Release pipelines, as it could be used to deploy the current solution into Azure. During the import process it is required to perform some amendments to ensure that the build is used and other settings.
[CI] Dev 'Solution name'.yml This is the build definition in YAML format. This file can be used as-is to create a valid build definition based in YAML in Azure pipelines to compile the solution and generate the artifacts required to the release steps.
*.props These are files required by .NET Core, as they include some references of the packages used to build the solution. It ensures that the latest version of Sidra packages are used every time.
*.sln This is the solution file.

If the solution file is opened with Visual Studio, the content showed in the Solution Explorer will be:

tutorial-solution-explorer

At this stage, the solution can be built and includes all the artifacts required.

NOTE:

Possible issue with run.cmd files. Sometimes the run.cmd files are installed using UTF-8-BOM encoding instead of UTF-8, which could generate issues during deployment phase.

After creating the solution please ensure -and change if required- the encoding of these files to UTF-8

Deployment and environment configuration

Every template generates a deployment project used to create the infrastructure and deploy the solution. It can be found in the Visual Studio solution and it is named <<Solution name>>.Deployment.

The deployment project contains:

  • Several PowerShell scripts to create and configure the infrastructure.
  • Azure Resource Manager (ARM) templates used to define the infrastructure and programmatically create it in Azure.
  • Artifacts needed for the configuration and population of the infrastructure (e.g. Databricks notebooks).

The deployment is orchestrated by a PowerShell script (CoreDeploy.ps1 for a Core solution and ClientDeploy.ps1 for a Client app solution) that can be found in the 'Scripts' folder of the deployment project. To increase reusability and reduce the risk of differences between the several environments of the release process, the same orchestrator script is used for all the environments (development, preproduction, production...).

In order to allow to configure specific parameters for each environment, a environment data file is used. The environment data files are basically a dictionary of values, they will have the following naming convention <Environment Name>Data.psd1. For example, if the environment is named "Dev", the file name will be DevData.psd1. All the environment data files must be stored in the 'Script' folder. There are samples of enviroment data files for Core and Client app solutions in the 'Scripts\Documentation' folder.

The orchestrator script will know which environment data file must use by the environment parameter of the script. For example if the orchestrator script is executed like this:

1
.\CoreDeploy.ps1 -environment "Test"

it will look for a TestData.psd1 file to get the specific environment parameters. It is expected to have an environment data file per environment, for example in case of having four environments (Dev, Test, Pre, Pro) it will be four environment data files: DevData.psd1, TestData.psd1, PreData.psd1, ProData.psd1.

SECURITY NOTE:

For security reasons, no password, client secret or any other sensitive information should be stored in the environment data files. For the same reason, no connection string should be stored in any of the appsettings.json files.

CI/CD Integration

The solution will be deployed in a Continuous Integration (CI) / Continuous Delivery (CD) pipeline process. The VS template generates a build definition in YAML format as well as a release definition in JSON format that can be used in Azure DevOps to configure the CI/CD.

Source code repository

The first step will be to set up the source code repository and push the Visual Studio solution to it. Sidra is specifically prepared to use Azure DevOps which provides integration with several repositories, but the most commonly used is the private Git repository.

Once the repository is created, it is necessary to implement the branching strategy. The build and release definitions provided by the VS template are configured to use two branches, /master and /dev, and it expects them to be created in the repository.

Continuous integration

This step is quite straightforward since Azure DevOps allows to configure a build using a YAML file, so the only configuration required is to select the YALM file that contains the build definition from the repository (e.g. [CI] Dev Sidra.Tutorial.Core.yml) when the build pipeline is created in Azure DevOps.

This way everytime a new code is pushed to the repository, the build will be launched and if a change is required in the build, it can be achieved by modifying the YAML and pushing those changes to the repository.

Continuous delivery

The build definition can be used as is, but the release definition needs some additional configuration before use it. The release definition is provided in JSON format and can be imported into an Azure DevOps release pipeline, but unfortunately once it is imported it is not synchronized. So, in order to keep the release definition updated, any changes made to the release pipeline in Azure DevOps must be exported to a JSON format and updated in the repository.

Before importing for the first time the release definition, the following parameters must be checked:

  • The service connection used in the release is properly configured. It is the connection between Azure DevOps and the Azure subscription where the resources will be deployed.
  • All Azure DevOps variables used in the release are created and contain the appropriate values in Azure DevOps. The variables can be created in the pipeline or be grouped into an Azure library -also known as variable groups-.
  • The release pipeline is triggered by the build pipeline created in the CI integration.
  • The Azure DevOps extension called 'Replace Tokens' must be installed. This extension is used to replace placeholders that are present in the appsettings.json file by the appropriate values, so those possible sensitive values are not stored in the appsettings.json file in the repository but in a secure place. Here it is an example of the 'Replace token' task in the release definition:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
{
    "environment": {},
    "taskId": "a8515ec8-7254-4ffd-912c-86772e2b5962",
    "version": "0.*",
    "name": "Replace tokens in **/appsettings.json",
    "alwaysRun": false,
    "continueOnError": false,
    "condition": "succeeded()",
    "enabled": true,
    "timeoutInMinutes": 0,
    "inputs": {
        "rootDirectory": "staging/Sidra.Tutorial.Core.DataFactory.CustomActivity.GenerateTransferQuery/unzip/",
        "targetFiles": "**/appsettings.json",
        "encoding": "auto",
        "writeBOM": "true",
        "verbosity": "normal",
        "actionOnMissing": "fail",
        "keepToken": "false",
        "tokenPrefix": "__",
        "tokenSuffix": "__",
        "emptyValue": "(empty)",
        "escapeType": "none",
        "escapeChar": "",
        "charsToEscape": ""
    }
},

There are several extensions in the Azure Marketplace that provides the same functionality, so those release tasks can be changed to use any of them -e.g. an alternative is Colin's ALM Corner Build & Release Tools-.

So the complete workflow for the CI/CD would be:

  1. The developer will push source code changes to the Git repository.
  2. The build pipeline will be automatically triggered by the changes in the repository.
  3. The release pipeline will be triggered by the finalization of the build pipeline. The release pipeline will use the artifacts generated by the build to deploy the infrastructure and the metadata for the solution.

Customization

The purpose of the template is to provide as-is the base project to use in a solution -a Core solution or a Client app solution-. But it could be more things required for the solution that are not covered in the template. If it is the case, the solution created by the template is open to extension and new projects or files can be added. Then, updating the definitions for the build and releases the changes can be deployed into the environments.

Adding new projects to the solution

The solution is a set of .NET Core projects, all of them using Sidra packages that contains the code which is actually implementing the features.

In case of a new project is required -e.g. a new custom activity, a website...- it can be added to the solution. New projects must be added inside of src/ folder -following the convention already used- and if any test projects are also required then they must be created inside a test/ folder.

The build definition should be modified to compile the new project, execute the tests and generate the artifacts to be used later. Those changes can be done directly on the build definition YAML file and they will be automatically updated in the Azure build pipeline.

The release definition should be modified with the required tasks to deploy the new project. It must be done directly on the Azure release pipeline, but once it is working, the modified pipeline must be exported to JSON and the release definition updated in the repository.

Infrastructure deployment and setup

The deployment project includes all the scripts required to deploy the base solution, but if new requirements arise then it could be not enough:

  • New ARM templates could be required to deploy new resources. In that case a CustomTemplates folder must be created in the root of the deployment project to store them.
  • New artifacts could be necessary to deploy. In that case an Artifacts folder must be created and used to store the new content to deploy.
  • New scripts must be used to deploy the new ARM templates and new artifacts. The provided scripts should not be modified since they are included in a NuGet package and so they will be overwritten if a new version of the NuGet package is released and any change made to them will be lost.
  • A new orchestration script (with the same approach that CoreDeploy.ps1/ ClientDeploy.ps1) could be necessary. Using the provided scripts is recommended but not mandatory.

To ensure that the changes are included in the deployment project, the new files added should have the following properties:

  • Build action selected as "None"
  • Copy to output directories selected as "Copy Always"

NOTE:

All the changes in the infrastructure must be performed by the scripts. No manual changes should be performed outside of the deployment project; otherwise those change will be reverted on the next deployment.

The release definition should be updated with new tasks necessary to deploy the changes. This task must be done directly on the Azure release pipeline but once the modified pipeline is working, it must be exported to JSON and the release definition updated in the repository.

For example, if a new PowerShell script has been created to use some ARM templates for creating new resources then an Azure PowerShell task must be added to the release pipeline to execute that PowerShell.

Summary

This document provides information to be able to:

  • Understand the pre-requisites to use Sidra templates.
  • Create a solution based on the Visual Studio templates.
  • Understand how the Build and Release templates are provided.
  • Customize the solution (adding projects and updating the build and release definitions).