Sidra Client App creation tutorial

1. Introduction

This tutorial provides information about how to create, deploy and setup a client app from scratch. The steps to follow are: - Get all the required values (covered in Prerrequisites section) - Create the client solution - Setup the client solution to be deployed and integrated with its relative Core and DSU - Setup pipelines to load data from DSU

2. Prerequisites

Values that are required for following this tutorial

When checking the code, please replace the values {} with the proper variables defined below.

2.1. Installation

  • FeedId = Feed id for resources. Ask for this one.
  • Solution name: Name of the solution, example: Sidra.ExampleApp
  • App name: Internal application name (note how there are only alphanumerical characters ) example: PMSidraExampleApp
  • Service connection name: Name of the Service Connection in DevOps
  • DevOps Organization: URL for DevOps, example: https://myorg.visualstudio.com/
  • DevOps Project: Name of the DevOps project. Example: Data Platform - Plain Concepts

2.2 Setup

2.2.1 Basics for deployment

  • SubscriptionId: Subscription Id where this application will be deployed
  • Tenant: Name of the tenant
  • Resource Group Name: Name of the resource group (it will be created in Step 4)
  • Location: Location of the resource group created in Step 4
  • Email for security alerts: Some address to receive alerts information
  • Environment description: Some description about this environment.
  • Admin user: User of the database created in the client application.
  • ManagerServicePrincialId: Service Principal Id of the Manager AAD application
  • ManagerApplicationId: ID of the Manager AAD application
  • DevOpsServicePrincipalId: Service Principal Id of the VSTS/DevOps AAd application

2.2.2 Naming convention

  • ProjectName: Name of the project. Example: ExampleApp
  • ProductName: Name of the core side. Example: Sidra
  • DeploymentName: Name of the environment. Example: dev
  • DevelopmentTeam: Plain Concepts
  • PublicProjectName: Name of the solution: Sidra.ExampleApp (same as 2.1)
  • ProjectAssemblyName: Name of the assembly; by convention the same as solution

2.2.3 Core & DSU integration

  • APIClientResourceId = "plainconcepts.sidra.api"
  • ProductCoreName = Same value as ProductName in Core application.
  • ProjectCoreName = Same value as ProjectName in Core application.
  • ApplicationName = Same value as "ProjectName" in 2.2.2
  • CoreResourceGroupName = Name of the resource group where Core is deployed.
  • DataLakeResourceGroupName = Name of the resource where DSU is deployed.
  • DataLakeLocation = Name of the location where the DSU is deployed
  • IdentityServerEndpoint = Endpoint of Identity Server instance. Example: "https://sidra-core-identityserver.azurewebsites.net"
  • IdentityServerScope = "plainconcepts.sidra.api"
  • SidraAPIClientID = "coreClient"
  • CoreClientId="coreClient"

2.2.4 Specifics for the client app

  • CreateClientDatabase = $true
  • ClientDatabaseName = "Client"
  • ClientDatabaseServiceObjective = "S0"
  • ClientDatabaseSizeInGB = 250
  • InstallationSize = "S"

2.2.5 Specifics for the naming/tags customization

  • ShortName = "Sidra"
  • CreatedDateTag = "20/07/22"
  • EnvironmentTag = "Dev"
  • ProductOwnerTag = "..."
  • ServiceOwnerTag = "..."
  • ServiceReferenceTag = "TBC"
  • CostCenterTag = "TBC"
  • etc...

3. Create repository

  • Create a Git repository in DevOps with VisualStudio default settings.
  • Clone the repository into your local machine.

4. Create resoruce group in Azure

Create a resource group in the same Azure subscription where the Core and DSU are. The resource group should follow any convention name that it is required to apply. In addition, check that the resource group has permissions with the required AAD applications that Sidra needs. In theory they should be as they are inherit from the current subscription: - VSTS application: The AAD application which ends on .VSTS as Owner/Contributor. - Manager application: The AAD application which ends .Manager as Contributor

6. Install the template

Uninstall a previous version of the template

1
dotnet new -u PlainConcepts.Sidra.DotNetCore.ClientAppTemplate

Install the template. In this case, ClientAppTemplate with 1.7.1 version. NOTE: In the example 1.7.1 is used, but other version could be used as well.

1
dotnet new -i PlainConcepts.Sidra.DotNetCore.ClientAppTemplate::1.7.1 --nuget-source "https://www.myget.org/F/plainconcepts-sidra/auth/{FeedId}/api/v3/index.json"

Go to the root of the folder where the repository has been cloned and execute the following command:

1
dotnet new sidra-app  --name "{Solution name}" --appName "{App name}" --serviceConnectionName "{Service connection name}" --force

7. Setup AzureDevOps library

Execute ConfigureDevOps from the installed solution for creating the required variables

1
.\ConfigureDevOps.ps1 -organization {DevOps Organization} -project "{DevOps Project}"

In this case there no variables to setup, as the only one that appear is non mandatory.

8. Setup client application

8.1 Using custom naming convention

If there is a custom naming/tag convention to apply, provide the files inside the *.Deployment\Functions folder.

Function for naming convention should have following name and parameters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
function GetResourceNameConvention
{
    Param(
        [Parameter(Mandatory=$true)]
        [string]$templatePath,
        [Parameter(Mandatory=$true)]
        [object]$deploymentData,
        [Parameter(Mandatory=$false)]
        [object]$coreProductName, # Not used but the parameter must be kept for compatibility
        [Parameter(Mandatory=$false)]
        [object]$coreProjectName, # Not used but the parameter must be kept for compatibility
        [Parameter(Mandatory=$true)]
        [object]$resourceNamingData,
        [Parameter(Mandatory=$true)]
        [string]$environment,
        [Parameter(Mandatory=$false)]
        [string]$function
    )

    # Logic to calculate the name

    return $resourceName
}

It should return a string with the name of the resource.

8.2 Using custom tag convention

Same approach for the file to customize the tags:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
function GetResourceTagsConvention
{
    Param(
        [Parameter(Mandatory=$true)]
        [object]$deploymentData,
        [Parameter(Mandatory=$true)]
        [string]$description,
        [Parameter(Mandatory=$true)]
        [string]$environment,
        [Parameter(Mandatory=$false)]
        [string]$owner,
        [Parameter(Mandatory=$false)]
        [string]$deploymentName
    )

    $tags = @{
        "my custom tag" = $deploymentData.MyCustomTagValue
        "other tag" = ...
    }

    return $tags

It should return an object with the tags. Each tag could be calculated with the parameters received or extracted from deployment data (the *.psd1 file associated with the environment)

8.3 Creating an environment definition

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
@{
    SubscriptionId = "{SubscriptionId value from 2.2.1}"
    TenantId = "{TenandId value from 2.2.1}"
    Tenant = "{Tenant name value from 2.2.1}"
    ResourceGroupName = "{Resource group name; created on Step 2}"
    ResourceGroupLocation = "{Resoruce group location; created on Step 2}"
    EmailForSecurityAlerts = "{Some address for sending alerts regarding security from 2.2.1}"
    EnvironmentDescription = "{A description about this environment from 2.2.1}"
    AdminUser = "{Admin user for the Client database 2.2.1}"    

    # Manager Azure AD application
    ManagerServicePrincipalId = "{From 2.2.1}" 
    ManagerApplicationId = "{From 2.2.1}"

    # Azure DevOps (VSTS) Azure AD application
    DevOpsServicePrincipalId = "{From 2.2.1}"

    ProjectName = "{From 2.2.2}"
    ProductName = "{From 2.2.2}"
    DeploymentName = "{From 2.2.2}"
    DevelopmentTeam = "{From 2.2.2}"
    PublicProjectName = "{From 2.2.2}"
    ProjectAssemblyName = "{From 2.2.2}"
    NamingConventionFile = "{OPTATIVE: From 2.2.2}"
    TagConventionFile ="{OPTATIVE From 2.2.2}"

    # Client default settings
    APIClientResourceId = "plainconcepts.sidra.api" # RESOURCE_ID (of the application to which the client want access. Usually the Core API)
    ProductCoreName = "{From 2.2.3}"
    ProjectCoreName ="{From 2.2.3}"
    ApplicationName = "{From 2.2.3}"
    CoreResourceGroupName = "{From 2.2.3}"
    DataLakeResourceGroupName  = "{From 2.2.3}"
    DataLakeLocation = "{From 2.2.3}"
    CreateClientDatabase = $true
    ClientDatabaseName = "{From 2.2.4}"
    ClientDatabaseServiceObjective = "{From 2.2.4}"
    ClientDatabaseSizeInGB = {From 2.2.4}
    SidraAPIClientID ="{From 2.2.3}"
    CoreClientId="{From 2.2.3}"
    InstallationSize = "{From 2.2.4}"
    IdentityServerEndpoint = "{From 2.2.3}"
    IdentityServerScope = "{From 2.2.3}"

    # Client settings (not actually used by the script)
    DataLakeSuffix = "not used"
    ClusterName ="not used"
    IdentityServerClientId = "dummy"
}

8.4 Creating permissions definition

Populate a file inside Scripts folder with the name of appsettings.json

1
2
3
4
5
6
7
8
9
[
  {
    "permissionsOn": [
      "{name of the Datalake}##{name of the provider}"
    ],
    "MetadataAccessLevel": "ReadWriteDelete",
    "DataAccessLevel": "ReadWrite"
  }
]

10. Deploy

Commit and push the changes to deploy the solution. The Azure Pipeline for Build and release process should be executed and the solution deployed without errors.

11. Loading data

11.1 SQL database

There is a SQL database project inside the solution. Create a staging schema and put inside the tables with the same structure defined in Core -same column types and order that the Entity is defined for its attributes-.

Convention: The staging table should be named with the following format: providerName_entityName

Create a store procedure inside staging schema with the following structure:

1
2
3
4
5
6
7
CREATE PROCEDURE [staging].[orchestrator]
    @pipelineRunId NVARCHAR(100)
AS
    SELECT 1
    -- TODO: POPULATE WITH LOGIC
RETURN 0
GO

11.2 Pipeline definition

Create a script that will be inside of "ClientContext" folder inside the DatabaseBuilder project. Ensure that this file is marked with "Copy Always" in the build properties.

The purpose of this script is to create a pipeline to handle the data movement between the DSU and this client application, dropping the content into the staging tables. As the convention was used, the pipeline will do this automatically.

NOTE: For further information about how pipelines works in Client apps, please refer to https://docs.sidra.dev/Sidra-Data-Platform/Sidra-Client-App/Pipelines/ and https://docs.sidra.dev/Sidra-Data-Platform/Tutorials/Add-new-pipeline-for-client/

Base the pipeline creation on this script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
DECLARE @extractPipelineTemplateId INT =
(
    SELECT [Id]
    FROM [Sidra].[PipelineTemplate]
    WHERE [ItemId] = '19C95A0E-3909-4299-AEE1-15604819E2B0'
)

DECLARE @extractPipelineItemId UNIQUEIDENTIFIER = '6860B34B-A53F-42EF-8FD7-D2CAF5690CAF'

-- Cleanup
DELETE FROM [Sidra].[EntityPipeline] WHERE [IdPipeline] IN (SELECT [Id] FROM [Sidra].[Pipeline] WHERE [ItemId] = @extractPipelineItemId)
DELETE FROM [Sidra].[Pipeline] WHERE [ItemId] = @extractPipelineItemId

-- Script content
INSERT INTO [Sidra].[Pipeline] ([ItemId],[Name],[ValidFrom],[ValidUntil],[IdTemplate],[LastUpdated],[LastDeployed],[IdDataFactory],[IsRemoved],[IdPipelineSyncBehaviour],[Parameters],[ExecutionParameters])
VALUES 
(@extractPipelineItemId, N'extract_example', GETUTCDATE(), NULL, @extractPipelineTemplateId, GETUTCDATE(), NULL, 1, 0, 1, N'{}', N'{
"storedProcedureName": "[staging].[orchestrator]"
}')

DECLARE @extractPipelineId INT =
(
    SELECT [Id]
    FROM [Sidra].[Pipeline]
    WHERE [ItemId] = @extractPipelineItemId
)

INSERT INTO [Sidra].[EntityPipeline]([IdEntity],[IdPipeline],[IsMandatory],[PipelineExecutionProperties])
VALUES
(91, @extractPipelineId, 1, NULL)
  • A new pipeline will be created with the ItemId '6860B34B-A53F-42EF-8FD7-D2CAF5690CAF'
  • The pipeline will be associated with the Entity with Id '91'. This should be modified with all the Ids that wants to be integrated.

12. Examples

12.1 GetResourceNameConvention

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
function GetResourceNameConvention
{
    Param(
        [Parameter(Mandatory=$true)]
        [string]$templatePath,
        [Parameter(Mandatory=$true)]
        [object]$deploymentData,
        [Parameter(Mandatory=$false)]
        [object]$coreProductName, # Not used but the parameter must be kept for compatibility
        [Parameter(Mandatory=$false)]
        [object]$coreProjectName, # Not used but the parameter must be kept for compatibility
        [Parameter(Mandatory=$true)]
        [object]$resourceNamingData,
        [Parameter(Mandatory=$true)]
        [string]$environment,
        [Parameter(Mandatory=$false)]
        [string]$function
    )

    $environmentSuffix = switch -Wildcard ($environment)
    {
        'Dev' { 'D' }
        'Test' { 'T' }
        'Prod' { 'P' }
    }

    $instance = switch -Wildcard ($function)
    {
        'Frontend' { '02' }
        'is' { '03' }
        'bs' { '04' }
        # For DSU, instance for Storage account will be 01 for DataLake (which is the default), 02 for Stage
        'Stage*' { '02' }
        'contractviewer' {'11'}
        'linux' { '12' } # For the linux app
         default { '10' } # For all resources in the environment, the instance will be 10
    }

    $application = switch -Wildcard ($function)
    {
        'is' { $deploymentData.ProjectName }
        'bs' { $deploymentData.ProjectName }
        'API' { $deploymentData.ProjectName }
        'linux' { $deploymentData.ProjectName }
        'contractviewer' { $deploymentData.ProjectName }
        'Frontend' { $deploymentData.ProjectName }
        $null { $deploymentData.ProjectName } # Values with null or empty are from Core
        '' { $deploymentData.ProjectName } # Values with null or empty are from Core
        default { $deploymentData.ShortName + $function } # All values from DSU have function populated
    }

    # For Core related resources, the default instance will be 01
    if ($coreProjectName -ne $null)
    {
        $instance = switch -Wildcard ($function)
        {
            'is' { '03' }
            'bs' { '04' }
             default { '01' } # For all resources in the environment, the instance will be 10
        }
        $application = $deploymentData.ProjectCoreName
    }

    $prefix = switch -Wildcard ($templatePath)
    {
        '*\batchaccount.json' { 'PT' } # Not included in naming convention
        '*\cognitiveservices.json' { 'CG' } # Not included in naming convention
        '*\*webapp*.json' { 'WEBAPP' }
        '*\datafactory*.json' { 'DF' }
        '*\datalake.json' { 'DL' }
        '*\elasticjobagent.json' { 'EJASQL' } # Not included in naming convention
        '*\elasticpool.json' { 'EPSQL' } # Not included in naming convention
        '*\keyvault.json' { 'KV' }
        '*\logicapp.json' { 'LA' }
        '*\oms.json' { 'LAW' } # Not included in naming convention
        '*\sqlserver.json' { 'DBSQL' }
        '*\storage.json' { 'sa' }
        '*\databricks.json' { 'ADB' }
        '*\insights.json' { 'INS' } # Not included in naming convention
        '*\serverfarm.json' { 'APPSERV' }
        '*\acr.json'{ 'ACR' } # Not included in naming convention
        '*\aci.json'{ 'ACI' } # Not included in naming convention
        '*\dns.json'{ 'DNS' } # Not included in naming convention
        '*\appfunction.json'{ 'FN' }
        '*\mlworkspace.json'{ 'MLW' } # Not included in naming convention
        '*\searchservices.json'{ 'SE' } # Not included in naming convention
        '*\signalr.json'{ 'SGr' } # Not included in naming convention
        '*\servicebus.json'{ 'SB' } # Not included in naming convention
        default { 'not found for $templatePath' }
    }

    switch -Wildcard ($templatePath)
    {
        '*batchaccount.json'
        {
            # Not supported by initial naming convention
            # It should be only in lowercase letters and numbers: https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/resource-name-rules#microsoftbatch
            $resourceName = $prefix + $environmentSuffix + $application + $instance
            $resourceName = $resourceName.ToLower()
        }
        '*acr.json'
        {
            # Not supported by initial naming convention
            # It should be only in alphanumeric: https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/resource-name-rules#microsoftcontainerregistry
            $resourceName = $prefix + $environmentSuffix + $application + $instance
            $resourceName = $resourceName.ToUpper()
        }
        '*storage.json' 
        { 
            $resourceName = $prefix + $environmentSuffix + $application + $instance
            $resourceName = $resourceName.ToLower()
        }
        '*searchservices.json' 
        { 
            # {Resource Initial}-{Environment}-{Application/System}{Number}
            # The naming convention says the following format: {Resource Initial}-{Environment}-{Application/System}{Number} -> SQLDB-P-SAP-BE01
            # But this is not supported, as Search Services provides following errur during an upgrade: is invalid: Service name must only contain lowercase letters, digits or dashes, cannot start or endwith or contain consecutive dashes and is limited to 60 characters.
            $resourceName = $prefix + '-' + $environmentSuffix + '-' + $application + $instance
            $resourceName = $resourceName.ToLower()
        }
        '*sqlserver.json' 
        { 
            # {Resource Initial}-{Environment}-{Application/System}{Number}
            # The naming convention says the following format: {Resource Initial}-{Environment}-{Application/System}{Number} -> SE-P-SAP-BE01
            # But this is not supported, as Azure SQL server should be only in lowercase letters,numbers and hyphens: https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/resource-name-rules#microsoftsql
            $resourceName = $prefix + '-' + $environmentSuffix + '-' + $application + $instance
            $resourceName = $resourceName.ToLower()
        }
        default 
        { 
            # {Resource Initial}-{Environment}-{Application/System}{Number}
            $resourceName = $prefix + '-' + $environmentSuffix + '-' + $application + $instance
            $resourceName = $resourceName.ToUpper()
        }
    }

    # Write-Host "Generated name: $resourceName"
    return $resourceName
}   

12.2 GetResourceTagsConvention

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
function GetResourceTagsConvention
{
    Param(
        [Parameter(Mandatory=$true)]
        [object]$deploymentData,
        [Parameter(Mandatory=$true)]
        [string]$description,
        [Parameter(Mandatory=$true)]
        [string]$environment,
        [Parameter(Mandatory=$false)]
        [string]$owner,
        [Parameter(Mandatory=$false)]
        [string]$deploymentName
    )

    $tags = @{
        Role = "Sidra Contract Analytics"
        Location = "UK South"
        "Data Classification"  = "Confidential"
        Description = "Sidra Client App installation"
        "Created By" = "Plain Concepts"
        "Created Date" = $deploymentData.CreatedDateTag
        # Resource group tags
        Environment = $deploymentData.EnvironmentTag
        "Product Owner" = $deploymentData.ProductOwnerTag
        "Service Owner" = $deploymentData.ServiceOwnerTag
        "Service Ref" = $deploymentData.ServiceReferenceTag # In the naming convention, it appear with Service Ref. but actually in the resource groups appears as Service Ref withtout the point.
        "Cost Center" = $deploymentData.CostCenterTag
    }

    return $tags
}

13. Delete client app

To delete the client application next steps should be applied: - Remove the resource group created in Step 4. - Cleanup the client registration from Core. Otherwise, next deployment will return a 409 error because an application with same name already exists

In Core Database:

1
2
3
4
5
DELETE FROM Auth.ApiKeys WHERE Name='myappname'
DELETE FROM Auth.RoleSubjects WHERE SubjectId IN (SELECT id from Auth.Subjects WHERE Name LIKE '%myappname%')
DELETE FROM Auth.Subjects WHERE Name like '%myappname%'
DELETE FROM Apps.App WHERE Name='myappname'
DELETE FROM Auth.Applications WHERE Name='myappname'

In IdentityServer database, there should be at least two clients to remove. Note that Clients could have related rows to ClientScope and ClientGrantTypes table:

1
2
3
DELETE FROM ClientScopes WHERE ClientId IN (...)
DELETE FROM ClientGrantTypes WHERE ClientId IN (...)
DELETE FROM Clients WHERE Id IN (...)