How to create a custom activity in Sidra¶
As commented in Custom activities overview, Sidra provides accelerators to easy the creation of new custom activities -in this document, as in most of the documentation, the term 'custom activity' will refer to the custom application, not to the Azure Data Factory (ADF) activity-:
- The Visual Studio template available in PlainConcepts.Sidra.DotNetCore.CustomActivityTemplate generates the Visual Studio solution for a custom activity.
- The NuGet package PlainConcepts.Sidra.DataFactory.CustomActivities.Common -shorten as the Common package- provides a set of base classes and helpers to read and manage the information that the custom activity receives from ADF.
Custom activity template¶
The section Create new solution explains how to install a new Visual Studio template and use it to generate a Visual Studio solution.
The template for generating a custom activity does not receive any additional parameter. The generic parameter --name
can be used to configure the name of the output folder which will be also used as name of the custom activity.
For example, assuming that the template for custom activities is already installed, the following command can be used to create a custom activity named 'CA'.
1 2 3 |
|
If the solution is opened with Visual Studio, the content showed in the Solution Explorer will be:
The previous solution contains two projects:
- PlainConcepts.Sidra.DataFactory.CustomActivities.CA is a .NET Core class library project that will include the code that will be executed as custom activity. It will be called CA project for shorten.
- PlainConcepts.Sidra.DataFactory.CustomActivities.CA.Tests is a .NET Core unit testing project using xUnit and Moq to unit test the CA project.
The CA project contains some boilerplate code and a reference to the Common package. The project is prepared to just configure the parameters and add the specific logic of the custom activity.
IMPORTANT
At this stage, the solution will not build because it is missing some
.props
files. The reason is that the previous projects are meant to be added to an existing project solution -it could be a Core solution or a Client app solution- that already contains those.props
files.
Classes and helpers for custom activities¶
The CA project includes a set of classes that handles the JSON files activity.json
, linkedServies.json
and datasets.json
which provide the input parameters to the activity. Those parameters can be classified into the 'referenceObjects' and the 'extendedProperties'. It also defines a set of base parameters that are available for all the custom activities.
Using those classes, it is very easy to access all the parameters not being necessary to write the code to deserialize the JSONs and accessing its properties every time that a new custom activity is created.
For example, having the following activity.json
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
|
It can be seen that it includes the following parameters:
- Six 'extendedProperties':
tableName
,projectsNumber
,role
,AzureKeyVaultLinkedServiceName
,AzureSqlLinkedServiceName
andAzureSqlTableDatasetName
- Three 'referenceObjects': two linked services
MyAzureSqlLinkedService
andMyAzureKeyVaultLinkedService
and a datasetMyAzureSqlTable
The code snips below show how to configure the provided classes to get access to the previous parameters.
Constants¶
The Constants
class is used to define the names of the parameters. It can also be used to define default values in case a parameter wants to be treated as optional.
1 2 3 4 5 6 7 8 9 10 11 |
|
ActivityParameters¶
The ActivityParameters
class is used to:
- Define the C# properties that will receive the values of the parameters.
- Assign default values to the previous properties.
- Validate the values.
The ActivityParameters
inherits from BaseParameter
which is provided by the Common package.
BaseParameters¶
The BaseParameter
class defines:
- A virtual method called
Validate
, which must be overwritten in case it is necessary to validate the parameters passed to the activity. For example checking that mandatory parameters are present. - Some common parameters already defined to all custom activities:
AzureKeyVaultLinkedServiceName
is the name of the linked service that references the Azure Key Vault that stores the sensitive information used by other linked services. For example a linked service to a SQL Server can store the connection string in the aforementioned Azure Key Vault.IsKeyVaultEnabled
is a flag that represents if the Key Vault is enabled. It is mandatory, so, if there are referenced objects that store sensitive data and the Key Vault is not enabled, the custom activity execution will throw an error.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
|
ActivityParametersResolver¶
The ActivityParametersResolver
class is used to map and cast the raw values received from ADF in the activity.json
file into the C# properties defined in ActivityParameters
.
It inherits from ParameterResolver
class which needs an object that implements ICustomActivityReferenceProvider
for it instantiation. The dependency injector is configured to provide an implementation of ICustomActivityReferenceProvider
called CustomActivityReferenceProvider
that is included in the Common package.
ParameterResolver¶
The ParameterResolver
class defines two methods:
Resolve
must be overwritten. It creates an instance ofActivityParameters
with all parameters mapped.GetValue
allows to obtain a parameter from theactivity.json
file by its name.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
|
Activity¶
The Activity
class contains the main code to be executed by the custom activity. It inherits from BaseActivity
, which is a generic type and needs to be typed with an instance of BaseParameter
.
BaseActivity¶
The BaseActivity
defines:
Execute
is virtual method that must be overwritten to include the specific business logic of the custom activity.- Methods to obtain the referenced objects by their names. For example, using the method
GetAzureSqlDatabase
and passing as parameter theAzureSqlLinkedService
it retrieves theAzureSqlDatabase
that can be used to connect to the database.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
|
ActivityExecutor¶
The ActivityExecutor
class provides the entrypoint to execute the custom activity which is the Execute
method. It will be requested by the custom activity host project.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
|
DependencyInjector¶
The DependencyInjector
class configures the dependency injector. If the custom activity does not use any linked service, the configuration provided by the Visual Studio template should be enough but if it uses any linked service then some additional configuration is necessary.
Azure Key Vault configuration¶
The secrets that linked services use to connect to the data sources -for example, the connection string to connect to a database in an Azure SQL linked service- should be stored in Azure Key Vault.
The custom activity needs to access to that Azure Key Vault in order to retrieve the secrets, so it will need an Azure Key Vault a linked service for that purpose. The name of that Azure Key Vault linked service will be configured in the AzureKeyVaultLinkedServiceName
which is the parameter inherited by the BaseParameter
class.
The code below shows the additional configuration to use the Azure Key Vault linked service:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
|
Additionally it will be necessary to add a reference to the NuGet package PlainConcepts.SIDRA.DotNetCore.Common.
Integrating CA projects¶
CA projects can be managed in two different ways depending on if they are going to be included in the set of custom activities provided by Sidra or if they are going to be used exclusively in a specific Sidra installation.
Included in Sidra¶
Those CA projects that cover an activity generic enough to be used by several Sidra installations can be included in the set of custom activities provided by Sidra. The source code of those CA projects will be added to the Sidra repository and exposed as NuGet packages so any Sidra installation could use them by means of a host project as commented in Custom apps in Sidra.
Specific Sidra installation¶
The CA projects too specific or tight to a Sidra installation can be used directly. In that case, the source code of those CA projects will be added to the specific installation solution and a couple of changes must be performed:
- Change the type of the CA project to .NET Core console application instead of class library.
- Add a
Program
class that calls the entrypoint of the custom activity. TheProgram
class is exactly the same that the one included in the host projects and can be seen in the Custom apps in Sidra.