Skip to content

ADF Activities

Every activity template can refer to a native task supported in Data Factory -native activity- or to a custom task specifically created for a particular situation -custom activity-. Sidra provides a set of custom activities that covers all the tasks required by the pipelines templates that are not natively supported. Additional custom activities can be implemented and integrated in Sidra if necessary.

Activities, as datasets, can define parameters. The set of parameters of each activity can be consulted in Azure Data Factory documentation -for native activities- or in the documentation of the custom activity.

ActivityTemplate table

This table is used to define activity templates that represents an ADF activity but containing some placeholders that will be substituted in order to generate a valid JSON structure that defines an activity in ADF. The activity templates are associated to pipeline templates and can be reused by several of them.

The same way it happens to datasets, Sidra does not store activities by themselves but only activity templates. The generation of the JSON configurations to create the activities in Data Factory is implemented as part of the generation of the pipelines.

Column Description Format Required
Id ActivityTemplate identifier. It is unique within all the activity templates in the same Sidra installation int Yes
ItemId The global identifier of the template. It is unique within all the activity templates in all the Sidra installations uniqueidentifier Yes
Name Name of the ActivityTemplate, e.g. "Lookup" varchar(80) Yes
Description A description to tell what the ActivityTemplate does, e.g. "Executes a lookup activity" nvarchar(512)
Template A JSON structure defining the activity. As it is a template, it can contain placeholders nvarchar(max) Yes
DefaultValue A JSON structure that contains key-value pairs. The keys are placeholder names and the value is the default value to replace placeholder in case those values are not specified by other elements associated to the activity template. nvarchar(max) Yes

Template column example

{
    "name": "##lookupBlobActivityName##",
    "type": "Lookup",
    "dependsOn": [##dependsOn##],
    "policy": {
        "timeout": "03:00:00",
        "retry": 0,
        "retryIntervalInSeconds": 30,
        "secureOutput": false
    },
    "typeProperties": {
        "source": {
            "type": "BlobSource",
            "recursive": true
        },
        "dataset": {
            "referenceName": "##lookupBlobDatasetName##",
            "parameters": {             
                "fileName": "##lookupBlobFileName##",
                "folderPath": "##lookupBlobFolderPath##"
            },
            "type": "DatasetReference"
        },
        "firstRowOnly": ##firstRowOnly##
    }
}