Skip to content

ADF component templates

Sidra provides a set of ADF components templates that are used to create the workflows to ingest the data from the sources into the platform -specifically into Sidra- and also to move data from Sidra into the Data Products.

The templates are the same JSON structures that define an ADF component but containing placeholders. The placeholders are identified by beginning and ending with ##. When the placeholders are resolved with actual values, the resulting JSON structure is the definition of a component that can be used to programmatically create the component in ADF.

Examples: templates defining a trigger

Case 1

{
    "name": "##name##",
    "properties": {
        "type": "BlobEventsTrigger",
        "typeProperties": {
            "blobPathBeginsWith": "##basePath##",
            "scope": "##scope##",
            "events": [
                "Microsoft.Storage.BlobCreated"
            ]
        },
        "pipelines": [
            ##Pipelines##
        ]
    }
}

It contains the following placeholders:

  • ##name##: it will be resolved with the name of the trigger, e.g. "Sidra Storage Blob Created".
  • ##basePath##: it will be resolved with the path of the blob where the trigger will be checking if there are new files, e.g. "/landing/blobs/".
  • ##scope##: it will be resolved to the reference to the Azure Storage account, e.g. "/subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/resourceGroups/Dev.NorthEurope/providers/Microsoft.Storage/storageAccounts/xxxxx".
  • ##Pipelines##: it will be resolved with the pipelines that are executed by this trigger. Those pipelines are obtained by the resolution of the following Pipeline reference template, which at the same time is using the ##pipelineName## placeholder.

Case 2

{
    "pipelineReference": {
        "type": "PipelineReference",
        "referenceName": "##pipelineName##"
    },
    "parameters": {
        "folderPath": "@triggerBody().folderPath",
        "fileName": "@triggerBody().fileName"
    }
}

Every ADF component that defines a template, also defines a set of default values for the placeholders of the template. Although, it can happen that there is no default value for a specific placeholder.

The values used to resolve the placeholders are contained in the metadata database. So by using ADF component templates and their resolving values, Sidra can automatically recreate -create, update, delete- all the data workflows. All this work -resolving the placeholders and updating the Data Factories- is implemented by the Data Factory Manager.

The way to extend the Sidra platform with new workflows is by adding more information to the metadata database that defines new pipelines, activities, triggers, etc.