Skip to content

Triggers: How they work

Sidra defines triggers based on the triggers templates. Each trigger:

  • is related to the Data Factory where the trigger will be created.
  • is related to the pipelines that the trigger executes.
  • provides some parameters that will be resolved with the placeholders of the trigger template.

Key aspects

  • Sidra incorporates different templates for ADF triggers in order to schedule the execution of the data extractor pipelines.

  • Triggers are basically representations of the specific schedule for running an ADF pipeline (example, daily executions at a fixed time).

  • Sidra provides TriggerTemplates in order to instantiate a new trigger. Sidra Service API provides an endpoint to create a new trigger from the Id of the template.

  • The purpose of the trigger is to execute an associated pipeline at the desired time. Existing triggers can also be used instead of creating a new trigger.

  • In any case a request needs to be made to Sidra API to associate the data extractor pipeline with the newly created or existing trigger.

Triggers parameters

The parameters in a trigger are defined in a JSON format with this semantic:

  • The properties are the name of the placeholders to be replaced without the outer "##".
  • The values are the value used to replace the placeholder.

Example

{
    "basePath": "##landingZone/1/BasePath##",
    "scope": "##landingZone/1/StorageResourceId##"
}

So, in the trigger template associated to this trigger:

  • The placeholder ##basePath## will be replaced by ##landingZone/1/BasePath##.
  • The placeholder ##scope## will be replaced by ##landingZone/1/StorageResourceId##.

As it can be seen in the previous example, the resolution has introduced two new placeholders in the trigger template. That is ok since the trigger template goes through several rounds of resolution of placeholders.

The resolution of placeholders can be divided in two parts:

  1. Generate the pipeline references.
  2. Generate the trigger JSON file used to create the trigger in Data Factory.

Pipeline reference generation

It uses the pipelines associated to the trigger table and the Pipeline reference template to obtain a JSON file with all the pipeline references. Then it replaces the placeholder ##Pipelines## in the trigger template with the JSON file.

Trigger generation

It involves going through the following rounds. Every round resolves the Parameter source with the trigger template:

Round Parameters source
1 Trigger parameters. This is the one that has been explained above. It takes the parameters included in Trigger and replace them in the TriggerTemplate.
2 Default values from the TriggerTemplate. All templates -including TriggerTemplates- have defined a DefaultValue for the placeholders defined in the template. If some of the placeholders have not been replaced in the round 1 then they will be replaced with the default values of the template.
3 Global placeholders. It resolves some global placeholders that can be used in different types of templates, e.g. the landingZone placeholders -like landingZone/1/BasePath- and the dates placeholders -like DateTime.Today.Time-.
4 Configuration placeholders with their corresponding values. They are stored -both placeholder and value to resolve- in the [Management].[Configuration] table in the Core database.

After all these rounds, the resulting JSON file is used to create the trigger in the Data Factory.