Triggers: How they work¶
Sidra defines triggers based on the triggers templates. Each trigger:
- is related to the Data Factory where the trigger will be created.
- is related to the pipelines that the trigger executes.
- provides some parameters that will be resolved with the placeholders of the trigger template.
Sidra incorporates different templates for ADF triggers in order to schedule the execution of the data extractor pipelines.
Triggers are basically representations of the specific schedule for running an ADF pipeline (example, daily executions at a fixed time).
TriggerTemplatesin order to instantiate a new trigger. Sidra Core API provides an endpoint to create a new trigger from the Id of the template.
The purpose of the trigger is to execute an associated pipeline at the desired time. Existing triggers can also be used instead of creating a new trigger.
In any case a request needs to be made to Sidra API to associate the data extractor pipeline with the newly created or existing trigger.
The parameters in a trigger are defined in a JSON format with this semantic:
- The properties are the name of the placeholders to be replaced without the outer "##".
- The values are the value used to replace the placeholder.
So, in the trigger template associated to this trigger:
- The placeholder ##basePath## will be replaced by ##landingZone/1/BasePath##.
- The placeholder ##scope## will be replaced by ##landingZone/1/StorageResourceId##.
As it can be seen in the previous example, the resolution has introduced two new placeholders in the trigger template. That is ok since the trigger template goes through several rounds of resolution of placeholders.
The resolution of placeholders can be divided in two parts:
- Generate the pipeline references.
- Generate the trigger JSON file used to create the trigger in Data Factory.
Pipeline reference generation¶
It uses the pipelines associated to the trigger table and the Pipeline reference template to obtain a JSON file with all the pipeline references. Then it replaces the placeholder ##Pipelines## in the trigger template with the JSON file.
It involves going through the following rounds. Every round resolves the Parameter source with the trigger template:
|1||Trigger parameters. This is the one that has been explained above. It takes the parameters included in Trigger and replace them in the
|2||Default values from the
|3||Global placeholders. It resolves some global placeholders that can be used in different types of templates, e.g. the landingZone placeholders -like landingZone/1/BasePath- and the dates placeholders -like DateTime.Today.Time-.|
|4||Configuration placeholders with their corresponding values. They are stored -both placeholder and value to resolve- in the
After all these rounds, the resulting JSON file is used to create the trigger in the Data Factory.