Skip to content

Sync UI for Data Products

Sidra's wizard now includes the option to check and interact with the different artifacts of a Sidra Data Product without the need to do changes in the database, Azure App Service or ADF. This is possible thanks to the Settings button added inside a specific Data Product (image below). If the Settings button is not available is because the Data Product needs to be updated.

Sync UI settings image

By clicking on it, the Sidra's UI will allow the following actions.

1. Pipelines

This section shows the Data Product intake pipelines. Also, the creation of new pipeline is now possible as well as the management of existing pipelines, changing:

  • The template used
  • The behaviour to synchronize Assets
  • The Entities to be consolidated
  • The consolidation mode used
  • The definition of the Staging tables

Sync UI settings pipelines details image

The different edition options are described as follows.

1.1 New pipeline

By the top right button New, the user now is able to add a new pipeline with the desired template and its parameters as, for example, the stored procedure name which loads the data, that is, the orchestrator. Furthermore, the sync behaviour can be configured here with the values: Load All New Dates, Load First New Date, Load Last New Date, Load Pending Up To Valid Date...

Sync UI new pipeline

1.2 Configuration pipeline

After clicking on the gear button, a new page for details of configuration is available which is composed of several sections:

  1. Details: informative box the chosen pipeline.

    Sync UI details

  2. Associated Entities

    In this part the Entities associated with this particular pipeline can be checked. This is the relationship depicted in the Sidra.EntityPipeline table in the database of the Data Product. This means, the selected Entities will be the ones extracted from the DSU to the Staging tables.

    Sync UI associated entities

    An Edit button is enabled in order to establish this Entity-pipeline relationship directly, avoiding the manual change in the database for associating Entities as described here.

    Sync UI associated entities edition

    In this screen, DSU, Provider and Entity information is depicted. The Consolidation Mode can be modified here as well as including Entities in the pipeline or mark the field Mandatory. When an Entity is included here, the Staging Configuration section is updated with the new Entity data.

  3. Staging Configuration

    This section incorporates the table Sidra.StagingConfiguration of the Data Product, which is responsible for adding additional configuration to the Entities associated with the pipelines: schema name, table name, adding a query for the Databricks, etc.

    This part will be automatically populated with default values when adding Entity-pipeline relationships in the section 2 (Edit button in Associated Entities).

    Sync UI staging

  4. Status of Load Processes

    This important section is showing the data load information (ADF pipelines status) from Sidra Data Product database and ADF. When an error occurs in the data load, an info icon will be shown with the ADF message on it.

    Run End and Duration fields will depend on ADF records so, with the time, this information will be no longer shown.

    Sync UI staging

1.3 Reload Entities

A button is included in order to reload Entities after any error or situation that could happen. This button replaces the manual change of status in the database for these Entities meaning in a huge time saving. Regarding the Start Date window:

  • Start Date will need to be specified for the Entities reload for a specific date range (start date to current date).
  • If it is not specify, the entire Entity will be reloaded.
  • A warning will appear informing about the scale-up of resources.

Sync UI reload entities

1.4 Edit pipeline

The same parameters configured when we create a new pipeline, can be changed in this section by the button for editing (pencil).

Sync UI edit pipeline

1.5 Delete pipeline

When doing the delete option, the pipeline in the Data Product database will appear as removed (IsRemoved column set to 1) and the current screen in the UI will not show the pipeline. In the backend, the API will trigger the DataFactoryManager webjob deploying the changes in ADF, removing the pipeline.

2. Assets

This part acts as an informative part depicting the Assets of the Data Product sorted by date.

Sync UI settings pipelines details image

Known issues

  • When clicking on Edit button or when creating a new Data Product, Description field must be filled.
  • Description field has a maximum of 256 characters.
  • Title field has a maximum of 250 characters.
  • When removing the whole title in Edit option, some unexpected behavior can happen returning the original value.
  • When editing a Data Product to add a new image, image selector will not open.
  • When the user name for a Data Product is changed, the image must be loaded again.