Skip to content

Sync UI for Data Products

The Sync UI is Sidra's reverse ETL engine, enabling intuitive data synchronization between Data Storage Units (DSUs) and Data Products. Through a visual interface, users can configure how data is transferred, transformed, and staged downstream.

Info

Before proceeding, ensure you have deployed a Data Product. Follow the deployment guide.

Once deployed, you can manage data synchronization by selecting the Sync button. If a Data Product version becomes incompatible after a Sidra Service upgrade, the button will be disabled and a tooltip will explain the required upgrade.

Sync UI settings

Clicking the Sync button opens the synchronization configuration interface, where you can define and manage Data Synchronization Processes (DSPs).

Data Synchronization Process (DSP) Overview

The Sync UI enables the creation and management of Data Synchronization Processes (DSPs) — processes that copy data from DSUs to staging areas in the Data Product. Configuration options include:

  • Sync Mode
  • Consolidation behavior
  • Entity selection
  • Staging table configuration

Pipeline Details

Create a New Data Synchronization Process

Use the New button in the top right corner to define a new DSP.

Create New Pipeline

Execution Parameters

This field only appears for Sync Modes that require parameters, such as Databricks-based orchestrations.

Example parameters:

{
  "storedProcedureName": "[staging].[Orchestrator]",
  "orchestratorNotebookPath": "/Shared/MyNotebook"
}

Configure Existing Data Synchronization Process

Click the gear icon on any DSP to view and edit its settings.

Details

Displays summary information about the selected DSP.

Details

Associated Entities

Lists all Entities linked to the Data Synchronization Process (DSP). Only selected Entities will be synchronized from DSU to the Data Product.

Associated Entities

Click Edit to modify Entity associations.

Edit Associated Entities

This screen allows:

  • DSU/Provider/Entity selection
  • Mandatory entity flag
  • Consolidation Mode changes
ConsolidationMode

The ConsolidationMode setting determines how data is written: Overwrite, Merge (default), or Append. Defined in PipelineExecutionProperties. Learn more.

Mandatory

A Data Synchronization Process will not execute until data for all mandatory Entities is available in the DSU.

Staging Configuration

This section reflects the contents of the StagingConfiguration table. It defines how synchronized data is mapped to SQL staging tables.

Staging Table Configuration

Status of Load Processes

Displays the latest ADF pipeline executions for the DSP, including timestamps and durations.

Pipeline Status

Example

Values for Run End and Duration are retrieved from ADF and may become unavailable over time.

Reload Entities

Allows for selective data reloads from DSU to the Data Product. You can reload a full Entity or define a starting date for incremental reload.

Reload Entities

Warning

Reloading full Entities may be resource intensive and should be used with caution.

Edit Data Synchronization Process

You can update all configuration settings, including the Sync Mode and parameters, by clicking the pencil icon.

Edit Sync Pipeline

Note

A description is required when creating or editing a Data Synchronization Process (DSP).

Delete Data Synchronization Process

Deleting a DSP removes it from Sidra Web and ADF. The deletion is logical, meaning it can be recovered later if needed.

Info

For practical examples using Sync UI, refer to the Data Domain Data Product template.

Limitations and Known Issues

  • Description field: max 256 characters
  • Title field: max 250 characters
  • Empty title values in edit mode may reset to original unexpectedly
  • Adding a new image fails to trigger image selector
  • Renaming a Data Product may require the image to be reloaded