Sync UI for Data Products¶
The Sync UI is Sidra's reverse ETL engine, enabling intuitive data synchronization between Data Storage Units (DSUs) and Data Products. Through a visual interface, users can configure how data is transferred, transformed, and staged downstream.
Info
Before proceeding, ensure you have deployed a Data Product. Follow the deployment guide.
Once deployed, you can manage data synchronization by selecting the Sync
button. If a Data Product version becomes incompatible after a Sidra Service upgrade, the button will be disabled and a tooltip will explain the required upgrade.
Clicking the Sync button opens the synchronization configuration interface, where you can define and manage Data Synchronization Processes (DSPs).
Data Synchronization Process (DSP) Overview¶
The Sync UI enables the creation and management of Data Synchronization Processes (DSPs) — processes that copy data from DSUs to staging areas in the Data Product. Configuration options include:
- Sync Mode
- Consolidation behavior
- Entity selection
- Staging table configuration
Create a New Data Synchronization Process¶
Use the New
button in the top right corner to define a new DSP.
Execution Parameters
This field only appears for Sync Modes that require parameters, such as Databricks-based orchestrations.
Example parameters:
{
"storedProcedureName": "[staging].[Orchestrator]",
"orchestratorNotebookPath": "/Shared/MyNotebook"
}
Configure Existing Data Synchronization Process¶
Click the gear icon on any DSP to view and edit its settings.
Details¶
Displays summary information about the selected DSP.
Associated Entities¶
Lists all Entities linked to the Data Synchronization Process (DSP). Only selected Entities will be synchronized from DSU to the Data Product.
Click Edit
to modify Entity associations.
This screen allows:
- DSU/Provider/Entity selection
Mandatory
entity flag- Consolidation Mode changes
ConsolidationMode
The ConsolidationMode
setting determines how data is written: Overwrite
, Merge
(default), or Append
. Defined in PipelineExecutionProperties
. Learn more.
Mandatory
A Data Synchronization Process will not execute until data for all mandatory Entities is available in the DSU.
Staging Configuration¶
This section reflects the contents of the StagingConfiguration
table. It defines how synchronized data is mapped to SQL staging tables.
Status of Load Processes¶
Displays the latest ADF pipeline executions for the DSP, including timestamps and durations.
Example
Values for Run End
and Duration
are retrieved from ADF and may become unavailable over time.
Reload Entities¶
Allows for selective data reloads from DSU to the Data Product. You can reload a full Entity or define a starting date for incremental reload.
Warning
Reloading full Entities may be resource intensive and should be used with caution.
Edit Data Synchronization Process¶
You can update all configuration settings, including the Sync Mode and parameters, by clicking the pencil icon.
Note
A description is required when creating or editing a Data Synchronization Process (DSP).
Delete Data Synchronization Process¶
Deleting a DSP removes it from Sidra Web and ADF. The deletion is logical, meaning it can be recovered later if needed.
Info
For practical examples using Sync UI, refer to the Data Domain Data Product template.
Limitations and Known Issues¶
Description
field: max 256 charactersTitle
field: max 250 characters- Empty title values in edit mode may reset to original unexpectedly
- Adding a new image fails to trigger image selector
- Renaming a Data Product may require the image to be reloaded