Supported data synchronization mechanisms¶
Sidra contains different mechanisms along the way to ensure the schema evolution and the data change tracking. This page is aimed to give a summarized view of the main features about this topic and clarify where they take place in the Sidra data ingestion process.
| Sync Mechanism | Data Intake type | Configuration | Where it is applied | Goal |
|---|---|---|---|---|
| Schema Evolution | Data Intake Process | Do you want the metadata to be refreshed automatically with each data extraction? option as true in the Sidra Web configuration for a DIP |
Attributes from source tables schema | Automatic evolution of the source tables schema whenever new columns are added, resulting this in new Attributes in the Sidra Core metadata. New tables (if there are) will be added as well (new Entities). |
| Incremental Load non-Change Tracking | Data Intake Process | EntityDeltaLoad table |
Core database | Identification of new inserts and updates through the IdDeltaAttribute column. This is done by configuring manually the table EntityDeltaLoad using a specific Attribute, IdDeltaAttribute column, which acts as a unique key to identify new inserts or updates in the source database. After that, the intake process automatically updates the table for the next incremental load. |
| Incremental Load with built-in Change Tracking (CT) in Azure SQL and SQL Server plugins | Data Intake Process | For information about how to enable it, review the official documentation. | Core database | Change Tracking creates change records that the data extractor pipeline in Sidra Core accesses on a per-table (Entity) basis during incremental updates. The intake pipeline detects automatically if it is enabled in origin by the user (it must be activated from there). This CT detects inserts, updates and deletes, updating the EntityDeltaLoad table with the last value for the CT and use it to query the next incremental load. |
| ConsolidationMode | Data Intake Process Data intake via landing zone |
Field consolidationMode in AdditionalProperties column in the Entity table |
Through the DSU ingestion script during the intake process in Databricks | To merge the data or append it. |
| Change Data Feed | Data Intake Process Data intake via landing zone |
GenerateDeltaTable column in the Entity table |
Databricks delta tables | To track row-level changes between versions of a Delta table. |
Last update:
2022-11-24