Supported data synchronization mechanismsΒΆ
Sidra contains different mechanisms along the way to ensure the schema evolution and incremental load. This page is aimed to give a summarized view of the main features about this topic and clarify where they take place in the Sidra data ingestion process.
| Sync Mechanism | Data Intake type | Configuration | Where it is applied | Goal |
|---|---|---|---|---|
| Schema Evolution | Data Intake Process | Do you want the metadata to be refreshed automatically with each data extraction? option as true in the Sidra Web configuration for a DIP | Attributes from source tables schema | Automatic evolution of the source tables schema whenever new columns are added, resulting this in new Attributes in the Sidra Service metadata. New tables (if there are) will be added as well (new Entities). |
| Incremental Load non-Change Tracking | Data Intake Process | EntityDeltaLoad table | Core database | Identification of new inserts and updates through the IdDeltaAttribute column. This is done by configuring manually the table EntityDeltaLoad using a specific Attribute, IdDeltaAttribute column, which acts as a unique key to identify new inserts or updates in the source database. After that, the intake process automatically updates the table for the next incremental load. |
| Incremental Load with built-in Change Tracking (CT) in Azure SQL and SQL Server connectors | Data Intake Process | For information about how to enable it, review the official documentation. | Core database | Change Tracking creates change records that the data extractor pipeline in Sidra Service accesses on a per-table (Entity) basis during incremental updates. The intake pipeline detects automatically if it is enabled in origin by the user (it must be activated from there). This CT detects inserts, updates and deletes, updating the EntityDeltaLoad table with the last value for the CT and use it to query the next incremental load. |
| Incremental Load with built-in System-period temporal tables in DB2 and DB2 iSeries connectors | Data Intake Process | For information about how to enable it, review the documentation. | Core database | The system-period temporal table stores current versions of data and uses its associated history table to transparently store the updated and deleted data rows. |
| ConsolidationMode | Data Intake Process Data intake via landing zone | Field consolidationMode in AdditionalProperties column in the Entity table | Through the DSU ingestion script during the intake process in Databricks | To merge the data or append it. |
| Change Data Feed | Data Intake Process Data intake via landing zone | GenerateDeltaTable column in the Entity table | Databricks delta tables | To track row-level changes between versions of a Delta table. |