Skip to content

Supported data synchronization mechanisms

Sidra contains different mechanisms along the way to ensure the schema evolution and the data change tracking. This page is aimed to give a summarized view of the main features about this topic and clarify where they take place in the Sidra data ingestion process.

Sync Mechanism Data Intake type Configuration Where it is applied Goal
Schema Evolution Data Intake Process Do you want the metadata to be refreshed automatically with each data extraction? option as true in the Sidra Web configuration for a DIP Attributes from source tables schema Automatic evolution of the source tables schema whenever new columns are added, resulting this in new Attributes in the Sidra Core metadata. New tables (if there are) will be added as well (new Entities).
Incremental Load non-Change Tracking Data Intake Process EntityDeltaLoad table Core database Identification of new inserts and updates through the IdDeltaAttribute column. This is done by configuring manually the table EntityDeltaLoad using a specific Attribute, IdDeltaAttribute column, which acts as a unique key to identify new inserts or updates in the source database. After that, the intake process automatically updates the table for the next incremental load.
Incremental Load with built-in Change Tracking (CT) in Azure SQL and SQL Server plugins Data Intake Process For information about how to enable it, review the official documentation. Core database Change Tracking creates change records that the data extractor pipeline in Sidra Core accesses on a per-table (Entity) basis during incremental updates. The intake pipeline detects automatically if it is enabled in origin by the user (it must be activated from there). This CT detects inserts, updates and deletes, updating the EntityDeltaLoad table with the last value for the CT and use it to query the next incremental load.
ConsolidationMode Data Intake Process
Data intake via landing zone
Field consolidationMode in AdditionalProperties column in the Entity table Through the DSU ingestion script during the intake process in Databricks To merge the data or append it.
Change Data Feed Data Intake Process
Data intake via landing zone
GenerateDeltaTable column in the Entity table Databricks delta tables To track row-level changes between versions of a Delta table.

Sidra Ideas Portal

Last update: 2022-11-24
Back to top