Sidra Data Platform (version 2024.06: Whistling Wealthy)¶

released on Jun 28, 2024

Consolidation Improvements and Housekeeping

As highlighted in last month's release notes, for this sixth release of the year, we are keeping new features to a minimum. This allows us to dedicate enough bandwidth to support our customers and partners engaged in the private preview of Microsoft Fabric's DSU. However, we have managed to release a highly requested feature: the ability to specify the consolidation mode of the Data Intake Processes via the Web user interface. Additionally, we have introduced a new Overwrite consolidation mode.

We are also including some fixes and quality of life improvements, detailed in these release notes.

What's new in Sidra 2024.06¶

Consolidation Mode from the UI¶

While Sidra has supported different consolidation modes on the DIPs for several years, changing them required amending the Additional Properties JSON column in the metadata database. Based on user feedback and in line with our efforts to make more Sidra capabilities accessible via the Web UI, we have now made this configuration part of the Data Intake Process wizard.

With this release, users can specify the consolidation mode of new DIPs or amend existing DIPs consolidation modes through the UI.

Overwrite Consolidation Mode¶

We are also introducing a new consolidation mode for the DIP: Overwrite mode. Previously, the available consolidation modes for loading data to the DSU were Append and Merge. With the new Overwrite mode, the system removes the current content of the entity and loads the new data.

Find out more about the intake consolidation modes.

Pinned Clusters¶

By default, Databricks clusters are automatically removed after thirty days of inactivity. This has impacted some users, especially in lower environments (Dev, Test, etc.), often requiring redeployment of the DSU. The deployment process adds configuration to the cluster and creates the ADF linked service.

In this release, we have pinned the DSU and Data Product clusters, ensuring they will not be auto-deleted. This will not impact pricing, as clusters are not charged if they are not actively used.

Issues Resolved in Sidra 2024.06¶

These are the issues resolved during this release in Sidra:

Fixed an issue that would allow the DQ service to create reports that the previous versions of Sidra's UI would not manage correctly. #7715
Fixed an issue that would prevent the update of an environment variable under certain scenarios. # 7644
Fixed an issue with the CSV intake that would prevent the creation of the Fabric Lakehouse. #7634
Change metadata creation to avoid an ADF issue that cause a partial metadata creation when the source schema return more than 5000 records. #7603

Coming soon...¶

Our next release will also be limited in new functionality, as we will focus on making improvements to Llagar—Sidra's back office—to create a more streamlined plugin distribution mechanism with different maturity channels. Additionally, with some well-deserved planned holidays for our team, we will keep our backlog realistic for the July release.

We Want Your Feedback!¶

Your thoughts make Sidra better. For suggestions, issues, or questions, please reach out to us at [email protected].