Sidra Data Platform (version 2024.05: Vigilant Virginia Gold)¶

released on May 31, 2024

Welcome to the Unity Catalog!

We are excited to announce the latest release of Sidra, which introduces a significant update to our platform's architecture. With this release, we are fully embracing Databricks' Unity Catalog, unlocking advanced scenarios for data sharing, security, and governance. We are also announcing today our private preview programm for a Microsoft's OneLake-backed DSU type, enabling our users and partners to adopt Microsoft's new storage and compute layer on Microsoft Fabric.

Additionally, we are rolling out major enhancements to our Data Quality reports. These improvements provide a simplified and intuitive way to visualize and navigate the results of the validation processes directly from Sidra's UI.

What's new in Sidra 2024.05¶

Unity Catalog in Sidra¶

After some time working behind the curtains on this, we can now announce that we have adopted Databrick's Unity Catalog as part of the deployment of Sidra. Unity Catalog serves as a centralized hub for managing access control, conducting audits, tracing data lineage, and discovering data across various Databricks workspaces. Additionally, Unity Catalog lays the groundwork for the integration of upcoming features in subsequent releases of Sidra.

Find out more about how to deploy Sidra using Unity Catalog.

Support for OneLake¶

With this release of Sidra, we are taking our first steps toward enabling our customers to adopt Microsoft Fabric at a fraction of the cost while mitigating the inherent risks of adopting new technology. We are excited to introduce a new OneLake-backed DSU type, which allows users to continue using the Databricks-backed DSU while evaluating OneLake capabilities.

This feature is currently in private preview. If you are interested in trying it out, please contact us at [email protected], and we will get back to you promptly.

Find out more about how to deploy a OneLake-backed DSU type.

Consolidated Data Quality Reports¶

Based on feedback from our users, we have made several UX improvements to the Data Quality reports. You can now download a single comprehensive report for an entire Provider, which includes the data docs for all entities associated with that Provider.

Issues Resolved in Sidra 2024.05¶

These are the issues resolved during this release in Sidra:

Fixed an issue in the SQL Server connector that would prevent performing a data intake from a source database whose name had a hypen on it. #6237
Fixed an issue in which tables with long names can cause the intake process to DSU to fail. #7423
Fixed an issue in which data ingestion of tables with a very large number of columns could fail silently. #7424
Fixed an issue in which, under certain conditions, would allow Databricks to connect to the Core DB using a public IP instead the VNet IP. #7519

Coming soon...¶

The upcoming release is anticipated to be relatively light on new features, as the majority of our engineering efforts are focused on refining our support for OneLake in collaboration with our early adopters. However, we do plan to introduce a new consolidation mode and a new connector, so stay tuned for those updates.

We Want Your Feedback!¶

Your thoughts make Sidra better. For suggestions, issues, or questions, please reach out to us at [email protected].