Sidra Data Platform (version 2024.07: Yummy York)¶

Released on July 31, 2024

A very stable summer ahead...

As the Sidra team gears up for a well-deserved summer break, we are aware that the enjoyment of our break is directly proportional to the stability of the platform for our current customers. Therefore, we are releasing a new Sidra version where the main theme is stability and resiliency.

What's new in Sidra 2024.07¶

Networking Improvements¶

One of the main objectives of this release was to optimize the communications between different services, ensuring all communications remain within private endpoints for the best performance and privacy possible. This was achieved by refactoring all private endpoints, including storage, SQL databases, and DNS Zone handling.

Improved Retry Policies¶

In response to recent scenarios where transient network problems were experienced within the infrastructure of some customers, we have revised our retry policies in several areas of the code. The most significant changes involve handling updates to asset metadata after intake and inserts on time-travel enabled Delta tables in Data Products. These changes increase the platform's resilience to potential external network infrastructure issues.

Azure Resource Tag Management¶

Many organizations have Azure resource tagging policies and mandatory tags. To simplify Sidra deployment under these conditions, Sidra now supports defining tags that will be automatically applied to all Resource Groups created by the platform, both at installation and execution time.

Note that in this release, we are only creating tags for resource groups, not individual resources. Applying tags to resource groups and inheriting them for individual resources is planned for a future Sidra version.

New Data Product Sync Template¶

Based on feedback from customers and partners, we have added a new Sync template to the Data Domain data product, which replicates data only to Databricks. After this change, the Data Domain now has two Sync modes: "Copy Entities to Databricks and Azure SQL Database" and "Copy Entities to Databricks," depending on the intended usage.

Removed Local User Authentication¶

Starting with this release, Sidra no longer supports username/password access to the platform, making Azure Active Directory access the only option. While some earlier customers requested non-AD based authentication, this has not been requested or used recently. Removing it helps us reduce Sidra's potential security risks by reducing its surface area.

Issues Resolved in Sidra 2024.07¶

These are the issues resolved during this release in Sidra:

Fixed an issue where a secret could not be retrieved from the Key Vault when installing Sidra with a OneLake DSU. #7789
Fixed an issue in the virtual network rules configuration for some resources of the DSU when injecting Sidra on an existing VNET. #7821
Fixed an issue on all data intake processes which would prevent merging the data in the DSU if the entity had a multi-column primary key. #7863
Fixed an issue that would prevent the creation of users which subject identifier cointained an underscore character via the Sidra UI. #7868
Fixed an issue in the Data Preview of an Entity which prevented seeing masked data. #7809
Fixed an issue in the Data Product data synchronization that, under certain scenarios, could create duplicated files. #7798
Fixed an issue which would prevent the default Sidra Provider to be deployed on a fresh installation. #7827
Fixed an issue in the Data Catalog where the data was not shown even it the user had the rights to see them. #7815
Reduced a number of redundant calls to Sidra API from the SDK to prevent flooding and improve performance. #7921

Coming soon...¶

Although our team is committed to releasing a version in August, the limited capacity due to holidays and the Mojito-Index™ makes it inadvisable to aim for any major new feature releases. For this reason, we are planning another stability release.

We Want Your Feedback!¶

Your ideas make Sidra better. For suggestions, issues, or questions, please reach out to us at [email protected].