Sidra Data Platform (version 2025.07: Fearless Felechosa)¶

Released on July 18, 2025

Another Databricks in the wall...

Welcome to the July 2025 release of the Sidra Data Platform! This month, we continue our journey toward deeper modularization and platform resiliency. We're laying important groundwork for long-term scalability with the brand-new DSU Management Service and aligning with the latest Databricks LTS runtime. As always, our focus remains on automation and a better developer and admin experience with every release.

New DSU Management Service¶

This release introduces the foundation of the DSU Management Service, a new Sidra component designed to handle the full lifecycle of Data Storage Units (DSUs) — including creation, installation, configuration, and future management operations. The long-term goal is to centralize and streamline DSU operations by progressively decoupling responsibilities currently handled by Sidra Core and Supervisor. Over the next releases, this service will evolve to become the primary control point for DSU deployments, regardless of type (databricks and OneLake) or environment.

As an initial milestone, the DSU Management Service now provides a dedicated API for managing Python package dependencies per Sidra service and version. This ensures that each service installs only the packages it requires—based on specific version constraints—reducing conflicts and avoiding unnecessary cluster restarts. A new database model has also been introduced in Llagar to track these dependencies across plugins and services, laying the groundwork for a more dynamic, service-specific runtime environment. This enhancement improves reliability, simplifies upgrade cycles, and sets the stage for a future where services like Data Quality or Data Intake can manage their runtime dependencies independently.

Updated Databricks Runtime¶

In this release, we've upgraded all Sidra Data Product templates and DSUs to use Databricks Runtime 16.4 LTS. This version includes Spark 3.5.2, delivering improved Delta Lake performance, automatic file compaction with liquid clustering, and enhanced support for structured streaming and SQL programmability. It also introduces native workspace file support for CI/CD scenarios and streamlines Auto Loader cleanup options. This upgrade ensures consistency across Sidra Core, DSUs, and Data Products, providing a more robust and scalable foundation for the entire platform.

Installation Process Improvements¶

A significant portion of Sidra's installation process depends on orchestrating and executing numerous Bicep/ARM templates and scripts. The original implementation through this release was exceedingly granular, containing a multitude of different scripts and templates. While this approach was convenient from a development perspective, we have rearchitected this system to reduce the total number of scripts and modify the relevant parts of our orchestration mechanism. This change significantly reduces the total time required for Sidra installation, as fewer PowerShell/Azure CLI execution contexts need to be created. More importantly, it increases installation process stability by greatly reducing the likelihood of transient failures.

Issues Resolved in Sidra 2025.07¶

The following issues have been resolved in this release:

Fixed a scenario that could lead to redundant executions of the ExtractMetadata pipeline in SQL Server Data Intake Processes. #9587
Fixed an issue where the Auto-Describe button was enabled for users without the necessary permissions to use it. #9590
Fixed a potential issue with filtering items when an Entity Framework global query filter is defined. #9685
Fixed an issue that led to an EF warning message regarding the return of the first item instead of the unique item. #9686
Explicitly configured Entity Framework QuerySplittingBehavior when a query includes collection navigation. #9687
Fixed a potential Entity Framework issue where a Value Converter was added without a Value Comparer. #9688
Adjusted Hangfire log traces to Debug level to reduce unnecessary log verbosity. #9689
Added an endpoint to check if a Provider exists. This prevents error messages in logs when attempting to retrieve a non-existent Provider. #9690

Coming Soon¶

We are developing the next release of Sidra, which will bring significant changes to the product. Due to the ambitious scope of this upcoming release and to provide our team with well-deserved time off, we are planning our next release for mid-September, skipping our August release. This schedule allows us to be more ambitious and release our new UI layer across parts of the product. Here's a preview of upcoming features:

Supervisor UI Redesign A major visual and usability upgrade of the Supervisor interface is in progress, delivering a more intuitive and responsive experience for platform administrators.
Data Product Improvements We will be applying to the data products the same event-based installation process the we currently implement on the services, which will improve the experience of updating a Data Product to a newer version. Also, we will be unleashing significant performance improvements to both the Sync process and the general Data Products' UI.
Entity Reload Next release will include a new API endpoint to automate entity reload to the DSU, improving some operational scenarios that currently had to be done via metadata management.

We Want Your Feedback¶

Your ideas make Sidra better. For suggestions, issues, or questions, please reach out to us at [email protected].