Sidra Data Platform (version 2024.01: Radiant Rubinette)¶
released on Feb 2, 2024
We're excited to announce the first Sidra release of the year! Our Data Quality Service has been enhanced with a more intuitive interface in Sidra Web and a better validations guidance. Plus, PII detection is now seamlessly integrated into Sidra version 2.0. We've also optimized the Databricks cluster setup times by incorporating local packages. Explore these updates and discover more!
Sidra 2024.01 Release Highlights¶
- Enhanced Data Quality Service
- PII Detection Sidra 2.0
- Integrated local packages in Databricks cluster
What's new in Sidra 2024.01¶
Enhanced Data Quality Service¶
Since its launch, the Data Quality Service in Sidra has undergone significant evolution! Now available are enhanced UI features in Sidra Web and comprehensive documentation, complete with test cases, to facilitate Expectations configuration.
PII Detection Sidra 2.0¶
With this release, we are excited to introduce a fully functional PII Detection feature for Sidra 2.0. This advanced capability ensures the identification and secure handling of Personally Identifiable Information, enhancing data protection and compliance within our system.
See documentation of PII Detection in Sidra.
Integrated local packages in Databricks cluster¶
To overcome geolocation-based connectivity issues that hinder package downloads in Databricks clusters, Sidra Service will now preload necessary packages directly to the Databricks File System. This approach bypasses the need to access third-party locations from customer environments and optimizes cluster initialization time.
Issues Resolved in Sidra 2024.01¶
These are the issues resolved during this release in Sidra:
- Resolved a problem related to the comparison of incremental load data in database connectors. #5490
- Solved an issue where Data Product name was not sanitized properly and failed to register. #5761
- Modified the Sidra dashboard on Sidra Web to make the Available Services section interactive, allowing each service to be clickable and display related information. #5844
- Increased the character limit for the Data Domain resource group name to up to 50 characters. #5926
- Fixed an issue where Sidra Web Supervisor was not displaying any error when not retrieving the parameters form for Data Quality Service during its installation. #5974
- Adjusted name of Data Product in Authorization section on Sidra Web. #6049
- Resolved a problem in SQL Server data intake pipeline template where ExtractData and Orchestrator pipelines were not deploying correctly. #6055
- Fixed a deployment issue of an additional DSU which created a storage trigger referring to the default storage account instead of its own. #6076
- Improved view of Data Quality Service and validations section in Sidra Web. #6083 #6085 #6088 #6089 #6097 #6154
- Fixed an issue when accessing with DataQualityReader role in the validation list within Data Catalog section. #6084
- Resolved an issue in the main ingestion script occurring when no secret was created when the Data Quality service was not installed. #6086
- Adjusted displayed Expectations for Data Quality in Sidra Web, since some of them were not working properly with the Service. #6115
- Fixed a problem where Sidra Service was not redeploying additional DSUs. #6133
- Solved an issue in the main ingestion script where the validation parameter failed to reset to null upon record deletion. #6134
- Adjusted time zones in Data Products settings screen in Sidra Web. #6153
- Fixed an issue where available Data Products version in Sidra Web were appearing as disabled. #6159
- Improved the protection of sensitive information in Container App Console Log. #6167
- Resolved a problem in Sidra Web's Data Quality validation section, where configuring several validations for the same Entity and JSON raised an error. #6191
- Fixed an issue in the SDK related to SparkContext that was preventing the proper functioning of PII. #6280
- Adjusted some duplicated data intake icon in Sidra Web. #6324
- Set Supervisor Replicas to a minimum and maximum of one to prevent incorrect values in status tables. #6381
- Fixed an issue with the synchronization of certain Entities between Sidra Service Core database and Data Product. #4337
- Fixed specific errors in ADF pipelines associated with Data Intake Processes. #5464
- Resolved an issue with Authentication Service policies that prevented the installation of Sidra from scratch. #6293
- Fixed an issue where the additional deployed DSU failed to retrieve cluster information. #6294
- Resolved an issue with dependencies affecting topic creation in the Integration Hub. #6309
- Solved a problem where Data Product synchronization failed if the Entity had no new Assets. #6364
- Improved tables size in Data Product Sync section in Sidra Web Manager. #6416
- Resolved an issue with plugin retrieval in the Data Product section during creation. #6466
Coming soon...¶
Get ready for thrilling new developments heading your way shortly. Our team is enhancing the Integration Hub feature and creatively crafting a BI Data Product solution. Additionally, we're refining the Sidra Web interface to make it more user-friendly, a continual goal on our roadmap. Keep an eye out for further updates!
We Want Your Feedback!¶
Your thoughts make Sidra better. For suggestions, issues, or questions, please reach out to us at [email protected].