Sidra Data Platform (version 2023.R1: Nosy Northern Spy)¶
released on May 22, 2023
In this latest release, we have aimed at improving the user experience through Sidra’s web interface, highlighting self service replication from DSU to Data Product and UI driven Data Intake Processes edition and plugin upgrade.
Sidra 2023.R1 Release Overview¶
- Self service replication from DSU to Data Products
- UI driven Data Intake Process edition and plugin upgrade
- Incremental data ingestion for DB2 and DB2 iSeries connector
- Databricks and SQL Server securitization through Virtual Networks
- Integration Hub Service
What's new in Sidra 2023.R1¶
Self service replication from DSU to Data Products¶
As part of our strategic initiative to increase the self-service area of our product, Sidra's frontend now enables the user to set up the data replication from the DSU to the Data Products without having to leave the web UI, nor making changes to the database, Azure Application Service or Azure Data Factory (ADF). Some of the new capabilities are:
- Overview of the current Sync status and configuration, as well as options to enable/disable it and change its configuration.
- Data Product replication pipelines management.
- Entity-Pipeline association.
- Configuration of the
- Load process status information from ADF.
- New UI for reload of entities from a given point in time.
- An overview of the status of the last assets replicated to the client application.
This has been a major development at Sidra, so detailing all the capabilities is well outside the scope of these release notes. However, you can find more information at the documentation.
UI driven Data Intake Process edition and plugin upgrade¶
Starting in this release, the Data Intake Processes section of Sidra's UI features has included two new options:
Update, enabling the following:
- Editing the DIP parameters enables the user to change data sources, add or remove Entities to ingest, amend trigger information and any other parameter without having to leave Sidra's UI.
- Plugin versions updates whenever a more recent version is ready for use.
In order to benefit from the new edit and upgrade features in Sidra 1.14, the connector plugins need to be updated as well. At this moment only Azure SQL, SQL server, Oracle and DB2 plugins have these options available, but as upgrades move forward, the rest of the plugins will be updated to support these features. More information can be found at the documentation.
Incremental data ingestion for DB2 connectors¶
During the development of this latest release, DB2 and DB2 iSeries connectors of Sidra have been modified to include their built-in System-period temporal tables, DB2 native mechanism to allow incremental data ingestion.
Databricks and SQL Server securitization through Virtual Networks¶
Sidra is currently installed within a Virtual Network (VNET), which is one of the changes for this release. This means that a regular deployment would create the Web Apps as public, although the rest of resources is in the VNET. The main changes have been:
The securitization of Databricks and SQL Server resources deployed in VNETs.
Private endpoints can be deployed for storages accounts.
Find more information about this feature and Virtual Networks in the Sidra documentation.
Integration Hub Service¶
For this version, Sidra has included an Integration Hub Service that allows asynchronous message delivery between Sidra Core and Data Products. This service functions as a high-quality mechanism for data exchange between Core and Data Products operations, enhancing the applications' dependability and scalability.
Find more information about this feature on the Sidra documentation site.
Issues fixed in Sidra 2023.R1¶
The following list of solved issues is an example of the continuous development to which Sidra is committed:
- Fixed an endpoint to ensure retrieving the right containers only for a certain DSU. #3354
- Solved an issue with SharePoint plugin where
GetEntitiesactivity was failing due to an ambiguous name in
AdditionalPropertiescolumn generated by the query. #3642
- Solved an issue where SharePoint plugin did not filter by the correct list of containers assigned for the selected DSU. #3355
- Fixed an error getting the correct plugin version on a plugin controller endpoint. #3321
- Solved a problem where Esquio was not properly working from Sidra UI to enable/disable features. #86
- Added a new policy to avoid false success executions in pipelines activities after the execution of an Extract notebook. #3340
- Fixed an issue where the Sync was not taking into account loading an Asset which had failed more than three times in the load. #1169
- Fixed an issue that prevents start the cluster after updating the path for the initialization script. #1429
- Solved a problem where Sidra was not retrieving the logs from AppInsights due to the value for the key
- Solved an issue related with the execution of plugins when binaries are not available. #3136
- Solved an issue on Sidra Web where if the
Subjectwas configured using the name instead the email for
Namecolumn, the validation failed. #1457
- Fixed a problem when creating new Data Products on Sidra Web. #3343
- Fixed an issue when clicking on the
Available servicesnot displaying the proper information about the service on Sidra Web. #3380
- Solved a problem in the Data Catalog Providers screen which was not displaying the information when clicking on the third Provider. #3409
- Fixed a problem when showing roles in Data Products screen on Sidra Web. #3512
- Fixed a problem with edition of Long Description in Data Products configuration on Sidra Web. #1547
- Solved a problem where long Data Products name was no displayed properly. #1614
- Fixed an issue in the Data Catalog graph which was not filtering by Provider properly. #121485
- Fixed an issue in the Data Catalog graph which was not filtering by Data Storage Unit. #121486
- Fixed an issue whereby ADF Pipeline
RunIndexMaintenancewas throwing failures as false positives. #153600
- Fixed an issue in metadata search where more fields than
Descriptionwere used to search. #154551
- Fixed an issue in Data Intake graph in which it shown more days than expected when
Last 7 dayswas selected. #155325
- Fixed an issue in an activity template that did prevent to use Azure Data Factory user interface. #156247
- Fixed an issue in the Data Catalog page when there was no data preview for an Entity. #156611
- Fixed an issue in Sync WebJob that prevents synchronize entity that already exists but recently with permission over the Data Product. #156735
- Solved an issue in the order of the Attributes when schema evolution creates a new Attribute. #157436
- Fixed an issue in
FileIngestionDatabrickspipeline where the status of the Assets is not updated to final status. #157636
- Fixed an issue in the
Orchestratorpipeline for the data extraction in Core where it was not possible to parametrize the field that allows create Assets where there is not incremental data available. #157771
- Fixed an issue in the Data Catalog advanted search where using special characters cause the search fail. #158471
- Solved an issue in the Data Catalog Attribute edition. #159863
- Fixed an issue for Google Analytics plugin that it is not properly installed in Sidra 2022.R3 version. #160361
- Fixed an issue in the DSU Ingestion due to a conflict with an internal Attribute name. #160755
- Solved an issue in incremental intake pipelines when the record to get the next incremental data in the source data not longer exists. #162374
- Fixed an issue creating the table due to a modification in a python package about JWT deserialization. #163209
Breaking Changes in Sidra 2023.R1¶
DB2 connector with incremental load¶
Existing pipelines that rely on this connector should continue to function properly. However, the pipeline can automatically detect and perform incremental loads if the source database has this feature enabled, which may affect the pipeline behaviour.
This 2023.R1 release shows a remarkable improvement through the Sidra interface, automating tasks done before through Azure Data Factory, databases or Azure App Services and making easier any edition, update or management process related to the Data Product, plugins or Data Intake Process.
As part of the upcoming version, we intend to keep enhancing this interface experience with a Supervisor tool for self-service deployment.
We would love to hear from you! You can make a product suggestion, report an issue, ask questions, find answers and propose new features by reaching out to us in [email protected].