Skip to content

Sidra Data Platform (version 2023.R1: Nosy Northern Spy)

released on May 22, 2023

In this latest release, we have aimed at improving the user experience through Sidra’s web interface, highlighting self service replication from DSU to Data Product and UI driven Data Intake Processes edition and plugin upgrade.

Sidra 2023.R1 Release Overview

What's new in Sidra 2023.R1

Self service replication from DSU to Data Products

As part of our strategic initiative to increase the self-service area of our product, Sidra's frontend now enables the user to set up the data replication from the DSU to the Data Products without having to leave the web UI, nor making changes to the database, Azure Application Service or Azure Data Factory (ADF). Some of the new capabilities are:

  • Overview of the current Sync status and configuration, as well as options to enable/disable it and change its configuration.
  • Data Product replication pipelines management.
  • Entity-Pipeline association.
  • Configuration of the Sidra.StagingConfiguration table.
  • Load process status information from ADF.
  • New UI for reload of entities from a given point in time.
  • An overview of the status of the last assets replicated to the client application.

This has been a major development at Sidra, so detailing all the capabilities is well outside the scope of these release notes. However, you can find more information at the documentation.

UI driven Data Intake Process edition and plugin upgrade

Starting in this release, the Data Intake Processes section of Sidra's UI features has included two new options: Edit and Update, enabling the following:

  • Editing the DIP parameters enables the user to change data sources, add or remove Entities to ingest, amend trigger information and any other parameter without having to leave Sidra's UI.
  • Plugin versions updates whenever a more recent version is ready for use.

In order to benefit from the new edit and upgrade features in Sidra 1.14, the connector plugins need to be updated as well. At this moment only Azure SQL, SQL server, Oracle and DB2 plugins have these options available, but as upgrades move forward, the rest of the plugins will be updated to support these features. More information can be found at the documentation.

Incremental data ingestion for DB2 connectors

During the development of this latest release, DB2 and DB2 iSeries connectors of Sidra have been modified to include their built-in System-period temporal tables, DB2 native mechanism to allow incremental data ingestion.

See more details about this feature in the documentation for DB2 connectors and the incremental load documentation related to it.

Databricks and SQL Server securitization through Virtual Networks

Sidra is currently installed within a Virtual Network (VNet), which is one of the changes for this release. This means that a regular deployment would create the Web Apps as public, although the rest of resources is in the VNet. The main changes have been:

  • The securitization of Databricks and SQL Server resources deployed in VNets.

  • Private endpoints can be deployed for storages accounts.

Find more information about this feature and Virtual Networks in the Sidra documentation.

The VNet documentation for the latest version of Sidra can be found here.

Integration Hub Service

For this version, Sidra has included an Integration Hub Service that allows asynchronous message delivery between Sidra Core and Data Products. This service functions as a high-quality mechanism for data exchange between Core and Data Products operations, enhancing the applications' dependability and scalability.

Find more information about this feature on the Sidra documentation site.

Issues fixed in Sidra 2023.R1

The following list of solved issues is an example of the continuous development to which Sidra is committed:

  • Fixed an endpoint to ensure retrieving the right containers only for a certain DSU. #3354
  • Solved an issue with SharePoint plugin where GetEntities activity was failing due to an ambiguous name in AdditionalProperties column generated by the query. #3642
  • Solved an issue where SharePoint plugin did not filter by the correct list of containers assigned for the selected DSU. #3355
  • Fixed an error getting the correct plugin version on a plugin controller endpoint. #3321
  • Solved a problem where Esquio was not properly working from Sidra UI to enable/disable features. #86
  • Added a new policy to avoid false success executions in pipelines activities after the execution of an Extract notebook. #3340
  • Fixed an issue where the Sync was not taking into account loading an Asset which had failed more than three times in the load. #1169
  • Fixed an issue that prevents start the cluster after updating the path for the initialization script. #1429
  • Solved a problem where Sidra was not retrieving the logs from AppInsights due to the value for the key ApplicationInsights--InstrumentationKey. #3125
  • Solved an issue related with the execution of plugins when binaries are not available. #3136
  • Solved an issue on Sidra Web where if the Subject was configured using the name instead the email for Name column, the validation failed. #1457
  • Fixed a problem when creating new Data Products on Sidra Web. #3343
  • Fixed an issue when clicking on the Available services not displaying the proper information about the service on Sidra Web. #3380
  • Solved a problem in the Data Catalog Providers screen which was not displaying the information when clicking on the third Provider. #3409
  • Fixed a problem when showing roles in Data Products screen on Sidra Web. #3512
  • Fixed a problem with edition of Long Description in Data Products configuration on Sidra Web. #1547
  • Solved a problem where long Data Products name was no displayed properly. #1614
  • Fixed an issue in the Data Catalog graph which was not filtering by Provider properly. #121485
  • Fixed an issue in the Data Catalog graph which was not filtering by Data Storage Unit. #121486
  • Fixed an issue whereby ADF Pipeline RunIndexMaintenance was throwing failures as false positives. #153600
  • Fixed an issue in metadata search where more fields than Title and Description were used to search. #154551
  • Fixed an issue in Data Intake graph in which it shown more days than expected when Last 7 days was selected. #155325
  • Fixed an issue in an activity template that did prevent to use Azure Data Factory user interface. #156247
  • Fixed an issue in the Data Catalog page when there was no data preview for an Entity. #156611
  • Fixed an issue in Sync WebJob that prevents synchronize entity that already exists but recently with permission over the Data Product. #156735
  • Solved an issue in the order of the Attributes when schema evolution creates a new Attribute. #157436
  • Fixed an issue in FileIngestionDatabricks pipeline where the status of the Assets is not updated to final status. #157636
  • Fixed an issue in the Orchestrator pipeline for the data extraction in Core where it was not possible to parametrize the field that allows create Assets where there is not incremental data available. #157771
  • Fixed an issue in the Data Catalog advanced search where using special characters cause the search fail. #158471
  • Solved an issue in the Data Catalog Attribute edition. #159863
  • Fixed an issue for Google Analytics plugin that it is not properly installed in Sidra 2022.R3 version. #160361
  • Fixed an issue in the DSU Ingestion due to a conflict with an internal Attribute name. #160755
  • Solved an issue in incremental intake pipelines when the record to get the next incremental data in the source data not longer exists. #162374
  • Fixed an issue creating the table due to a modification in a python package about JWT deserialization. #163209

Breaking Changes in Sidra 2023.R1

DB2 connector with incremental load

Description

Existing pipelines that rely on this connector should continue to function properly. However, the pipeline can automatically detect and perform incremental loads if the source database has this feature enabled, which may affect the pipeline behavior.

Required Action

See DB2 connector configuration information in the documentation and other documentation pages related to it.

Coming soon...

This 2023.R1 release shows a remarkable improvement through the Sidra interface, automating tasks done before through Azure Data Factory, databases or Azure App Services and making easier any edition, update or management process related to the Data Product, plugins or Data Intake Process.

As part of the upcoming version, we intend to keep enhancing this interface experience with a Supervisor tool for self-service deployment.

Feedback

We would love to hear from you! You can make a product suggestion, report an issue, ask questions, find answers and propose new features by reaching out to us in [email protected].