Update connectors versions and edit Data Intake Processes¶

New features have been implemented in the Sidra UI in order to contemplate different scenarios regarding the DIPs update due to new connectors versions. So far, the connectors with these new features available in Sidra are the Azure SQL, SQL Server, Oracle, and DB2.

Thus, the options Edit, Update, Export configuration and Delete have been included in the Data Intake Processes section of the Sidra UI. As a general description:

• Edit. DIPs parameters can be now edited by this button. In the case that the data source has changed, for example, it is not necessary a new DIP creation using this new feature.
• Update. Connector versions are updated by this button with the newest version available.
• Delete. The Delete button removes the resources (Data Factory and Databricks resources) from the DIP creation and the DIP data.

Data Intake Processes, once configured by first time, are using the current version of the connector selected (to know more about connectors reference here). However, due the continuous development, the connector version can get outdated easily.

Check on your connector version¶

A way to check the connector version used in a specific DIP is by checking on the API endpoint related.

[
    {
        "availableUpdatePluginVersion":"4.0.41",
        "enabled": true,
        "editable": true,
        "versionGap": "Minor",
        ...
        "pluginVersion":"4.0.40",
        ...
    }
]

Where:

• When availableUpdatePluginVersion equals to null, there is not a new version available for updating. "Enabled" and "Editable" in this case, are values related to the installed connector, and "versionGap" is null.

• When availableUpdatePluginVersion equals to a version number, there is a new version available for updating:

  • "Enabled" and "Editable" then, are the values related to the connector version ready for updating ("enabled": true) and the DIP available for editing ("editable": true).
  • The DIP edition and connector version update options will be available whenever the values for "Enabled" and "Editable" are true.
  • If the "versionGap" has the value "Minor", there is a change in the last number of connector version available (4.0.XX); with the "Major" value, the change between versions can be in the first numbers of the version pointing out a breaking change in the connector (X.X.XX).

• The versionGap with the values Minor or Major will be necessary to update the connector.

Buttons¶

1. Edit¶

Edit DIP with connector version update¶

In the case that a newest connector version is available, this will appear as a mandatory step before the edition.

Afterwards, the parameters of the chosen DIP can be edited.

Edit DIP¶

On the other hand, if there is not a new version for that connector, by clicking on Edit the general page of the DIP creation will be opened. Clicking on Continue, the user will be in the old configuration for that DIP. All the parameters except DSU name or Provider name, will be available for edition.

The password for the database will be required again as a security factor. A Clear button has been eased to do a full parameters reset if desired.

In some cases, depending on the connector type and version, this button can be disabled unlike the Update button. Then, an info box with the reason will be shown by hovering on the button.

2. Update¶

This button will be enabled when there are available new versions of the connectors. By clicking on it, the next screen will be shown.

When the versionGap mentioned before has the value "Major", clicking on this button will open the "Edit" options due the possibility of new fields added during this new major version or possible changes in the values of the fields.

If this button appears as disabled, an info box with the reason will be shown by hovering on the button.

3. Delete¶

This button removes any DIP resources associated in ADF and a soft deletion in Sidra (Core).

Known issues¶

• When creating or editing Data Intake Processes, Number of Tables per Batch acts as a mandatory field in "Configure Metadata Extractor" step.
• When editing any DIP and returning to the page where the Data Intake Process list is displayed, a refresh of the page can be necessary in order to visualize the DIP's update.