About Sidra connectors and Data Intake Processes¶

This article is aimed to explain basic concepts about the Data Intake Process and the connector relationship.

Overview¶

Data Intake Process in Sidra is an abstraction concept that relates a set of configurations for ingesting data from a given data source (e.g., SQL Server database, etc.), as well as all the related data extraction infrastructure generated for the data intake (e.g., metadata, trigger, data extraction pipelines). The purpose of the Data Intake Process then is making possible an end-to-end process fully configurable with the Sidra's UI for the data intake, unlike the Data Intake via landing or with indexing which requires previous configurations.

When configuring and executing a new data intake for a new or existing data source system, several underlying steps are involved:

• On one hand, the necessary metadata and data governance structures are created in Sidra. This includes the creation of the data source that is specific to the engine and type of the source system (e.g., SQL Server, Azure SQL, DB2 database, etc.). This also includes the generation of the metadata.
• On the other hand, the actual data integration infrastructure elements (e.g., ADF pipelines and triggers) are created, configured and deployed.

Sidra API provides different API methods for creating the needed metadata structures to configure a new Data Intake Process. This includes creating the Provider, creating the Data Source and creating and deploying the pipelines. Sidra provides a mechanism that allows the configuration of data intake for a set of source systems through Sidra Web. Users can choose from a gallery of available connectors to configure a Data Intake Process. The connector is transparently installed in Sidra Service, and a wizard is displayed to the user. Each wizard contains a set of configuration parameters, so the user can enter those parameters to submit (confirm) the creation of the respective Data Intake Process.

Just after providing the configuration parameters in the respective wizard form steps, such as connection string parameters, or trigger selection, all the orchestration to create the underlying Data Intake Process happens. Thanks to the Sidra Data Intake Process wizards, the creation of this Data Intake Process infrastructure is configured in less than five minutes. Once the Data Intake Process for the data source is up and running, all the underlying metadata (Sidra metadata) and infrastructure (Azure Data Factory objects) will be in place.

Under this documentation section there are different documentation pages for specific Sidra connectors being released in Sidra.

Sidra Web Management UI includes a list with all configured Data Intake Processes configured in the respective installation environment. For accessing the Data Intake Process list in Sidra Web, you need to access the section Data Intake. A list will display the different configured Data Intake Processes in the system, and a button to Add a new Data Intake Process.

Sidra connectors¶

Sidra offers as data source connectors:

• Azure SQL connector

• SQL Server connector

• DB2 connector

• Oracle connector

• MySQL connector