About Sidra plugins and Data Intake Processes¶
This article is aimed to explain basic concepts about the Data Intake Process and the connector plugins relationship.
Data Intake Process in Sidra is an abstraction concept that relates a set of configurations for intaking data from a given data source (e.g., SQL Server database, Sharepoint library, etc.), as well as all the related data extraction infrastructure generated for the data intake (e.g., metadata, trigger, data extraction pipelines). The purpose of the Data Intake Process then is making possible a end-to-end process fully configurable with the Sidra's UI for the data intake, unlike the Data Intake via landing or with indexing which requires previous configurations.
Sidra incorporates several types of plugins. For plugins to create Data Intake Processes, the
plugin type = connector.
When configuring and executing a new data intake for a new or existing data source system, several underlying steps are involved:
- On one hand, the necessary metadata and data governance structures are created in Sidra. This includes the creation of the data source that is specific to the engine and type of the source system (e.g., SQL Server, Azure SQL, DB2 database, etc.). This also includes the generation of the metadata.
- On the other hand, the actual data integration infrastructure elements (e.g., ADF pipelines and triggers) are created, configured and deployed.
Sidra API provides different API methods for creating the needed metadata structures to configure a new Data Intake Process. This includes creating the Provider, creating the Data Source and creating and deploying the pipelines. However, with the latest versions of Sidra there is a new mechanism in Sidra web, that allows the configuration of data intake for a set of source systems. Users can choose from a gallery of available connector plugins to configure a Data Intake Process. The plugin is transparently installed in Sidra Service, and a wizard is displayed to the user. Each wizard contains a set of configuration parameters, so the user can enter those parameters to submit (confirm) the creation of the respective Data Intake Process.
Just after providing the configuration parameters in the respective wizard form steps, such as connection string parameters, or trigger selection, all the orchestration to create the underlying Data Intake Process happens. Thanks to the Sidra Data Intake Process wizards, the creation of this Data Intake Process infrastructure is configured in less than five minutes. Once the Data Intake Process for the data source is up and running, all the underlying metadata (Sidra metadata) and infrastructure (Azure Data Factory objects) will be in place.
Configure a new data intake
More details and concepts on what happens when configuring a new data intake via connector plugins is described in detail in this page .
Under this documentation section there are different documentation pages for specific Sidra connector plugins being released in Sidra.
Sidra Web Management UI includes a list with all configured Data Intake Processes configured in the respective installation environment. For accessing the Data Intake Process list in Sidra Web, you need to access the section Data Intake. A list will display the different configured Data Intake Processes in the system, and a button to Add a new Data Intake Process.