How to associate an Entity with a pipeline in Data Intakes via landing zone¶
This tutorial only applies to Data Intakes Processes done via landing zone. You can see more information about the different types of Data Intake Processes in this page.
Data Factory pipelines can be defined in a JSON format. Using that JSON, they can be created programmatically using the Data Factory API.
There is a key component in Sidra, the Data Factory Manager, that composes the pipelines JSON from the templates and parameters stored in the Sidra Core metadata database and use it to create the pipeline in Data Factory.
In order to perform the parameter substitution in the templates, Data Factory Manager needs to know which pipeline to be launched for which Entity. This relationship is stored in the metadata database in the EntityPipeline table.
There are two ways to create a new Entity-Pipeline relationship:
- Add a new Entity-Pipeline relationship by creating an SQL script to insert the relationship in the database.
- Add a new Entity-Pipeline relationship by using the Sidra API endpoint to associate Entities and pipelines.
EntityPipeline general information¶
This is the information about an Entity-Pipeline relationship that must be included when it is added to the metadata database:
|IdPipeline||[Required] Identifier of the pipeline|
|IdEntity||[Required] Identifier of the related Entity|
- All the Entities must be associated to the
FileIngestionDatabrickspipeline, since it will be used by all the Entities -no matter the method used to add their Assets to the platform- to ingest the raw copy of the Asset into Databricks.
- Pipelines have a Globally Unique Identifier (GUID) that is unique above all the Sidra installations. This GUID is stored in the Pipeline table in the column
ItemId. It is recommended to use the
ItemIdto get the
Idof the pipeline.
ItemIdof the FileIngestionDatabricks is
Add Entity-Pipeline relationship using an SQL script¶
Add an SQL script following the naming conventions to the
Scripts\CoreContext folder or to the place configured in the DatabaseBuilder from where it retrieves the scripts. A sample of the script to add a relationship between an Entity and a pipeline can be found below:
-- DECLARE DECLARE @Id_Entity INT = 10 -- Id of the Entity DECLARE @Id_Pipeline INT = ( SELECT [Id] FROM [DataIngestion].[Pipeline] WHERE [ItemId] = 'F8CD4AD9-DCAE-4463-9185-58292D14BE99' ) -- SCRIPT INSERT [DataIngestion].[EntityPipeline] ([IdEntity], [IdPipeline]) VALUES (@Id_Entity, @Id_Pipeline) -- ROLLBACK DELETE FROM [DataIngestion].[EntityPipeline] WHERE [IdEntity] = @Id_Entity AND [IdPipeline] IN (@Id_Pipeline)
Add Entity-Pipeline relationship using Sidra API¶
Sidra API requires requests to be authenticated, the section How to use Sidra API explains how to create an authenticated requests. For the rest of the document, it is going to be supposed that Sidra API is deployed in the following URL:
Before creating an relationship between an Entity and a pipeline, it is required to know the Id of each of them.
Associate an Entity to a pipeline¶
The only required pieces if information are the Ids of the Entity and the pipeline, and include them in the URL of the request: