Skip to content

How to associate an Entity with a pipeline in Data Intakes via landing zone

This tutorial only applies to Data Intakes Processes done via landing zone. You can see more information about the different types of Data Intake Processes in this page.

Data Factory pipelines can be defined in a JSON format. Using that JSON, they can be created programmatically using the Data Factory API.

There is a key component in Sidra, the Data Factory Manager, that composes the pipelines JSON from the templates and parameters stored in the Sidra Core metadata database and use it to create the pipeline in Data Factory.

In order to perform the parameter substitution in the templates, Data Factory Manager needs to know which pipeline to be launched for which Entity. This relationship is stored in the metadata database in the EntityPipeline table.

There are two ways to create a new Entity-Pipeline relationship:

  • Add a new Entity-Pipeline relationship by creating an SQL script to insert the relationship in the database.
  • Add a new Entity-Pipeline relationship by using the Sidra API endpoint to associate Entities and pipelines.

EntityPipeline general information

This is the information about an Entity-Pipeline relationship that must be included when it is added to the metadata database:

Column Description
IdPipeline [Required] Identifier of the pipeline
IdEntity [Required] Identifier of the related Entity


  • All the Entities must be associated to the FileIngestionDatabricks pipeline, since it will be used by all the Entities -no matter the method used to add their Assets to the platform- to ingest the raw copy of the Asset into Databricks.
  • Pipelines have a Globally Unique Identifier (GUID) that is unique above all the Sidra installations. This GUID is stored in the Pipeline table in the column ItemId. It is recommended to use the ItemId to get the Id of the pipeline.
  • The ItemId of the FileIngestionDatabricks is F8CD4AD9-DCAE-4463-9185-58292D14BE99.

Add Entity-Pipeline relationship using an SQL script

Add an SQL script following the naming conventions to the Scripts\CoreContext folder or to the place configured in the DatabaseBuilder from where it retrieves the scripts. A sample of the script to add a relationship between an Entity and a pipeline can be found below:

DECLARE @Id_Entity INT = 10 -- Id of the Entity  
DECLARE @Id_Pipeline INT =
    SELECT [Id]
    FROM [DataIngestion].[Pipeline]
    WHERE [ItemId] = 'F8CD4AD9-DCAE-4463-9185-58292D14BE99'

INSERT [DataIngestion].[EntityPipeline] 
     ([IdEntity],   [IdPipeline]) 
     (@Id_Entity,   @Id_Pipeline)

DELETE FROM [DataIngestion].[EntityPipeline]
WHERE [IdEntity] = @Id_Entity AND [IdPipeline] IN (@Id_Pipeline)

Add Entity-Pipeline relationship using Sidra API

Sidra API requires requests to be authenticated, the section How to use Sidra API explains how to create an authenticated requests. For the rest of the document, it is going to be supposed that Sidra API is deployed in the following URL:

Before creating an relationship between an Entity and a pipeline, it is required to know the Id of each of them.

Associate an Entity to a pipeline

The only required pieces if information are the Ids of the Entity and the pipeline, and include them in the URL of the request:




    "idEntity": 10,
    "idPipeline": 5,
    "entityName": "MyNewEntityViaAPI",
    "pipelineName": "FileIngestionDatabricks",
    "pipelineTemplateItemId": "cfbe563a-9674-2c2b-1e6b-566a56352486",
    "pipelineTemplateName": "AutoGeneratedImportAsset"

Sidra Ideas Portal

Last update: 2022-11-10
Back to top