Skip to content

Linked Services and Data Source

One of the key elements of ADF that need to be created and deployed in ADF besides pipelines is the Linked Service. They are used as a way to define a connection from the Data Factory to an external resource. There is no metadata about Linked Services in Sidra. Since they are very close to resources, the management of Linked Services is realized in the deployment project, just where the resources are also created.

Linked Services in Sidra are abstracted under the term Data Source. A Data Source in Sidra represents the connection to the source data system. A Data Source abstracts the details of creating a Linked Service in Azure Data Factory. Then Linked Services behave as connection strings, allowing Data Factory to connect to different data sources like SQL servers, REST endpoints, Blob folders, etc. through several Data Source templates provided by Sidra for common source systems as, e.g., "AzureSql", "HttpServer", etc.

The Linked Services creation is realized by using ARM templates. Sidra provides REST endpoints to register a new Data Source from a template, as well as to automatically deploy such infrastructure in Azure Data Factory for the pipelines to use. This is the recommended method for registering a Data Source in Sidra and therefore deploying a Linked Service in ADF. For some types of data sources, like SQL database, there is even a more automated and friendly way, that is using Connectors from the Web.

Configure a new Data Source

For configuring a new Data Source:

  1. The fields required are the Integration Runtime and the connection string in the case of a database.
  2. Sidra requires to register a new Data Source registry in Sidra metadata DB.
  3. Sidra metadata DB will trigger the deployment of a Linked Service in Azure Data Factory with this connection.
  4. The created Linked Service will use the Key Vault in Sidra Service to store the connection string to the database.

Step-by-step

Connecting to new data sources through linked services

For more details on the manual process of what constitutes the creation of a new linked service from a template, you can visit the page Connecting to new data sources through linked services .

Example of ARM templates for creation of Linked Services

Here it is a sample of two ARM templates that create two Linked Services for the Data Factory "MyDataFactory". One is a Linked Service to an Azure KeyVault "MyAzureKeyVaultLinkedService" and the other is to an Azure Storage "MyAzureStorageLinkedService".

{
    "type": "linkedservices",
    "name": "MyAzureKeyVaultLinkedService",
    "dependsOn": [
        "MyDataFactory"
    ],
    "apiVersion": "2017-09-01-preview",
    "properties": {
        "type": "AzureKeyVault",
        "typeProperties": {
            "baseUrl": "[concat('https://', 'myAzureKeyVault', '.vault.azure.net')]"
        }
    }
},
{
    "type": "linkedservices",
    "name": "MyAzureStorageLinkedService",
    "dependsOn": [
        "MyDataFactory", 
        "MyAzureKeyVaultLinkedService"
    ],
    "apiVersion": "2017-09-01-preview",
    "properties": {
        "type": "AzureStorage",
        "typeProperties": {
            "connectionString": {
                "type": "AzureKeyVaultSecret",
                "secretName": "MyAzureStorageLinkedService",
                "store": {
                    "referenceName": "MyAzureKeyVaultLinkedService",
                    "type": "LinkedServiceReference"
                }
            }
        }
    }
},

The names of the Linked Services are used to provide the values for the placeholders in the templates of datasets, activities... In particular, they are commonly used in the DefaultValue column of the ActivityTemplate and DatasetTemplate.

For example, the name of the Azure Storage Linked Service "MyAzureStorageLinkedService" can be used to replace the placeholder ##linkedService## in the LandingZoneFileDataset template:

{
    "name": "LandingZoneFileDataset",
    "properties": {
        "type": "AzureBlob",
        "linkedServiceName": {
            "referenceName": "MyAzureStorageLinkedService",
            "type": "LinkedServiceReference"
        },
        "typeProperties": {
            "folderPath": "@dataset().folderPath",
            "fileName": "@dataset().fileName",
            "format": {
                "type": "TextFormat",
                "rowDelimiter": "\n"
            }
        },
        "parameters": {
            "folderPath": {
                "type": "String",
                "defaultValue": ""
            },
            "fileName": {
                "type": "String",
                "defaultValue": ""
            }
        }
    }
}