How to add a new Provider

A Provider is a Sidra element that represents a logical collection of Entities.

One Provided can be related to a unique data source or several data sources, which together conform a logical unit.

Providers are stored in the Provider table in the Sidra Core metadata database.

There are two ways to create a new Provider:

  • Create a new Provider by creating an SQL script to insert the new Provider in the Sidra Core database.
  • Create a new Provider by using the Sidra API endpoint to add new Providers.

The method used will depend, for example, on having access to the source code of the solution -where the SQL scripts must be placed to be executed- or having to integrate with a third-party tool. In the latter case, the Sidra API is the recommended approach.

Provider information

This is the information about the Providers that must be included when a new Provider is stored in the metadata database:

Column Description
Id [Required] Provider identifier.
ItemId [Required] Provider GUID
ProviderName [Required] Name of the Provider. Spaces are not allowed and it is usually wrote in camel case, e.g. MyNewProvider.
DatabaseName [Required] Name of the database in the metastore of the Data Storage Unit (DSU). Spaces are not allowed and it is usually named following this convention: "dw_xyz", where "dw" stands for Data Warehouse.
Description [Optional] Description of the Provider.
Owner [Optional] Identification of the person responsible of the Provider, it can be a name, an email, etc.
IdDataStorageUnit [Required] Id of the DSU -previously called DataLake- where the Entities of the Provider will be stored.
ParentSecurityPath [Optional] The security path of the parent following the metadata model hierarchy.
Image [Optional] Image of the Provider to be displayed in the Data Catalog.
SecurityPath [Computed] The security path of the Provider.
CreationDate [Required] Date of creation of the Provider.
Detail [Optional] Detailed description (markdown allowed) of the Provider to be displayed in the Data Catalog.

Add a new Provider by using an SQL script

Add an SQL script following the naming conventions to the Scripts\CoreContext folder or to the place configured in the DatabaseBuilder from where it retrieves the scripts. A sample of the script can be found below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
-- DECLARE
DECLARE @Id_Provider INT = 
(
    SELECT COALESCE(MAX([Id]) +1, 1)
    FROM [DataIngestion].[Provider]     
)
DECLARE @Id_Dsu INT = 
(
    SELECT TOP 1 [Id] 
    FROM [DataIngestion].[DataLake] 
    WHERE [Name] = 'Name of the DSU where the entities of the provider will be stored'
)

-- ROLLBACK
DELETE FROM [DataIngestion].[Provider]
WHERE [Id] = @Id_Provider

-- SCRIPT
SET IDENTITY_INSERT [DataIngestion].[Provider] ON 
INSERT [DataIngestion].[Provider] (
    [Id],
    [ProviderName], 
    [DatabaseName], 
    [Description], 
    [Owner], 
    [IdDataStorageUnit],
    [ParentSecurityPath],
    [CreationDate]) 
VALUES (
    @Id_Provider,
    N'MyNewProvider', 
    N'dw_newprovider', 
    N'Description of MyNewProvider', 
    N'John Doe - jdoe@email.com', 
    @Id_Dsu,
    @Id_Dsu,
    GETUTCDATE())
SET IDENTITY_INSERT [DataIngestion].[Provider] OFF

Add a new Provider by using Sidra API

Sidra API requires requests to be authenticated, the section How to use Sidra API explains how to create an authenticated requests. For the rest of the document, it is going to be supposed that Sidra API is deployed in the following URL:

1
https://core-mycompany-dev-wst-api.azurewebsites.net

This is the sequence of requests requiered to create a new Provider.

Some of the requests are used to gather information about the Provider, if that information is already available, those requests will not be necessary.

Step 1. Get the identifier of the Data Storage Unit (DSU)

Before creating a Provider, it is required to know the Id of the DSU where the data of the Entities associated to the Provider will be stored.

If the identifier is already known, this step can be skipped. If not, this is the request retrieve all the DSUs:

Request

1
GET https://core-mycompany-dev-wst-api.azurewebsites.net/api/datastorageunit/datastorageunits?api-version=1.0

It will return an object with the list of DSUs, including their Ids.

Response

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
    "TotalItems": "{TOTAL NUMBER OF ITEMS}",
    "Items": [{
            "id": 1,
            "name": "MyDSU",
            "resourceGroupName": "{RESOURCE GROUP NAME}",
            "clusterName": "{CLUSTER NAME}",
            "idClusterType": "{ID CLUSTER TYPE}",
            "idLocation": "{LOCATION ID}",
        }
    ]
}

From the previous response, the Id of the DSU can be retrived and used in the next step to populate the field idDataStorageUnit.

Step 2. Create a new Provider

By using Sidra API some of the fields of the Provider are automatically populated -like Id, ParentSecurityPath or CreationDate-.

Once the rest of the information of the Provider is gathered, it can be created using this request:

Request

1
POST https://core-mycompany-dev-wst-api.azurewebsites.net/api/metadata/providers?api-version=1.0

And adding the following content as part of the body:

Request body

1
2
3
4
5
6
7
{
    "providerName": "MyNewProvider",
    "databaseName": "dw_newprovider",
    "owner": "John Doe - jdoe@email.com",
    "description": "Description of MyNewProvider",
    "idDataStorageUnit": 1
}

The response will return the Id of the Provider created, which can be used, for example, for adding a new Entity.

Response

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
{
    "id": 10,
    "itemId": "6C12DF27-2C38-4B0B-9ACD-DFEC2930ACDA",
    "tags": [{
            "name": "string"
        }
    ],
    "providerSize": 0,
    "dataStorageUnitId": 1,
    "creationDate": "2020-01-22T18:03:45.938Z",
    "owner": {
        "name": "John Doe - jdoe@email.com",
        "picture": "string"
    },
    "providerName": "MyNewProvider",
    "databaseName": "dw_newprovider",
    "description": "Description of MyNewProvider",
    "idDataStorageUnit": 1
}