How to add a new Provider¶
A Provider is a Sidra element that represents a logical collection of Entities.
One Provider can be related to a unique data source or several data sources, which together conform a logical unit.
Providers are stored in the Provider table in the Sidra Core metadata database.
There are two ways to create a new Provider entry in Sidra:
- Preferred: Create a new Provider by using the Sidra API endpoint;
- Not recommended: Create a new Provider with a SQL script, to insert a new Provider entry in the Sidra Core database.
IMPORTANT
Some fields of the Provider will be linked to Azure resources; naming conventions and restrictions will apply. Therefore, using the Sidra API to create a new Provider is preferred, since the API will perform the needed validation checks, redusing the chances for errors.
Add a new Provider by using the Sidra API¶
For illustrative purposes, we will assume that the Sidra API is deployed at the following URL; the host-name follows the Sidra's naming conventions:
Sidra API requires requests to be authenticated: the section How to use Sidra API explains how to create an authenticated requests.
The API comes with the Swagger UI as a test ground for manual calls to the API. Swagger will ease the process authenticating the calls:
This is the sequence of requests required to create a new Provider:
Step 1. Get the identifier of the Data Storage Unit¶
To create a Provider, it is required to know the ID of the Data Storage Unit (DSU). The DSU will hold the data of the Entities associated with the Provider.
If the DSU identifier is already known, this step can be skipped. If not, the following request will retrieve all the DSUs:
Request¶
GET https://sidr-core-dev-wst-api.azurewebsites.net/api/datastorageunit/DataStorageUnits?api-version=1.0
It will return an object with the list of DSUs, including their IDs.
Response¶
{
"TotalItems": "{TOTAL NUMBER OF DSUs}",
"Items": [{
"id": 1,
"name": "MyDSU",
"resourceGroupName": "{RESOURCE GROUP NAME}",
"clusterName": "{CLUSTER NAME}",
"idClusterType": "{ID CLUSTER TYPE}",
"idLocation": "{LOCATION ID}",
"iconURL": "{URL ICON}",
"description" : "{DESCRIPTION}",
"detail" : "{DETAIL}"
}
]
}
From the previous response, the ID of the DSU can be retrived and used in the next step to populate the field idDataStorageUnit.
Step 2. Create a new Provider entry¶
To create a Provider entry, you will need information that needs to be gathered: see below details about all the fields of a Provider entry.
By using Sidra API, some of the fields of the Provider are automatically populated, like Id, ParentSecurityPath or CreationDate.
Once the rest of the information of the Provider is gathered, its entry can be created using this request.
IMPORTANT
For the DatabaseName field, lower letters, figures and spaces are allowed; no underscore (_) or minus (-) may be accepted.
Request¶
And adding the request payload content as required fileds of a Provider entry:
Request body¶
{
"providerName": "MyNewProvider",
"databaseName": "dwmynewprovider",
"owner": "John Doe - jdoe@email.com",
"description": "Description of MyNewProvider",
"idDataStorageUnit": 1
}
The response will return the ID of the Provider created, which can be used, for example, when adding a new Entity.
Response¶
{
"id": 10,
"itemId": "6c12df27-2c38-4b0b-9Acd-dfec2930acda",
"tags": [{
"name": "string"
}
],
"providerSize": 0,
"dataStorageUnitId": 1,
"creationDate": "2020-01-22T18:03:45.938Z",
"owner": {
"name": "John Doe - jdoe@email.com",
"picture": "string"
},
"providerName": "MyNewProvider",
"databaseName": "dwmynewprovider",
"description": "Description of MyNewProvider",
"idDataStorageUnit": 1
}
Add a new Provider by using an SQL script¶
IMPORTANT
This method of adding a Provider entry is not recommended, since no validation checks will be performed on the fields. Adding fields in a wrong format may result in errors in the subsequent steps, such as when adding a Data Source for the Provider.
A sample of the script can be found below.
It is recommended that the SQL script should be added - following the naming conventions - to the Scripts\CoreContext folder or to the place configured in the DatabaseBuilder (from where it retrieves the scripts). This would ensure that the Provider entry will be re-created when a new Sidra deployment pipeline run takes place.
-- PROVIDER CONFIGURATION
DECLARE @ProviderItemId UNIQUEIDENTIFIER = NEWID()
DECLARE @ProviderName VARCHAR(MAX)= 'MyNewProvider'
DECLARE @ProviderDatabaseName VARCHAR(MAX)= 'dwmynewprovider'
DECLARE @ProviderDescription VARCHAR(MAX) = 'Description of MyNewProvider'
DECLARE @ProviderOwner NVARCHAR(MAX) = N'jdoe@email.com'
DECLARE @IdDSU INT = '{ID DSU}'
-- PROVIDER CREATION
INSERT INTO [DataIngestion].[Provider]
([ItemId], [ProviderName], [DatabaseName], [Description], [Owner], [IdDataStorageUnit], [ParentSecurityPath],
[CreationDate])
VALUES
(@ProviderItemId, @ProviderName, @ProviderDatabaseName, @ProviderDescription, @ProviderOwner, @IdDSU, @IdDSU,
GETUTCDATE())
Provider fields¶
This is the information about the Providers that must be included when a new Provider is stored in the metadata database:
| Column | Description |
|---|---|
| Id | [Required] Provider identifier. |
| ProviderName | [Required] Name of the Provider. Spaces are not allowed and it is usually written in camel-case; e.g. "MyNewProvider". |
| DatabaseName | [Required] Name of the database in the metastore of the Data Storage Unit (DSU). Lower letters, figures and spaces are allowed; no underscore (_) or minus (-) may be accepted. An example: "dwmynewprovider"; where "dw" stands for Data Warehouse. |
| Description | [Optional] Description of the Provider. |
| Owner | [Optional] Identification of the person responsible of the Provide; it can be a name, an email, etc. |
| IdDataStorageUnit | [Required] Id of the DSU -previously called DataLake- where the Entities of the Provider will be stored. |
| ParentSecurityPath | [Optional] The security path of the parent following the metadata model hierarchy. |
| Image | [Optional] Image of the Provider to be displayed in the Data Catalog. |
| CreationDate | [Required] Date of creation of the Provider. |
| Detail | [Optional] Detailed description (markdown allowed) of the Provider to be displayed in the Data Catalog. |
| ItemId | [Required] Provider GUID |
| SecurityPath | [Computed] The security path of the Provider. |