Skip to content

Sidra Client Applications main concepts

The Client Applications are the pieces of Sidra enabling to drive business cases. The Client Applications layer can be also described as a transformation and serving layer for Sidra Data Platform.

Client App overview

Some characteristics of the Client Apps are:

  • Client Applications (or Apps) are a set of Azure resources and code, enclosed in a Resource Group, which either access the data from one or multiple Data Storage Units via the secure APIs, or retrieve the data from the Data Lake, applying business transformations if needed.

  • Any actor that needs to access the data stored in the Data Lake for a specific business need is catalogued as a Client Application.

  • Client Apps consume the content from the Data Storage Units, it is mandatory then to know if new content has been ingested in order to extract it and incorporate to the Client Application storage.

  • The architecture allows for Client Applications to be built using any set of tools and components, so they can range from Power BI-enabled analytical workspaces to even full-blown Web apps.

  • All Client Applications must use the same security model, notifications and logging infrastructure, etc., via Sidra APIs.

  • This centralization and templatization of governance and transversal components is what makes Sidra Data Platform an accelerator for innovation on serving data use cases.

Note

In some places the terminology Consumer apps to refer to Client Applications may still be found. This term is now deprecated.

Overview of the Client Application creation and deployment processes

Sidra Client Application pipelines are designed to perform installation and deployment of the Client Applications in a highly automated way, following a Continuous Integration/Continuous Deployment (CI/CD) process.

The general process is as follows:

  • Step 1: We need to download locally the corresponding Client Application template.

  • Step 2: The source for the actual instance of the Client Application needs to be created from this template.

  • Step 3: Once the .NET solution has been obtained with its code projects, we need to push this code to a specific branch (depending on the environment) on an Azure DevOps repository. The repository and branches need to have been created previously if they did not exist yet.

  • Step 4: Finally, the solution will be deployed with CI/CD pipelines.
    1. For this, the Client Application generates a build+release definition in YAML format.
    2. This YAML definition will be used by Azure DevOps to configure the integration and deployment processes.
    3. A set of artifacts will be required for this, that may vary according to the type of Client Application template that is used. This artifacts contain mainly required parameters for creating and deploying the infrastructure of the Client Application in Azure. There are mainly two alternatives:
      1. Data configuration files (.psd1)
      2. Variable groups in DevOps, the recommended option for new templates creation, as this is compatible with the plugins approach for Client Applications.

Info

More details of the plugins approach in the Connectors documentation pages.

Step-by-step

Create a Client App from scratch

For more information, check the specific tutorial for creating a Client App .

Data movement and orchestration in Client Applications

The Client Applications solutions use Azure Data Factory V2 (ADF) to perform data movements -same as Sidra Core solutions-, but the deployment project creates its own instance of ADF in the Resource Group of the Client Application.

When working with both solutions at the same time -Core and Client Applications- it is important to differentiate the ADF instances. More specifically, there will be:

  • an ADF instance for each Data Storage Unit (DSU) in Core.
  • an ADF instance for each Client Application solution.

Job DataFactoryManager for Client Applications

ADF components (datasets, triggers and pipelines) in Client Applications are managed the same way than in Core, by means of the DataFactoryManager webjob.

The section Data Factory tables explains how it works in Core.

The DataFactoryManager job uses information stored in the metadata database to build and programmatically create the ADF components in Data Factory, which means that Client Applications need a metadata database to store the information about ADF components.

There are some minor differences between DataFactoryManager for Client Applications and for Core:

  • Core version includes the creation of the landing zones -creation of the Azure Storage containers- for the Data Storage Units.
  • The Pipeline table in Core can store ADF pipelines but also Azure Search pipelines. In Client Applications the Pipeline table only stores ADF pipelines.
  • DataFactoryManager in Client Applications has to filter the pipelines to create only those needed for ADF.

Understanding the synchronization between Sidra Core and Client Applications

Sidra provides templates to create Client Applications that can be configured to automatically retrieve the information from a Data Storage Unit (DSU) when it is available.

The process of discovering and extracting this information is called synchronization between a Client Application and Sidra Core.

Sidra provides a component that orchestrates this synchronization, the Sync webjob.

This job is deployed in all Client Applications with the template provided by Sidra. Without any additional configuration, the metadata in the Core database is already synchronized with the metadata in the Client Application database.

The synchronization will take place as long as the Client Application has the right permissions to synchronize with the data in the DSU. This is done by editing the Balea Authorization permissions tables (Users >Client Application Subject > Permissions).

The synchronization webjob is configured to be executed every 2 minutes.

How naming conventions for Client Application staging tables work

Sidra supports two different naming conventions for the Databricks and the Client Application staging tables:

  • The Sidra default naming convention. This naming convention is "databasename_schemaname_tablename". So, for example: for a table whose name is "table1" in the origin system, which is under a database "databaseA", and under a schema "schemaA", then the resulting name with this convention will be: "databaseA_schemaA_table1".
  • A custom naming convention through a couple of configuration parameters.

Please find below instructions on how to use these configuration parameters to specify this custom naming convention.

Steps of configuration for naming convention

Depending on the metadata scenario, different options of configuration are possible, as show below:

  1. For existing metadata that do not need to be updated (nor new metadata added): no action is required.

  2. For existing metadata that need to be updated or new metadata added, wanting to maintain the current names (so, not use the Sidra default naming convention): you need to enable the new Sidra custom naming convention.

    For this, you need to set the parameter EnableCustomTableNamePrefix to true, and also set the parameter CustomTableNamePrefix accordingly, with the prefix you want to use. The prefix is anything in the name that goes before the name of the table. For example:

    1. If your staging table name is "tableA", the prefix needs to be null.
    2. If your staging table is "schema_name.table_name_", then the prefix needs to be set to the placeholder "{SCHEMA_NAME}_".
    3. If your staging table is "database_name.table_name", then the prefix needs to be set to the placeholder "{DATABASE_NAME}_".

    Examples of use

    Independently of the value set to the parameter CustomTableNamePrefix, the names will be the new default naming convention provided by Sidra: "databasename_schemaname_tablename".

    Then you can use either a fixed value for the naming convention, or a placeholder:

    1. If using a fixed value to concatenate fixed prefixes to the actual table name, then you need to populate the field CustomTableNamePrefix with a string value. This will concat the prefix configured in customTableNamePrefix to the actual name of the table. For example:

      • If CustomTableNamePrefix is empty, the final name for the staging table of a table with name "table1" in origin, will be "table1".
      • If CustomTableNamePrefix="databaseA_", then the final name for the staging table of a table with name "table1" in origin, will be "databaseA_table1".
      • If CustomTableNamePrefix="schemaA_", then the final name for the staging table of a table with name "table1" in origin, will be "schemaA_table1".
    2. If using a placeholder value to concatenate a dynamic prefix to the actual table name, then you need to populate the field CustomTableNamePrefix with a placeholder string value. For example:

      • If CustomTableNamePrefix="{SCHEMA_NAME}_", then the final name for the staging table of a table in origin with name "table1, under schema "schemaA" will be: "schemaA_table1".
      • If CustomTableNamePrefix="{DATABASE_NAME}_", then the final name for the staging table of a table in origin with name "table" under the database "databaseA" will be: "databaseA_table1".
  3. For existing metadata that just need to be updated, you can also use the new Sidra default naming convention: "databasename_schemaname_tablename". In this case, some manual intervention is required:

    1. You need to update the stored procedures at the Client Application side, that reference these staging tables, as their names will have changed.
    2. You also need to consolidate Databricks tables (new Databricks tables will have been created with this naming convention after the naming convention change is applied; this means otherwise you would have different tables referencing the same origin data).

Identity Server token configuration

In order to copy the data from the Entities in the DSU, the Client Application needs to request a token to the Identity Server service. This token will only be valid for a restricted time frame. If the validity period of such token needs to be extended, the Identity Server database in the Core resource group allows to configure such setting. In order to do this, we need to increase the validity period of the tokens by doing the following:

Apply an UPDATE over the table [dbo].[Clients], extendign the value of the field [AccessTokenLifetime] for the respective Client Application. For example, to extend to 5 hours:

UPDATE [dbo].[Clients] SET [AccessTokenLifetime] = 18000 WHERE [ClientName] = 'ClientAppName'

where ClientAppName is the ClientName of the corresponding Client Application.


Sidra Ideas Portal


Last update: 2022-07-14
Back to top