Sidra API module: Management

This module contains endpoints for retrieving information about the Sidra installation: notifications, logs, configuration values, etc.

App

It allows to retrieve the list of Client Applications, also known as Apps, registered in the installation. More information about Client Apps can be found in this section.

Configuration

It contains CRUD endpoints to Configuration table. More information about management tables can be found in this section.

DataFactoryMetricsLog

It allows to retrieve the logs of Data Factory pipelines executions applying different sets of filters. The logs are retrieved from the Util.DataFactoryPipelineRun table in the Log database in Core.

DataFactoryPipelineRun

Log database stores a copy of the logs (traces) of the platform, by applying a set of filters.

This table stores information about pipeline executions in Azure Data Factory.

Name Description
LogDate Timestamp of log message.
RunId Run Id of Data Factory pipeline.
RunGroupId Run Group Id in Data Factory. Identifier that correlates all the recovery runs of a pipeline run.
IsLatest Flag indicating whether the pipeline run is the latest in its group.
PipelineName Name of the Pipeline.
LastUpdated The last updated timestamp for the pipeline run event.
RunStart Timestamp of start of pipeline run.
RunEnd Timestamp of end of pipeline run.
DurationInMs Total duration in ms of pipeline run.
Status Pipeline run status.
Message The message from a pipeline run.

You can see more detail information in Azure Data Factory help pages.

DataFactoryActivityRun

This table stores information about activity executions in Azure Data Factory.

Name Description
LogDate Timestamp of log message.
PipelineName Name of the pipeline to which this activity belongs.
PipelineRunId Run Id of the pipeline to which this activity belongs.
ActivityName Name of the activity.
ActivityRunId Id of the activity run.
ActivityStatus Status of the activity run.
ActivityRunStart The start time of activity run.
ActivityRunEnd The end time of activity run.
ActivityDuration The duration of activity run in ms.
PipelineDuration Total duration in ms of pipeline run.
ActivityError Error message of activity run.

You can see more detail information in Azure Data Factory help pages.

Image

It allows to retrieve the image associated to a provider, entity or app by its identifier. The valid values for the type are: entity, provider, app.

Logs

It allows to retrieve the logs of the platform applying different sets of filters. The logs are retrieved from the Util.Log table in the Log database in Core. The severity is the value stored in the Level field. It is an open field but usual values are: Error, Warning, Fatal, Information and Debug.

The fields that can be seen in the Log table are the following ones: TimeStamp, Message, MessageTemplate, Level, Exception, LogEvent. This last field includes the whole log object in JSON format.

The field Level is a severity indicator level, according to the standard development practices: Error, Warning, Fatal, Information, Debug. The table Log includes the traces generated by Core modules as well as Client Applications. Besides the Util.Log table, the table Util.DataFactoryPipelineRun stores information about Azure Data Factory pipelines execution, with information such as: RunId, name of the pipeline, last modification date, status and start and end execution timestamps. Sidra web also counts with a page to see the last logs as retrieved from this table.

Notifications

Besides the Logs table, Sidra incorporates a centralized management of notifications. Notifications are raised by operational and configuration processes at Sidra. If raised by configuration actions,usually, these consist of asynchronous unattended actions, such as the plugins execution (e.g., data intake process creation through connectors wizard). Notifications are also raised by the execution of data extraction and movement. Internally, the Azure service used to generate and manage these notifications is SignalR. Sidra notifications are stored in a table in Sidra Core, inside Notifications schema. These tables contain information about the notifications generated by the different platform modules, such as General, Intake and App.

The fields of a notification include description, creation date, type, sub-type and a JSON field with all this information. Sidra web also incorporates a page to see all the notification details, and to mark them as read. The Notification tables also store which notifications are read.

The actions allowed through the support of notifications tables are:

  • Get the number of notifications that are pending to read of each type: all, general, intake, app.
  • Get the notifications pending to read of a specific type.
  • Mark a set of notifications as read by theirs ids.

These are the available notification types:

Id NotificationType
1 General
2 Intake
3 App

ReleaseNotes

It allows to retrieve the release notes of all the Sidra versions and the version number of the installed one.

Service

It allows to retrieve generic information about the status of the installation:

  • Get the list of clusters, theirs type and status.
  • Get the list of all the services provided by Sidra platform an which ones are actived in the installation.
  • Get the list of DSUs configured in the installation.
  • Get the latest errors. Errors are entries of the Util.Log table with the Level field setup to Error.
  • Get the latest warnings. Warnings are entries of the Util.Log table with the Level field setup to Warning.
  • Get the number of entries in the Util.Log table filtered by a severity -which means a specific value in the Level field- an grouped by severity.
  • Get the number of errors filtered by a minimun id.
  • Get the number of warnings filtered by a minimun id.
  • Get a list of the selected measures about the installation with the type of measure it is.
  • Get a list of the selected measures during the selected dates.

These are the available measures:

Id MeasureName
0 TotalApps
1 StorageVolume
2 LastDayVolume
3 TotalEntities
4 TotalRows
5 TotalStreamingSources
6 TotalAssets
7 TotalDataLakeRegions
8 TotalProviders

These are the available measure types:

Id MeasureUnit
0 Bytes
1 Count
2 Seconds
3 Date
4 Text
5 Bool

These are the available measures for the selection using dates:

Id DatedMeasureName
0 ValidationErrors
1 LoadVolume

The information provided by these endpoints is used in the dashboard section of the Sidra Manager website.