Data Quality Service on Sidra Web¶

Sidra Web¶

The Data Quality Service is installed via Supervisor and is powered by Great Expectations. Once installed, you can start using Data Quality rules for your data. You can configure these rules in the Data Catalog on the Entity detail page.

To configure new validations, go to the desired Entities for Data Quality and click on Definition and a screen will pop up with validation types and a field to include the JSON definition. The JSON configuration is depicted in the official Great Expectations documentation per type of validation and specified for Sidra here.

You can find a list of Expectations already configured in the Validation section. There is an option to download reports associated with each entity and asset. These reports will include the performance of that entity or asset against their Expectation.

To edit your expectations, click on any item in your expectations list and make your changes. Remember to click the Save button afterwards.

JSON Configuration example¶

Here is a JSON example for the validation type expect_column_max_to_be_between:

{
        "column": "column_name",
        "min_value": 10,
        "max_value": 50,
        "strict_min": false,
        "strict_max": true,
        "parse_strings_as_datetimes": false,
        "output_strftime_format": null
}