PipelineLoadRestrictionModeΒΆ
The PipelineLoadRestrictionMode, PipelineLoadRestriction and the PipelineLoadRestrictionObject tables are required for storing the object (e.g.tables, views, etc.) inclusions or exclusions to be considered when calling the metadata extraction pipeline for certain data sources, e.g. databases.
This section about the configuration for Data Intake Processes via connectors describes the general conceptual steps about the configuration of a new data source in Sidra. One of the required steps is to configure and create the metadata structures about the data source (Provider, Entities and Attributes).
In the case of databases, Sidra usually incorporates templates for metadata extraction pipelines, which are able to read into the schema of the source databases and create the Entity and Attributes metadata in Sidra Service metadata tables from the information obtained in that schema. The metadata extraction pipeline also includes as a parameter a list of objects to include or exclude.
These set of objects to include or exclude are called in Sidra LoadRestrictionObjects. There are two types of Load restriction rule types: inclusion and exclusion mode. These types are stored in the PipelineLoadRestrictionMode reference table:
| Column | Description | Format | Required |
|---|---|---|---|
| 1 | Include | tinyint | Yes |
| 2 | Exclude | varchar(50) | Yes |
When using Inclusion mode, the list of load restriction objects will be applied with an inclusion policy (just include the objects in the LoadRestrictionObject tables), or exclusion policy (load all objects, except the objects in the LoadRestrictionObject tables).