Skip to content

Scenarios for metadata configuration

Below different example cases are described in order to clarify possible validation errors shown during the configuration of Entities and Attributes.

Case 1

  • Content file:

        Name,Code 
1   Afghanistan,AF 
2   Aland Islands,AX 
3   "",AL 
4   Algeria,DZ 
5   NN,AS 
6   Andorra,AD 
7   "   ",AO 
8   Anguilla,AI 
9   Antarctica,AQ 
10  Antigua and Barbuda,AG

  • Configuration options:

Option File Format IsNullable MaxLen NullText NeedTrim TreatEmptyAsNull Line numbers with validation error
1 CSV, XLSX, XLSM False 15 False False 10,3
2 XLS, Parquet False 15 10
3 CSV, XLSX, XLSM False 15 ‘NN’ False False 10,3,5
4 XLS, Parquet False 15 ‘NN’ False False 10,5
5 CSV, XLSX, XLSM False 15 True False 10,3
6 XLS, Parquet False 15 True False 10
7 CSV, XLSX, XLSM, Parquet False 15 False True 10,3
8 XLS False 15 False True 10
9 CSV, XLSX, XLSM, Parquet False 15 ‘NN’ False True 10,3,5
10 XLS False 15 ‘NN’ False True 10,5
11 CSV, XLSX, XLSM, Parquet False 15 True True 10,3,7
12 XLS False 15 True True 10,7

Where:

  • IsNullable, MaxLen, NeedTrim, and TreatEmptyAsNull columns belong to the Attribute table; and NullText column, belong to the Entity table.
  • Line 10 will always fail due to the MaxLen restriction.
  • Line 3 will always fail with a CSV, XLSX, XLSM files because, by default, is treated as NULL. For Parquet, only in case that TreatEmptyAsNull is True, is considered an error. For XLS files, there is not an error, but the value is NAN.
  • Line 5 fails when ‘NN’ is defined as NullText.
  • Line 7 fails when NeedTrim is applied and TreatEmptyAsNull is True.
  • If IsNullable is True, the only validation error shown will be for Line 10 because of the MaxLen restriction.

Case 2

  • Content file:

    Line    Field
1       -1
2       0
3       1
4       None
5       ""
6       "NN"
7       100
8       0.0
9       0.8

  • Configuration options:

Option File Format Field Type MaxLen IsNullable TreatEmptyAsNull NullText Line numbers with validation error
1 CSV, XLSX, XLSM, XLS INT 15 True True 6,9
2 CSV, XLSX, XLSM, XLS INT 15 True False 6,9
3 CSV, XLSX, XLSM, XLS INT 15 True True "NN" 9
4 CSV, XLSX, XLSM, XLS INT 15 False True 4,5,6,9
5 CSV, XLSX, XLSM, XLS INT 15 False False 4,5,6,9
6 CSV, XLSX, XLSM, XLS INT 15 False False "NN" 4,5,6,9
7 CSV, XLSX, XLSM, XLS INT 15 True True "NN" 7,9

Where:

  • IsNullable, MaxLen, and TreatEmptyAsNull columns belong to the Attribute table; and NullText column, belong to the Entity table.
  • Line 9 will always fail because it is not secure to convert to INT.
  • Line 7 fails if there is a restriction in the number of characters allowed for the field.
  • Line 6 fails if it is not NULL value because it is not convertible to INT, or if it is NULL value and the field it is not nullable.
  • Line 5 is only failing when NULL is not allowed, in the other case, when the value is read is converted to NULL.

Case 3

  • Content file:

    Line    Field
1       True
2       False
3       ok
4       notOk

Where:

  • If ReplacedText is ok and ReplacementText is True, the result data in the DSU for the previous file will be True for line 3.

Case 4

  • Content file:

        Name,Code
1   Bahamas,BS
2   "Bolivia, Plurinational State of",BO
3   Bosnia and Herzegovina,BA

Where:

  • If the configuration in ValidationText is "name_Processed LIKE 'Bo_%'", there will be an error for line 1 because the name is not starting with Bo. Note that the name of the field has a suffix _Processed because it is part of an internal process, and it is necessary to reference any source column.


Sidra Ideas Portal

Last update: 2022-09-29
Back to top