Data Validation

Data Validation is used to automatically or manually repair sensor data stored in the VDV database according to user defined rules. The Data Validation is useful to remove sensor readings that are clearly out-of-range, like readings with high noise caused from electrical interference or other disturbances. The user can define a maximum and minimum value, which if exceeded the validation toolkit will modify the sensor value according to the validation rule.

How does it Work?

  • Set individual out-of-range limits for each individual sensor that is to be included in the data validation process.
  • The data validation process reads the sensor data as stored in the VDV database; this process runs every few minutes and checks latest data for out-of-range readings.
  • If out-of-range data is found, it is replaced according to the validation rule.  If a change is made, the process is logged.
  • The validation process can run Automatically or the operator may choose a sensor or a site as well as the time period to validate and then Manually start the validation process.

Validation methods

  • Linear Interpolation between the good sensor readings on either side of the out-of-range sensor reading.
  • Out of range sensor readings are replaced with NaN (not a number) which results in a gap on a trend line.
  • Replace the out-of-range sensor reading with the last known good sensor reading.

When VDV validates data it first checks if the value is within the legal data range. If it is not within the legal data limit then it checks if the number of consecutive out-of-range sensor data readings is lower or equal to the selected Gap Size; if true then the sensor data readings are replaced according to the validation methods.  The purpose of the Gap Size is to ensure that a faulty sensor which constantly returns out-of-range readings is not validated which would hinder the fault to be discovered.  A second purpose of the Gap Size is to hinder that actual true and ongoing high sensor readings are not treated as sensor faults.

Example of validation

In this example a sensor is returning sensor data as “23.15, -100, 24.21, 23.98, -6999, -6999, -6999, 24.37, 25.14, 26.73”.  The lower data limit for this sensor is set to -60 and the upper data limit is set to 500.

  • If the Gap Size is set to 4 then after the validation process the same sensor data would read: “23.15; -23.15; 24.21; 23.98; 23.98; 23.98; 23.98; 24.37; 25.14; 26.73;”.
  • If the Gap Size had been set to 2 then after the validation process the same sensor data would read: “23.15; 23.15; 24.21; 23.98; -6999; -6999; -6999; 24.37; 25.14; 26.73;”.

In order to clean out single -6999 sensor reading the lower data limit may be set to -6000 and gap size to 1.