Data-quality checks before analysing mapping data
Statistics computed on bad data are precise and wrong. Check the data first.
Before any minimum, maximum or mean is trusted, the dataset should pass a set of checks that catch export, merge and sensor problems.
Checks
- Unit: confirmed explicitly, never inferred from the magnitude of the values.
- Timestamps: monotonic per sensor, no duplicates, consistent interval, no time-zone or daylight-saving jumps.
- Completeness: gaps identified per sensor, with duration; a sensor missing a large fraction of the study cannot support conclusions.
- Flatlines: long runs of identical values suggest a stalled logger or a fault.
- Abnormal jumps: step changes larger than physically plausible in one interval.
- Oscillation: rapid alternation that suggests electrical noise or a failing sensor.
- Identifiers: every column mapped to a unique sensor with a location.
Each finding should be visible in the report with a decision: excluded, corrected (with a record), or accepted with a stated limitation. ValiTrac's quality stage produces exactly these findings with PASS / WARNING / REVIEW REQUIRED / CRITICAL states.
How each check is actually made
A checklist is only useful if each item has a test behind it. These are the thresholds that catch real export and logger faults without drowning the report in false findings.
| Problem | Test | Usual cause |
|---|---|---|
| Wrong unit | Ask; never infer. If the file says 41.9 it is 41.9 °F or 41.9 °C and the data cannot tell you | A logger configured in °F for a US-sourced shipper |
| Clock jump | Any negative or zero time step, or a step of exactly ± 3600 s | Daylight saving, or two loggers in different time zones |
| Gap | Any interval more than 1.5 × the nominal logging interval | Battery, radio dropout, or a truncated export |
| Flatline | More than ~12 consecutive readings identical to the display resolution | Stalled logger, or a probe that has come loose in a fixture |
| Impossible step | A change larger than the medium can physically produce in one interval | A logger read while being carried, or a merged file with mixed sensors |
| Noise / oscillation | Repeated sign changes with amplitude above the sensor noise floor | Electrical interference, a failing thermistor, loose wiring |
A worked case
A 12-sensor, 72-hour study at 5-minute logging should contain 10,368 readings. The export contains 10,201. That difference of 167 is not a rounding issue; it is roughly fourteen hours of a single sensor, or short gaps spread across several. Which of the two it is changes the conclusion completely: fourteen contiguous hours missing from the sensor at the warmest position may remove the only evidence of the hot spot, while 167 readings scattered one at a time across twelve sensors is a radio artefact with no effect on any statistic. Counting readings tells you something is wrong; locating them tells you whether it matters.
What to do with a finding
Every finding needs a decision recorded beside it, and there are only three honest ones.
- Excluded: the data are not trustworthy and are removed, with the reason and the exact range excluded written down. The study then has to be reassessed — a study that loses its hot-spot sensor may no longer meet its own sensor-count requirement.
- Corrected: a defined, documented transformation is applied, such as shifting a logger whose clock was set to the wrong time zone. Record what was changed, by how much, by whom and on what evidence. Never correct values themselves on the grounds that they look wrong.
- Accepted with a limitation: the data stay, and the report states plainly what the reader should not conclude from them.
The one option that is not available is silent repair. A dataset quietly cleaned before the statistics ran cannot be defended, because the person reading the report has no way to see what was removed or why.
Run the checks before the statistics, not after. A mean computed over a flatlined sensor is a precise number describing a fault, and it will look entirely reasonable on the report.
Frequently asked questions
- What is a flatline in logger data?
- A long run of identical readings that suggests a stalled logger, a disconnected probe or a sensor fault; the affected period cannot support conclusions about the space.
- How much missing data is acceptable in a mapping study?
- The protocol should say. A sensor missing a large fraction of the study, or missing data around events of interest, generally cannot be used for hot/cold-spot conclusions.
References
- [1]WHO Technical Supplement 8 to TRS 961 Annex 9: Temperature mapping of storage areas (2015)
General technical guidance written against the cited sources. It is not regulatory or legal advice and does not replace the applicable standard, guideline or a qualified reviewer's judgement.
Have a question on this topic?
Ask ValiTrac AI and see the evidence and calculation behind the answer.
Ask ValiTrac AIFree calculator: Sampling interval
See what a logger actually records from the same 24 hours at 1, 5, 15, 30 or 60 minute intervals, and watch a real excursion disappear.
Open the calculatorTemperature Mapping on ValiTrac
Analyse logger data and identify data-quality issues, statistics, hot/cold spots and excursions.
Explore mappingRelated articles
Common temperature mapping mistakes
Most failed studies fail before the loggers are placed — in the plan, the metadata or the calibration.
How to identify hot and cold spots
Hot and cold spots are defined by the data and the limits together — and the definition should be fixed before the analysis starts.
Furnace and oven temperature mapping
Mapping the hot side: why a survey runs at every set point, what a thermocouple costs you in uncertainty at 600 °C, and where the pass mark comes from.