Resolution as an uncertainty component

A display showing 5.3 °C tells you the true indication lies somewhere in 5.25–5.35 °C with, absent better information, equal probability. That is a rectangular distribution of half-width 0.05 °C, whose standard uncertainty is 0.05/√3 ≈ 0.029 °C.

Consequences

• A 0.1 °C-resolution logger cannot be calibrated to better than about ±0.06 °C at k = 2 regardless of the reference.
• A 0.01 °C display contributes only 0.003 °C, so other terms dominate.
• If the instrument truncates rather than rounds, the distribution is offset; a correction of d/2 with the same uncertainty is appropriate.

For loggers with a resolution coarser than their sensor's noise, the resolution term is the practical floor and the reason a tighter reference brings no benefit.

The arithmetic, once

A display of resolution d rounds the true indication to the nearest step, so the true value lies within ± d/2 of what is shown, with no reason to prefer any point in that interval. That is a rectangular distribution of half-width a = d/2, and the standard uncertainty of a rectangular distribution is a/√3. So u = d/(2√3) = 0.2887 d — call it 0.29 d and you will never be more than a fraction of a percent out.

The third column is a floor and nothing beats it. A 0.1 °C logger cannot be calibrated to better than about ± 0.06 °C at k = 2 no matter what reference thermometer is used, which is the single most useful thing to know before buying a more expensive standard.

Two cases where the formula changes

A display that truncates rather than rounds — showing 5.3 for anything from 5.30 to 5.39 — has a rectangular distribution of full width d offset to one side, not centred. The standard uncertainty is the same d/(2√3), but there is also a systematic offset of d/2 that should be corrected rather than absorbed. Manufacturers rarely document which behaviour their firmware has; a quick test against a slowly ramping reference will show it.

The other case is an instrument whose own noise is larger than its resolution — a 0.001 °C display flickering over three or four counts. There the display is not the limiting factor and the repeatability term already captures the variation; counting both is double-counting. The resolution term matters when the reading sits rock-steady on one value, which is precisely when it looks most trustworthy.

What to do about it

• Read the instrument at its finest available resolution. Many loggers display 0.1 °C but export 0.01 °C, and using the export removes the dominant term for free.
• Check whether the calibration was performed at the resolution the instrument will be used at. Calibrating from an export and using the display is inconsistent.
• Before specifying a reference for a job, work out the resolution floor of the unit under test. If it is 0.058 °C there is no case for a 0.005 °C reference.
• Where the rounding step is unavoidable and the measurement matters, consider a ramp method that locates the transition point between two display steps rather than reading a static value.