[GRASS-user] Precision error in r.mapcalc

Glynn Clements glynn at gclements.plus.com
Thu Jul 22 02:28:48 EDT 2010


Jenny Turner wrote:

> I have a raster (FCELL ) that has a minimum value of 0.1312103122

The data is stored in binary floating point, and 0.1312103122 isn't
exactly representable in that format. The closest representable value
is 0.0010cb7fe (hex) = 0.13121031224727630615234375 (decimal).

FCELL (single-precision "float") has just under 7 significant digits
of precision. More precisely, it has 24 binary digits, resulting in an
"epsilon" of ~1.2e-7.

The next representable value below the above value is 0.0010cb7fc
(hex) = 0.1312102973461151123046875 (decimal), while the next value
above it is 0.0010cb800 (hex) = 0.1312103271484375 (decimal)

IOW, any value whose decimal expansion begins with 0.13121031 will
have the same binary representation, and thus all such values will be
indistinguishable.

> and If I
> try to have an image with just those values
> r.mapcalc "minim= if(image==0.1312103122,10,0)" I get an all Zero image.

This is to be expected. The decimal constant will be parsed as double
precision then converted to single precision. The result may not be
exactly equal to the value stored in the map.

In general, you shouldn't perform equality comparisions between
floating-point values unless you are certain that no rounding errors
can be introduced.

> Then I opened the minim metadata and i see that in comments I have:
> if(clustered.1 == 0.13121031, 10, 0)
> r.mapcalc has eliminated the last 2 digits.

Floating-point constants are printed with "%.8g", which exceeds the
precision of FCELL. Printing more digits would typically be pointless,
as it's highly unlikely that the data will actually accurate to a
relative error of 1e-7 in the first place.

> How can I avoid this to happen?

Replace the equality check with a range check, e.g.:

	if(abs(image-0.13121031) <= 1e-8,...

-- 
Glynn Clements <glynn at gclements.plus.com>


More information about the grass-user mailing list