[GRASS-dev] r.in.gdal fp precision loss

[Glynn Clements]

Markus Metz wrote:

> The range of the differences between the original and the re-import is
> 
> min=-6.10348170084762e-05
> max=6.10349170528934e-05
> 
> which is magnitudes larger than the 32 bit floating point precision limit.

No, that is the limit for values in that range.

The range of the source data is -6.68 .. 1952.0, so the largest values
are in the range [2^10,2^11).

Single-precision floating-point has a 24-bit significand, so values in
that range will have 13 bits to the right of the binary point,
resulting in the unit in the least place (ULP) having a magnitude of
2^-13 ~= 1.2207-04.

Rounding error is plus or minus half a ULP = 2^-14 ~= 6.1035-05.

	> import numpy as np
	> x = np.random.random_sample(1000000) * 1024 + 1024
	> x.min(), x.max()
	(1024.000187440173, 2047.9998948718983)
	> y = x.astype(np.float32)
	> d = y - x
	> d.min(), d.max()
	(-6.103512532718014e-05, 6.1035083945171209e-05)

-- 
Glynn Clements <glynn at gclements.plus.com>