[GRASS-user] Precision of raster/vector float values

[Glynn Clements]

Markus Metz wrote:

> So I misunderstood something about IEEE precision, sorry about the
> confusion! Coming back to the original problem, it seems that it can
> be solved for DCELL by using ".15g" instead of ".10f",

For realiable binary-decimal-binary round-trip conversions, it needs
to be %.17g.

DBL_EPSILON is ~2.220e-16, meaning that the next numbers after 1.0 are
1.000000000000000222, 1.000000000000000444, etc.

The first few numbers after 1.0 at various precisions are:

%.20g			%.17g			%.16g
1.0000000000000000000	1.0000000000000000	1.000000000000000
1.0000000000000002220	1.0000000000000002	1.000000000000000
1.0000000000000004441	1.0000000000000004	1.000000000000000
1.0000000000000006661	1.0000000000000007	1.000000000000001
1.0000000000000008882	1.0000000000000009	1.000000000000001

Clearly, %.16g (or less) will result in distinct binary floating point
values having the same decimal representation, meaning that the
round-trip conversion cannot be reliable.

As the numbers get larger, so does the spacing between them (doubling
at 2.0, 4.0 and 8.0). The last few before 10.0 are:

%.20g			%.17g			%.16g
9.9999999999999911182	9.9999999999999911	9.999999999999991
9.9999999999999928946	9.9999999999999929	9.999999999999993
9.9999999999999946709	9.9999999999999947	9.999999999999995
9.9999999999999964473	9.9999999999999964	9.999999999999996
9.9999999999999982236	9.9999999999999982	9.999999999999998

So %.16g may be enough for numbers at the end of each power-of-ten
range (after a few doublings but before the decimal point shifts) but
not for those at the beginning.

The value of DBL_DIG is 15, which indicates the the first 15 digits
are always exact, while later digits may include quantisation errors. 
So for a decimal-binary-decimal round-trip, you can rely upon decimal
values of 15 digits or less being preserved exactly. At 16 digits,
double precision doesn't have sufficient accuracy to reliably preserve
the 16th digit (e.g. 9.999999999999994 will come out as
9.999999999999995).

-- 
Glynn Clements <glynn at gclements.plus.com>