[GRASS-user] Precision of raster/vector float values

Markus Metz markus.metz.giswork at gmail.com
Sun Aug 26 01:26:45 PDT 2012


On Sun, Aug 26, 2012 at 12:19 AM, Glynn Clements
<glynn at gclements.plus.com> wrote:
>
> Markus Metz wrote:
>
>> So I misunderstood something about IEEE precision, sorry about the
>> confusion! Coming back to the original problem, it seems that it can
>> be solved for DCELL by using ".15g" instead of ".10f",
>
> For realiable binary-decimal-binary round-trip conversions, it needs
> to be %.17g.

Accordingly with single precision, for the binary-decimal-binary
round-trip it needs to be %.9g IIUC.

With regard to ticket #335 where it was decided to use %.7g and %.15g
for FCELL and DCELL, respectively, that should be changed to %.9g and
%.17g in cases where a binary-decimal-binary round-trip would take
place? Raster values and vector attributes can be anything where very
large and very small numbers can be perfectly legal.

With regard to coordinates, in particular window settings, it would be
nice if the binary values calculated by [G|Rast]_set_window() could be
preserved, i.e. the round trip of writing binaries as text to WIND
(and to cellhd/<raster>), then reading them back in would result in
identical values (irrespective of any accuracy considerations).

>
> DBL_EPSILON is ~2.220e-16, meaning that the next numbers after 1.0 are
> 1.000000000000000222, 1.000000000000000444, etc.
>
> The first few numbers after 1.0 at various precisions are:
>
> %.20g                   %.17g                   %.16g
> 1.0000000000000000000   1.0000000000000000      1.000000000000000
> 1.0000000000000002220   1.0000000000000002      1.000000000000000
> 1.0000000000000004441   1.0000000000000004      1.000000000000000
> 1.0000000000000006661   1.0000000000000007      1.000000000000001
> 1.0000000000000008882   1.0000000000000009      1.000000000000001
>
> Clearly, %.16g (or less) will result in distinct binary floating point
> values having the same decimal representation, meaning that the
> round-trip conversion cannot be reliable.
>
> As the numbers get larger, so does the spacing between them (doubling
> at 2.0, 4.0 and 8.0). The last few before 10.0 are:
>
> %.20g                   %.17g                   %.16g
> 9.9999999999999911182   9.9999999999999911      9.999999999999991
> 9.9999999999999928946   9.9999999999999929      9.999999999999993
> 9.9999999999999946709   9.9999999999999947      9.999999999999995
> 9.9999999999999964473   9.9999999999999964      9.999999999999996
> 9.9999999999999982236   9.9999999999999982      9.999999999999998
>
> So %.16g may be enough for numbers at the end of each power-of-ten
> range (after a few doublings but before the decimal point shifts) but
> not for those at the beginning.
>
> The value of DBL_DIG is 15, which indicates the the first 15 digits
> are always exact, while later digits may include quantisation errors.
> So for a decimal-binary-decimal round-trip, you can rely upon decimal
> values of 15 digits or less being preserved exactly. At 16 digits,
> double precision doesn't have sufficient accuracy to reliably preserve
> the 16th digit (e.g. 9.999999999999994 will come out as
> 9.999999999999995).

Thanks for the detailed explanation.

Markus M


More information about the grass-user mailing list