[GRASS-dev] CELL/FCELL/DCELL [was: Re: r.mapcal rand() strangeness]

[Glynn Clements]

Helena Mitasova wrote:

> P.S. And regarding the requirement that more 16 digits are supported
> that got all of this started - where do you need it?

I don't think the issue was that a lot of precision was actually
required, just that the way that the limitation manifests itself
appears strange if you are unfamiliar with the issues. Primarily, the
fact that once you exceed the precision, you get a string of seemingly
random digits.

BTW, there is an actual bug lurking here, IMHO. Any code which
displays arbitrary floating-point values should probably use the %g
specifier rather than %f.

%g will automatically switch to exponential notation if the magnitude
(decimal exponent) exceeds the precision (i.e. if the "units" digit is
subject to representational error), or if the value is smaller than
1e-4.

The net result is that %g will never print "random" digits unless you
provide an explicit precision specifier which exceeds the precision of
the data type (the default precision is 6 decimal places, i.e. "%g" is
equivalent to "%.6g", which is the limit for single precision).

OTOH, %f could print up to 308 digits before the decimal point for a
double, even though only the first 15 are meaningful.

In short, %f should only be used for values which will invariably be
relatively small (e.g. <= 1e6). Apart from the precision issues, once
you get past ~6 digits, you end up having to count the digits to
distinguish one million from 10 million.

If you know that the value represents something with a limited range
(e.g. slope, aspect, elevation), then %f is fine. But if it's
arbitrary (e.g. the range of a map, or some other quantity derived
from the values in an arbitrary map), then %g should be used.

Note that %g omits trailing zeros by default; you can use %#g to
retain them.

-- 
Glynn Clements <glynn at gclements.plus.com>