[GRASS-user] Precision of raster/vector float values

[Glynn Clements]

Markus Metz wrote:

> >> So I misunderstood something about IEEE precision, sorry about the
> >> confusion! Coming back to the original problem, it seems that it can
> >> be solved for DCELL by using ".15g" instead of ".10f",
> >
> > For realiable binary-decimal-binary round-trip conversions, it needs
> > to be %.17g.
> 
> Accordingly with single precision, for the binary-decimal-binary
> round-trip it needs to be %.9g IIUC.

AFAICT, that's correct.

> With regard to ticket #335 where it was decided to use %.7g and %.15g
> for FCELL and DCELL, respectively, that should be changed to %.9g and
> %.17g in cases where a binary-decimal-binary round-trip would take
> place?

Yes.

> Raster values and vector attributes can be anything where very
> large and very small numbers can be perfectly legal.

The magnitude of the values doesn't matter, so long as you're using %g
or %e rather than %f.

-- 
Glynn Clements <glynn at gclements.plus.com>