[geos-devel] WKT Precision

[Martin Davis]

Is it possible the problem is the use of std:fixed ?  (Which is invoked if
the trim option = FALSE, which is the default).

Currently in WKTWriter.writeNumber there is this code (and the defaults
invoke fixed precision):

if(! trim) {
        ss << std::fixed;
    }
    ss << std::setprecision(decimalPlaces >= 0 ? decimalPlaces : 0) << d;

This results in the following (as noted on the GeoSwift issue)

POINT (-0.4225977 46.3406448). ==>. POINT (-0.4225977000000000
46.3406447999999997)

This carries too much precision, obviously.  I think it might be exposing
the IEEE-754 guard digits unnecessarily.  FP output is notoriously tricky,
and I suspect it's better to let C++ just do the right thing.

Also, running reducePrecision causes problems, again I suspect due to to
imprecise FP representation:

bin/geosop -a "Point (-0.4225977 46.3406448)" -f wkt reducePrecision 100
POINT (-0.4200000000000000 46.3400000000000034)

If the std::fixed setting is dropped, the output looks more reasonable:

POINT (-0.4225977 46.3406448). ==>. POINT (-0.4225977234 46.3406448)

Check that all input sig digits are shown:

POINT (-0.4225977234 46.3406448) ==> POINT (-0.4225977234 46.3406448)

Reduced precision displays as expected:
bin/geosop -a "Point (-0.4225977 46.3406448)" -f wkt reducePrecision 100
POINT (-0.42 46.34)


Is the "trim" option needed at all?

On Tue, Jan 5, 2021 at 3:41 PM Paul Ramsey <pramsey at cleverelephant.ca>
wrote:

>
> What do people think is the best practice for outputing WKT precision?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/geos-devel/attachments/20210106/d10b6ea9/attachment-0001.html>