[gdal-dev] Numeric accuracy

[Jan Heckman]

Hi Ari and all,

why not use machine epsilon <https://en.wikipedia.org/wiki/Machine_epsilon>?
somewhat over 1e-16 for double precision, or an even more tolerant
tolerance? Most cases of meter coordinates 0.001 is realistic.
Given projection errors, a bit more is also defensible.

Am I to simplistic?

On Sat, Nov 11, 2017 at 11:05 AM, Ari Jolma <ari.jolma at gmail.com> wrote:

> I have been bitten twice, once with ArcGIS and now with Rasdaman WCS, with
> numeric accuracy.
>
> I'm making a data request on the corner of the bounding box, let's say it
> has minimum X of 75042.7273594. I'm setting my minX to that value and I'm
> enforcing it to that value with MAX (this was introduced because of a case
> with ArcGIS). The I print that to the request with "%.18g" (it was "%.15g"
> earlier but I changes it to that because of ArcGIS) and the result is
> 75042.7273593999998, which is not good for Rasdaman, since it is formally
> less than 75042.7273594. Although, in gdb
>
> (gdb) p 75042.7273594 > 75042.727359399999
> $7 = false
>
> Any ideas how to detect/prevent these kinds of situations? Keep the checks
> and go back to "%.15g"? Not ok with ArcGIS.
>
> Ari
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20171111/dfa46b03/attachment.html>