<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 11/11/2017 06:05 AM, Ari Jolma
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:67d2a0ab-8ba1-a519-1560-813ec4d57c48@gmail.com">I'm
making a data request on the corner of the bounding box, let's say
it has minimum X of 75042.7273594. I'm setting my minX to that
value and I'm enforcing it to that value with MAX (this was
introduced because of a case with ArcGIS). The I print that to the
request with "%.18g" (it was "%.15g" earlier but I changes it to
that because of ArcGIS) and the result is 75042.7273593999998,
which is not good for Rasdaman, since it is formally less than
75042.7273594. Although, in gdb
<br>
<br>
(gdb) p 75042.7273594 > 75042.727359399999
<br>
$7 = false
<br>
<br>
Any ideas how to detect/prevent these kinds of situations? Keep
the checks and go back to "%.15g"? Not ok with ArcGIS.
</blockquote>
All modern systems use IEEE 754 double-precision floating-point
representation. This has a mantissa size of 53 bits which
translates to a nominal decimal precision of 15.95 digits, but the
actual available precision wobbles between 15 and 17 decimal digits
depending on the number being represented. You will never be able
to get 18 "true" digits of precision.<br>
<br>
What I do in general to compensate for the kind of error you are
running into is generate the value with "%.16g" and then check the
15th and 16th generated digits (if they are actually generated) for
the pattern "99" or "01". If either pattern is present, I
re-generate the value with "%.15g". This generally gives you an
extra digit of precision without the problem of adding extraneous
"00001" or "99999" patterns to numbers that started out being
relatively round in decimal.<br>
<br>
<div class="moz-signature">-- <br>
Dr. Craig S. Bruce
<br>
Senior Software Developer
<br>
CubeWerx<i> </i>Inc.
<br>
<a href="http://www.cubewerx.com/">http://www.cubewerx.com</a></div>
</body>
</html>