[postgis-devel] 1.3.4rc1

[Mark Cave-Ayland]

Mark Cave-Ayland wrote:

> Okay, I've finally found a few moments to test on Win32. Compilation goes
> fine, but regression blows up big style on the geojson test for which I've
> attached the diff :(  It seems to be precision related, but there must be
> something different in the code since the problem doesn't occur at all in
> the rest of the tests.

I've spent some time today looking at this and the problem is a symptom 
of different floating point precision(s) and the use of the %f format 
specifier.

The first thing to note is that the differences in floating point 
implementations will always be present, no matter what we try and do. 
The trick, of course, is to restrict our regression tests to the lowest 
common precision denominator that will pass on all platforms.

Currently the geojson code format identifiers look like this: %.*f where 
the * is used to indicate the number of decimal places. Now 
unfortunately this is incorrect because the precision of the floating 
point answer is related to the size of the mantissa (and hence the 
number of significant figures) rather than the number of decimal places. 
  So depending upon whether the calculation is done in hardware or 
software, the compiler options and even program order, the resulting 
output is likely to be different.

The double-precision IEEE floating point format specifies a 64-bit 
number with a 53-bit mantissa, which translates into approximately 14 
significant figures. So in order to solve this, I'd like to propose 
changing the meaning of the precision parameter from decimal places to 
significant figures, and changing the geojson output formats to %*f with 
a maximum precision of 14.

In my testing here, this gives identical results between gcc and MingW 
for 99.9% of cases, with the small 0.1% being caused by MingW adding a 
trailing "0" to the output when rounding in some cases. I would suggest 
that if there are cases like this in the regression tests after 
conversion then we simply alter the input coordinates to prevent this 
from happening.

So I'd like to go ahead and do this to 1.3 branch / trunk so that we can 
cleanly pass regression tests on at least gcc/msvcrt in order to get 
1.3.4 out of the door. Anyone have any major objections?


ATB,

Mark.

-- 
Mark Cave-Ayland
Sirius Corporation - The Open Source Experts
http://www.siriusit.co.uk
T: +44 870 608 0063