[postgis-devel] [PostGIS] #703: measures test failure

[PostGIS]

#703: measures test failure
---------------------+------------------------------------------------------
 Reporter:  strk     |       Owner:  pramsey      
     Type:  defect   |      Status:  new          
 Priority:  medium   |   Milestone:  PostGIS 2.0.0
Component:  postgis  |     Version:  trunk        
 Keywords:           |  
---------------------+------------------------------------------------------

Comment(by nicklas):

 Ok, here is what I have found so far

 I have compared a 64 bit ubuntu 10.4 installation with a 32 bit 10.10
 installation on different hardware.

 Both is PostgreSQL 9.01

 The line numbers refere to revision 6238
  the query I have used to compare is.


 {{{
 select  st_astext(st_shortestline(a,b)) from (
         select geomfromtext('LINESTRING(-40 -20 ,10 5)') as a,
                 geomfromtext('LINESTRING(-10 20, 1 -2)') as b
         ) as foo;
 }}}

 Both systems seems to store a slightly wrong value in variable r on line
 882 in measures.c
 r=8.0000000000000004e-1 instead of 0.8

 Then, the difference between the systems occur on line 914 and 915
 On the 32 bit system
 A->x+r*(B->x-A->x) gives 2.22044607925031308e-15
 and the 64 bit system gives
 0.00000000000000000e+00[[BR]]


 A->x and B->x seems to be identical.

 So both systems get a floating point error when saving variable r but on
 64 bit system that error doesn't seem to be reflected in the result. It
 disappears in the next calculation.

 To me it seems like 2.22044607925031308e-15 is to small to handle in one
 system but possible to handle in the other. Or is it just rounded by some
 other reason.

 I will try to do the same outside postgis/postgresql in a standalone
 program to see if the differnce seems to be in the system itself. But I
 have no time right now.


 /Nicklas

-- 
Ticket URL: <http://trac.osgeo.org/postgis/ticket/703#comment:6>
PostGIS <http://trac.osgeo.org/postgis/>
The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project.