[fdo-trac] #642: Sqlite filtering on number with more that 5 decimal fails

FDO trac_fdo at osgeo.org
Wed May 5 10:27:17 EDT 2010


#642: Sqlite filtering on number with more that 5 decimal fails
-----------------------------+----------------------------------------------
 Reporter:  bscott           |         Owner:  traianstanev
     Type:  defect           |        Status:  new         
 Priority:  major            |     Milestone:  3.5.0       
Component:  SQLite Provider  |       Version:  3.5.0       
 Severity:  3                |    Resolution:              
 Keywords:                   |   External_id:              
-----------------------------+----------------------------------------------
Comment (by traianstanev):

 In general, you are not supposed to compare floating point numbers
 directly. 1.000001 is an infinite fraction in binary, which means that it
 could get rounded or truncated, depending on how the code was compiled. My
 guess is that the binary representation stored in the database differs by
 one bit from the number you use to compare due to different string to
 double conversion code being used resulting in rounding/truncation
 discrepancy.

-- 
Ticket URL: <http://trac.osgeo.org/fdo/ticket/642#comment:3>
FDO <http://fdo.osgeo.org/>
Feature Data Objects


More information about the fdo-trac mailing list