[fdo-trac] #642: Sqlite filtering on number with more that 5
decimal fails
FDO
trac_fdo at osgeo.org
Wed May 5 10:27:17 EDT 2010
#642: Sqlite filtering on number with more that 5 decimal fails
-----------------------------+----------------------------------------------
Reporter: bscott | Owner: traianstanev
Type: defect | Status: new
Priority: major | Milestone: 3.5.0
Component: SQLite Provider | Version: 3.5.0
Severity: 3 | Resolution:
Keywords: | External_id:
-----------------------------+----------------------------------------------
Comment (by traianstanev):
In general, you are not supposed to compare floating point numbers
directly. 1.000001 is an infinite fraction in binary, which means that it
could get rounded or truncated, depending on how the code was compiled. My
guess is that the binary representation stored in the database differs by
one bit from the number you use to compare due to different string to
double conversion code being used resulting in rounding/truncation
discrepancy.
--
Ticket URL: <http://trac.osgeo.org/fdo/ticket/642#comment:3>
FDO <http://fdo.osgeo.org/>
Feature Data Objects
More information about the fdo-trac
mailing list