[fdo-trac] #642: Sqlite filtering on number with more that 5
decimal fails
FDO
trac_fdo at osgeo.org
Wed May 5 11:11:32 EDT 2010
#642: Sqlite filtering on number with more that 5 decimal fails
-----------------------------+----------------------------------------------
Reporter: bscott | Owner: traianstanev
Type: defect | Status: new
Priority: major | Milestone: 3.5.0
Component: SQLite Provider | Version: 3.5.0
Severity: 3 | Resolution:
Keywords: | External_id:
-----------------------------+----------------------------------------------
Comment (by traianstanev):
I'm asking because floating point is different on the two, on 64 bit, all
operations on double are always done to 64 bit, while on 32 bit, the
floating point unit has 80 bit precision which doesn't always get
truncated to 64 bits before comparisons are performed, depending on the
compiler optimization settings used. So those operations can give
different results on 32 vs 64.
--
Ticket URL: <http://trac.osgeo.org/fdo/ticket/642#comment:9>
FDO <http://fdo.osgeo.org/>
Feature Data Objects
More information about the fdo-trac
mailing list