[QGIS-trac] Re: [Quantum GIS] #3395: Field Calculator Integer Conversion Errror

[Quantum GIS]

#3395: Field Calculator Integer Conversion Errror
------------------------------------------------+---------------------------
   Reporter:  alexckp                           |              Owner:  jef          
       Type:  bug                               |             Status:  new          
   Priority:  major: does not work as expected  |          Milestone:  Version 1.7.0
  Component:  Vectors                           |            Version:  Trunk        
   Keywords:                                    |   Platform_version:               
   Platform:  All                               |           Must_fix:  No           
Status_info:  0                                 |  
------------------------------------------------+---------------------------
Changes (by aghisla):

  * platform:  Windows => All


Comment:

 I tried to reproduce the bug.

 If I set the type of the new field as integer, the formula outputs 3 and
 10 as expected.[[BR]]
 If I set it as decimal, then I get the 9.99999.. even if the formula tries
 to output an integer.

 Computing partial results in a new decimal field, with input value as
 65.1, gives this: [[BR]]
 SURV_BRG * 100 = 6510 [[BR]]
 to int(SURV_BRG ) = 65 [[BR]]
 to int(SURV_BRG * 100) = 6509 [[BR]]

 If affects Fedora, so changing platform to All.

-- 
Ticket URL: <https://trac.osgeo.org/qgis/ticket/3395#comment:1>
Quantum GIS <http://qgis.org>
Quantum GIS is an Open Source GIS viewer/editor supporting OGR, PostGIS, and GRASS formats