[QGIS-Developer] Strange Behavior with QgsProcessingParameterNumber - Bug?

[Andrea Giudiceandrea]

C Hamilton wrote
> In the processing popup, if I enter a number with less than 20 digits then
> the rest of the digits become some random number. For example if I enter
> 
> -0.1556640799496235
> 
> the displayed result is
> 
> -0.15566407994962350170

AFAIK a Double-precision 64 bit floating-point (IEEE 754) can represent base
10 floating point numbers with about 15-17 significant decimal digits
precision. Floating-point numbers cannot exactly represent all the real
numbers in a range.

-0.1556640799496235 is represented as 0xBFC3ECCCF245D4FC as 64 bit
floating-point.
Anyway, the "most accurate representation" of 0xBFC3ECCCF245D4FC is
-0.155664079949623501697431038338

If you convert 0xBFC3ECCCF245D4FC to a base 10 real number rounding to 20
decimal digits you will display: -0.15566407994962350170

0xBFC3ECCCF245D4FB -> -0.15566407994962347
0xBFC3ECCCF245D4FC -> -0.15566407994962350
0xBFC3ECCCF245D4FD -> -0.15566407994962353


-0.15566407994962349 -> 0xBFC3ECCCF245D4FC
-0.15566407994962350 -> 0xBFC3ECCCF245D4FC
-0.15566407994962351 -> 0xBFC3ECCCF245D4FC

See:
https://en.wikipedia.org/wiki/Floating-point_arithmetic#Accuracy_problems
https://en.wikipedia.org/wiki/Double-precision_floating-point_format
https://www.binaryconvert.com/convert_double.html
https://babbage.cs.qc.cuny.edu/IEEE-754.old/64bit.html
https://www.h-schmidt.net/FloatConverter/IEEE754.html

So I think this explains the issue.

Regards.

Andrea




--
Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-Developer-f4099106.html