[QGIS-Developer] Strange Behavior with QgsProcessingParameterNumber - Bug?

[javier]

Hi Calvin,


That is not related to QGIS, but to the way the CPU represents floating points numbers internally.

You can see the same behaviour in the Python console (or any Python interpreter, and surely almost any other programming language that uses doubles):

```

>>> v = -0.1556640799496235

>>> print(f'{v:.20f}')

-0.15566407994962350170

```

The reason those extra digits are not '0' is because doubles are not stored in base 10, and anything beyond their standard ~16 significant digits is approximated. For a more detailed explanation you can take a look at Python documentation, they have a clear explanation on the topic (https://docs.python.org/3/tutorial/floatingpoint.html).



If you really, really, really need 20 significant decimal places (that seems an extremely rare situation, see https://xkcd.com/2170/ ) you should be looking at your code really carefully, as you need a larger (in number of bits) than usual floating point type; I think that neither QGIS (or standard Python or C++) support it out of the box.

Regards,

Javier

________________________________
De: QGIS-Developer <qgis-developer-bounces at lists.osgeo.org> en nombre de C Hamilton <adenaculture at gmail.com>
Enviado: lunes, 15 de marzo de 2021 15:50:23
Para: qgis-developer
Asunto: [QGIS-Developer] Strange Behavior with QgsProcessingParameterNumber - Bug?

In a processing algorithm I have a double QgsProcessingParameterNumber where I have set the number of decimals metadata to be 20.

param = QgsProcessingParameterNumber(
            'NUMBER',
            tr('Number'),
            QgsProcessingParameterNumber.Double,
            defaultValue=0,
            optional=False)
        param.setMetadata({'widget_wrapper': { 'decimals': 20 }})
        self.addParameter(param)

In the processing popup, if I enter a number with less than 20 digits then the rest of the digits become some random number. For example if I enter

-0.1556640799496235

the displayed result is

-0.15566407994962350170

I would think that if it pads with extra digits to fill out to 20, the digits should be 0. Have I exceeded the number of acceptable decimal digits, or is this a bug?

Thanks,

Calvin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20210315/7bdb1ff8/attachment.html>