[QGIS-Developer] Strange Behavior with QgsProcessingParameterNumber - Bug?

[C Hamilton]

I figured that something like that was probably causing this. No, I don't
need 20 digits. i was seeing data with 16 digits from KMLs.

Thanks,

Calvin

On Mon, Mar 15, 2021 at 11:54 AM javier <javier at panoimagen.com> wrote:

> Hi Calvin,
>
>
> That is not related to QGIS, but to the way the CPU represents floating
> points numbers internally.
>
> You can see the same behaviour in the Python console (or any Python
> interpreter, and surely almost any other programming language that
> uses doubles):
>
> ```
>
> >>> v = -0.1556640799496235
>
> >>> print(f'{v:.20f}')
>
> -0.15566407994962350170
>
> ```
>
> The reason those extra digits are not '0' is because doubles are not
> stored in base 10, and anything beyond their standard ~16 significant
> digits is approximated. For a more detailed explanation you can take a look
> at Python documentation, they have a clear explanation on the topic (
> https://docs.python.org/3/tutorial/floatingpoint.html).
>
>
>
> If you really, really, really need 20 significant decimal places (that
> seems an extremely rare situation, see https://xkcd.com/2170/ ) you
> should be looking at your code really carefully, as you need a larger (in
> number of bits) than usual floating point type; I think that neither QGIS
> (or standard Python or C++) support it out of the box.
>
>
> Regards,
>
> Javier
> ------------------------------
> *De:* QGIS-Developer <qgis-developer-bounces at lists.osgeo.org> en nombre
> de C Hamilton <adenaculture at gmail.com>
> *Enviado:* lunes, 15 de marzo de 2021 15:50:23
> *Para:* qgis-developer
> *Asunto:* [QGIS-Developer] Strange Behavior with
> QgsProcessingParameterNumber - Bug?
>
> In a processing algorithm I have a double QgsProcessingParameterNumber
> where I have set the number of decimals metadata to be 20.
>
> param = QgsProcessingParameterNumber(
>             'NUMBER',
>             tr('Number'),
>             QgsProcessingParameterNumber.Double,
>             defaultValue=0,
>             optional=False)
>         param.setMetadata({'widget_wrapper': { 'decimals': 20 }})
>         self.addParameter(param)
>
> In the processing popup, if I enter a number with less than 20 digits then
> the rest of the digits become some random number. For example if I enter
>
> -0.1556640799496235
>
> the displayed result is
>
> -0.15566407994962350170
>
> I would think that if it pads with extra digits to fill out to 20, the
> digits should be 0. Have I exceeded the number of acceptable decimal
> digits, or is this a bug?
>
> Thanks,
>
> Calvin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20210315/179556ed/attachment.html>