[QGIS-Developer] On precision in qgsRound() (3.3.0 master)

[andreaerdna]

qgsround is defined in qgis.h (since 3.0) as

 inline double qgsRound( double number, double places )
   {
    int scaleFactor = std::pow( 10, places );
    return static_cast<double>( static_cast<qlonglong>( number * scaleFactor
+ 0.5 ) ) / scaleFactor;
   }

and it seems only used by QgsUnitTypes:: scaledDistance and scaledArea which
are used in turn by QgsUnitTypes/QgsDistanceArea formatDistance/formatArea
and then by the homonymous ones of QgsMeasureDialog and QgsMapToolIdentify.

It seems to me that there is a flaw in qgsRound which leads to inconsistent
results of the Measure Tool in QGIS 3. (See the difference underlined in red
in the screenshot at
https://github.com/qgis/QGIS/pull/7836#issuecomment-419718554 between
segment length and total length values respectively rounded by
QLocale().toString() and by formatDistance through qgsRound()).

In fact, 'scaleFactor' is declared as Int and thus it can store up to the
9th power of 10, consequently the rounding provides incorrect results from
the 9th decimal place onwards.

Since it is allowed to set up to 12 as the number of decimal places in
Setting->Options->Map Tools (and moreover in Project Properties->General
there is no upper limit to the decimal places allowed to be set!), I think
'scaleFactor' should be qlonglong in order to correctly store at least the
12th power of 10 (while 'places' could be Int or less; and the
static_cast<qlonglong> could be <qulonglong>, provided that qgsRound() is
only used to round measurements, as it seems to be now).

In QGIS 2.18 this problem is not present, since both formatDistance and
formatArea rely on QString().arg(), which works similarly to
QLocale().toString(), to round measurements, instead of on qgsRound() (which
is also defined differently in 2.18 API).

I don't know if there could be a better approach to round numbers or how to
fix this flaw in a better way avoiding any overflow.

Hope this can help to improve QGIS 3.



-----
Andrea Giudiceandrea
--
Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-Developer-f4099106.html