[GRASS-user] Precision of raster/vector float values

[Markus Metz]

On Fri, Aug 24, 2012 at 11:27 AM, Markus Metz
<markus.metz.giswork at gmail.com> wrote:
> On Fri, Aug 24, 2012 at 10:40 AM, Johannes Radinger
> <johannesradinger at gmail.com> wrote:
>> Hi,
>>> So I misunderstood something about IEEE precision, sorry about the
>>> confusion! Coming back to the original problem, it seems that it can
>>> be solved for DCELL by using ".15g" instead of ".10f", i.e. use the
>>> same formatting rules like e.g. r.what and d.what.rast. I tested and
>>> the values uploaded to the database are correct with ".15g" for DCELL.
>>>
>>> Markus M
>>
>> sounds good if the origiinal problem can be solved for DCELL. If I
>> understand you
>> correctly this means that v.what.rast needs to be slighly changed?
>> Do I need to do it myself or will there be an update in a new revision
>> for 6.4.3 and/or 6.5?
>
> In my tests, uploading e.g. "2.5e-5" instead of "0.000025" to
> attribute tables works, but I am not sure if it works with all db
> backends. I am also not sure if you will get the same value out after
> some processing because sometimes "%lf" is used which prints by
> default only 6 decimal places, I think.

"%lf" is used by sscanf and correct for double. Maybe everything is fine now?

Markus M

> There are various binary <->
> text conversions happening in the db library, and they do not seem to
> be consistent. That is, depending on what you are going to do with the
> attributes, it may or may not work. v.what and v.db.select should
> work, though. v.what.rast is now fixed in all branches, r52869-71.
>
> Markus M