[GRASS-user] Precision of raster/vector float values

[Johannes Radinger]

On Fri, Aug 24, 2012 at 11:53 AM, Markus Metz
<markus.metz.giswork at gmail.com> wrote:
> On Fri, Aug 24, 2012 at 11:27 AM, Markus Metz
> <markus.metz.giswork at gmail.com> wrote:
>> On Fri, Aug 24, 2012 at 10:40 AM, Johannes Radinger
>> <johannesradinger at gmail.com> wrote:
>>> Hi,
>>>> So I misunderstood something about IEEE precision, sorry about the
>>>> confusion! Coming back to the original problem, it seems that it can
>>>> be solved for DCELL by using ".15g" instead of ".10f", i.e. use the
>>>> same formatting rules like e.g. r.what and d.what.rast. I tested and
>>>> the values uploaded to the database are correct with ".15g" for DCELL.
>>>>
>>>> Markus M
>>>
>>> sounds good if the origiinal problem can be solved for DCELL. If I
>>> understand you
>>> correctly this means that v.what.rast needs to be slighly changed?
>>> Do I need to do it myself or will there be an update in a new revision
>>> for 6.4.3 and/or 6.5?
>>
>> In my tests, uploading e.g. "2.5e-5" instead of "0.000025" to
>> attribute tables works, but I am not sure if it works with all db
>> backends. I am also not sure if you will get the same value out after
>> some processing because sometimes "%lf" is used which prints by
>> default only 6 decimal places, I think.
>
> "%lf" is used by sscanf and correct for double. Maybe everything is fine now?
>
> Markus M
>
>> There are various binary <->
>> text conversions happening in the db library, and they do not seem to
>> be consistent. That is, depending on what you are going to do with the
>> attributes, it may or may not work. v.what and v.db.select should
>> work, though. v.what.rast is now fixed in all branches, r52869-71.
>>
>> Markus M

I also tried v.what.rast again and it seems that is works now perfectly.
Thank you for the fast responce and fixing the problem.

/johannes