[GRASS-dev] r.in.gdal fp precision loss

[Markus Metz]

On Fri, Apr 5, 2013 at 8:55 AM, Glynn Clements <glynn at gclements.plus.com> wrote:
>
> Markus Metz wrote:
>
>> The range of the differences between the original and the re-import is
>>
>> min=-6.10348170084762e-05
>> max=6.10349170528934e-05
>>
>> which is magnitudes larger than the 32 bit floating point precision limit.
>
> No, that is the limit for values in that range.
>
> The range of the source data is -6.68 .. 1952.0, so the largest values
> are in the range [2^10,2^11).
>
> Single-precision floating-point has a 24-bit significand, so values in
> that range will have 13 bits to the right of the binary point,
> resulting in the unit in the least place (ULP) having a magnitude of
> 2^-13 ~= 1.2207-04.
>
> Rounding error is plus or minus half a ULP = 2^-14 ~= 6.1035-05.

I think I understand. If the largest values are in the range of 2^6 -
2^7, the ULP would have a magnitude of 2^-17 ~= 7.629395e-06? I am
asking because lately I observed slightly corrupted vector coordinates
in shapefiles created with recent ESRI software and suspect
float-double conversion artifacts. For latlon coordinates, the largest
values are in the range of 2^6 - 2^8, depending on the longitude.
v.in.ogr could then, based on the largest coordinate values, suggest a
threshold for snapping if topological errors are encountered.

Markus M