[Gdal-dev] OGR returns wrong floating values for shapefiles (and integer as real, in error)

[Maciej Sieczka]

Roger Bivand wrote:
> On Tue, 24 Oct 2006, Maciej Sieczka wrote:

>> ogrinfo (and other apps based on OGR, eg. OpenEV, QGIS) returns wrong
>> floating point values querying my shapefiles. Eg.:
>>
>> $ ogrinfo -al -q streams.shp
>>
>> Layer name: streams
>> OGRFeature(streams):523
>>   CAT (Real) =         484
>>   LCAT (Real) =          73
>>   Z (Real) =      101.583309999999997
>>   Z_BREACH (Real) =      101.583309999999997
>>   Z_BREACH1 (Real) =      100.583309999999997
>>   LENGTH (Real) =        2.036246000000000
>>   LINESTRING (598549.144524969975464 5677309.376777020283043,598550.0
>> 5677311.224603090435266)
>>
>>
>> After opening the dbf in oocalc 2.03, I can see the values should
>> recpectively be:
>>
>> CAT	  484
>> LCAT	  73
>> Z	  101.58331
>> Z_BREACH  101.58331
>> Z_BREACH1 100.58331
>> LENGTH	  2.036246
>>
>> Why the spurious "09999999997" in case of Z, Z_BREACH, Z_BREACH1 in
>> OGR? Note that, interestingly, LENGTH is OK though.

> Not spurious, just two different decimal "views" of the same underlying 
> floating-point value, see e.g. David Goldberg (1991), What Every Computer 
> Scientist Should Know About Floating-Point Arithmetic, ACM Computing 
> Surveys, 23/1, 548, also available via 
> http://docs.sun.com/source/806-3568/ncg_goldberg.html.

<snip>

> so your first issue is simply that floating point numbers have fuzz,
> which the application may shave or not.

Why doesn't the OGR "shave the fuzz" then? That would make sense I
think - data easier to read and parse, less for the OGR to print, the
user not puzzled. If it shouldn't - then why?

>> Morevover, CAT and LCAT are not Real numbers. They are integer. Why
>> reported as real?

> That will depend on the functions reading the underlying DBF, I see both 
> all reals or a mixture in several shapefiles. It may be that when 
> "integer" precision is non-zero, it may be being taken as real?

> In R:
> 
> library(rgdal)
> ogrInfo("streams", "streams")$iteminfo
> 
> says:
> 
> $name
> [1] "CAT"       "LCAT"      "Z"         "Z_BREACH"  "Z_BREACH1" "LENGTH"   
> 
> $precision
> [1] 2 2 2 2 2 2

Hmm, this looks wrong (?) - my CAT and LCAT *do* have a zero precision.
So I'd expect them to be treated as integer. And the other 4 columns
have 15 decimal places precision, not 2. Open the dbf in a spreadsheet
to see that:

CAT,N,11,0
LCAT,N,11,0
Z,N,24,15
Z_BREACH,N,24,15
Z_BREACH1,N,24,15
LENGTH,N,24,15

> $length
> [1] 11 11 24 24 24 24

This seems correct.

Maciek