[Gdal-dev] OGR returns wrong floating values for shapefiles (and
integer as real, in error)
tutey at o2.pl
Tue Oct 24 06:45:39 EDT 2006
Roger Bivand wrote:
> On Tue, 24 Oct 2006, Maciej Sieczka wrote:
>> ogrinfo (and other apps based on OGR, eg. OpenEV, QGIS) returns wrong
>> floating point values querying my shapefiles. Eg.:
>> $ ogrinfo -al -q streams.shp
>> Layer name: streams
>> CAT (Real) = 484
>> LCAT (Real) = 73
>> Z (Real) = 101.583309999999997
>> Z_BREACH (Real) = 101.583309999999997
>> Z_BREACH1 (Real) = 100.583309999999997
>> LENGTH (Real) = 2.036246000000000
>> LINESTRING (598549.144524969975464 5677309.376777020283043,598550.0
>> After opening the dbf in oocalc 2.03, I can see the values should
>> recpectively be:
>> CAT 484
>> LCAT 73
>> Z 101.58331
>> Z_BREACH 101.58331
>> Z_BREACH1 100.58331
>> LENGTH 2.036246
>> Why the spurious "09999999997" in case of Z, Z_BREACH, Z_BREACH1 in
>> OGR? Note that, interestingly, LENGTH is OK though.
> Not spurious, just two different decimal "views" of the same underlying
> floating-point value, see e.g. David Goldberg (1991), What Every Computer
> Scientist Should Know About Floating-Point Arithmetic, ACM Computing
> Surveys, 23/1, 548, also available via
> so your first issue is simply that floating point numbers have fuzz,
> which the application may shave or not.
Why doesn't the OGR "shave the fuzz" then? That would make sense I
think - data easier to read and parse, less for the OGR to print, the
user not puzzled. If it shouldn't - then why?
>> Morevover, CAT and LCAT are not Real numbers. They are integer. Why
>> reported as real?
> That will depend on the functions reading the underlying DBF, I see both
> all reals or a mixture in several shapefiles. It may be that when
> "integer" precision is non-zero, it may be being taken as real?
> In R:
> ogrInfo("streams", "streams")$iteminfo
>  "CAT" "LCAT" "Z" "Z_BREACH" "Z_BREACH1" "LENGTH"
>  2 2 2 2 2 2
Hmm, this looks wrong (?) - my CAT and LCAT *do* have a zero precision.
So I'd expect them to be treated as integer. And the other 4 columns
have 15 decimal places precision, not 2. Open the dbf in a spreadsheet
to see that:
>  11 11 24 24 24 24
This seems correct.
More information about the Gdal-dev