[gdal-dev] interpretation of nodata in OGR libraries

[Dylan Beaudette]

Hi,

I am not sure if this is a DBF issue, or something deeper in OGR, but there 
does not appear to be a mechanism for preserving 'nodata' or NULL values when 
exporting from GRASS to vector formats through v.out.ogr.

Here is an example using the sample 'Spearfish' data within GRASS:

Spearfish example:

# make a copy of one of the sample vector files
g.copy vect=vegcover,veg_temp

# add a new double precision column
# should be filled with NULL
v.db.addcol veg_temp column="x double"

# check within GRASS:
db.select veg_temp

cat|label|x
1|irrigated agriculture|
2|rangeland|
3|coniferous forest|
4|deciduous forest|
5|mixed forest|
6|disturbed|

# check outside of GRASS:
dbfxtrct -iveg_temp.dbf

1,irrigated agriculture,********************
2,rangeland,********************
3,coniferous forest,********************
4,deciduous forest,********************
5,mixed forest,********************
6,disturbed,********************

# OK- it looks like there *is* some way to encode nodata into a DBF
file for numeric data
# looking at the GRASS description of the columns:

column:x
description:
type:DOUBLE PRECISION
len:20
scale:6
precision:18
default:
nullok:yes
select:yes
update:yes

# cleanup
g.remove vect=v2,veg_temp


## another example:
g.copy vect=archsites,a_temp
v.db.addcol a_temp column="xyz double, abc integer"

# check:
db.select a_temp

cat|str1|xyz|abc
1|Signature Rock||
2|No Name||
3|Canyon Station||
[...]

# check outside of GRASS:
dbfxtrct -ia_temp.dbf

1,Signature Rock,********************,***********
2,No Name,********************,***********
3,Canyon Station,********************,***********
4,Spearfish Creek,********************,***********
[...]

# looks like encoding nodata for 'double' is different than for 'integer'

# try exporting through GDAL/OGR:
# first as shapefile:
v.out.ogr in=a_temp dsn=. type=point

# check:
dbfxtrct -ia_temp.dbf

1,Signature Rock,0.000000000000000,0
2,No Name,0.000000000000000,0
3,Canyon Station,0.000000000000000,0
4,Spearfish Creek,0.000000000000000,0

## not as bad as junk data, but 0 is not NA...

# try another vector driver: MapInfo:
v.out.ogr in=a_temp dsn=. type=point format=MapInfo_File

OGRFeature(a_temp):1
  cat (Integer) = 1
  str1 (String) = Signature Rock
  xyz (Real) = 0
  abc (Integer) = 0

## still wrong, trying GML
v.out.ogr in=a_temp dsn=output_dir layer=a_temp type=point format=GML

## null columns aren't exported..., trying SQLite
v.out.ogr in=a_temp dsn=a_temp.db layer=a_temp type=point format=SQLite

ERROR 1: sqlite3_step() failed:
  SQL logic error or missing database
ERROR 1: sqlite3_step() failed:
  SQL logic error or missing database
[...]

## results in a broken sqlite DB, with no data, and an incomplete schema


It looks like GRASS can read in a DBF file with NULL data correctly. I 
created 'x.dbf' such that the last record was had only NULL values:

dbfxtrct -ix.dbf
1,1.967017974471673
2,2.275017686421052
3,3.295665678335354
4,4.817330794176087
5,5.371476975502446
6,6.743488555075601
7,7.345701594371349
8,8.887322778813541
9,9.570328279398382
****,*******************

# try reading into GRASS:
db.in.ogr dsn=x.dbf out=x

# check in GRASS:
db.select x
a|b
1|1.967018
2|2.275018
3|3.295666
4|4.817331
5|5.371477
6|6.743489
7|7.345702
8|8.887323
9|9.570328
|                   <------- correct use of NA



It seems that on export via OGR there is some kind of mangline of nodata. This 
issue is a serious problem anytime you need to move vector data between 
applications, through the use of an intermediate file (usually a shapefile).

Any ideas on what could be causing this problem? Fixing it would solve several 
important issues. If need be I will file a new ticket on the topic.

Relevent tickets?
http://trac.osgeo.org/gdal/ticket/809
http://trac.osgeo.org/gdal/ticket/528

Cheers,

Dylan

-- 
Dylan Beaudette
Soil Resource Laboratory
http://casoilresource.lawr.ucdavis.edu/
University of California at Davis
530.754.7341