[gdal-dev] Virtual Filesystem Question(s)

[Stephen Woodbridge]

Hi Even,

Frank pointed me to your blog entry on the GDAL virtual file system at:

http://erouault.blogspot.com/2012/05/new-gdal-virtual-file-system-to-read.html

This is most excellent. I am trying to get this to work using the 
/vsicurl/ option to connect to a php page that generates CSV output, But 
it is not sucessful :(

I was hoping you could help me sort out how to get this to work.

Running GDAL 1.9.0

woodbri at mappy:/u/software/gdal-1.9.0$ ./config.status -V
config.status
configured by ./configure, generated by GNU Autoconf 2.68,
   with options "'--prefix=/usr/local' '--mandir=/usr/local/share/man' 
'--includedir=/usr/local/include/gdal' '--with-threads' '--with-hdf5=no' 
'--with-grass=no' '--with-ecw=no' '--with-mrsid=no' '--with-jp2mrsid=no' 
'--with-libtiff=internal' '--with-geotiff=internal' '--with-jasper' 
'--with-netcdf' '--with-xerces' '--with-geos' '--with-sqlite3' 
'--with-curl' '--with-pg' '--with-mysql' '--with-perl' '--with-python' 
'--with-ecw' '--with-mrsid=/u/software/Geo_DSDK-7.0.0.2167' 
'--with-fgdb=/u/software/FileGDB_API' '--with-cfitsio=no'"

I have created a file test.csv and copied the output of the php script 
into and created a test.vrt and ogrinfo is happy with it.

I change test.vrt to

<SrcDataSource>/vsicurl/http://example.com/ws/test-pg.php</SrcDataSource>

This is happy:

woodbri at mappy:~/work/mongo/ogr$ ogrinfo -ro test-pg.vrt
INFO: Open of `test-pg.vrt'
       using driver `VRT' successful.
1: test (Point)

But this is not:

woodbri at mappy:~/work/mongo/ogr$ ogrinfo -ro test-pg.vrt test
INFO: Open of `test-pg.vrt'
       using driver `VRT' successful.
ERROR 1: Failed to open datasource 
`/vsicurl/http://example.com/ws/test-pg.php'.

Layer name: test
Geometry: Point
Feature Count: 0
Layer SRS WKT:
GEOGCS["WGS 84",
     DATUM["WGS_1984",
         SPHEROID["WGS 84",6378137,298.257223563,
             AUTHORITY["EPSG","7030"]],
         TOWGS84[0,0,0,0,0,0,0],
         AUTHORITY["EPSG","6326"]],
     PRIMEM["Greenwich",0,
         AUTHORITY["EPSG","8901"]],
     UNIT["degree",0.0174532925199433,
         AUTHORITY["EPSG","9108"]],
     AUTHORITY["EPSG","4326"]]

Doing some additional work like find a php script the simulates byte 
range downloading. and turning on some debuging, it looks like I have 
run into a but in OGR.

So if you notice below, OGR gets the file size and then asks for more 
bytes then are in the file. The whole file is sent but OGR get the 200 
response code but (I guessing) because the size returned is smaller then 
the size requested, it repeats the request over and over.

I'll grab the 1.9.2 tarball and see if I can build that and get better 
results. If your php request is doing a sql query (like I'm doing) then 
this is going to be problematic if the results exceed a single request, 
because you can not be assured that the subsequent request for the next 
part of the file will generate the exact same query results. It would be 
very helpful if there is a mode or virtual driver where OGR reads the 
whole data stream and buffers it into a tmp file or whatever then deals 
with it, because there is no way for the server to put the request into 
a tempfile and then relate it to subsequent requests from that client 
versus other clients and know when to clean things up. It makes more 
sense for the client to handle these issues because it knows when it is 
done with the resource.

Any thoughts or ideas on all of this?

Thanks,
   -Steve

woodbri at mappy:~/work/mongo/ogr$ ogrinfo --debug on -ro test-pg.vrt test
OGR: OGROpen(test-pg.vrt/0x177c3a0) succeeded as VRT.
INFO: Open of `test-pg.vrt'
       using driver `VRT' successful.
OGR: GetLayerCount() = 1

VSICURL: GetFileList(/vsicurl/http://imaptools.com:8080/ws)
VSICURL: File[0] = geoip.php, is_dir = 0, size = 0, time = 2013/02/08 
15:44:00
VSICURL: File[1] = test-pg.php, is_dir = 0, size = 0, time = 2013/02/09 
15:55:00
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
VSICURL: Got reponse_code=200
VSICURL: GetFileSize(http://imaptools.com:8080/ws/test-pg.php)=13053 
response_code=200
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
VSICURL: Got reponse_code=200
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
VSICURL: Got reponse_code=200
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
VSICURL: Got reponse_code=200
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
VSICURL: Got reponse_code=200
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
VSICURL: Got reponse_code=200
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
VSICURL: Got reponse_code=200
VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
^C

I logged the php code and here is a partial output:

header: Accept-Ranges: 0-13053
request: HTTP_RANGE: bytes=0-16383
header: Content-Range: bytes 0-13052/13053
header: Content-Length: 13053

header: Accept-Ranges: 0-13053
request: HTTP_RANGE: bytes=0-16383
header: Content-Range: bytes 0-13052/13053
header: Content-Length: 13053

header: Accept-Ranges: 0-13053
request: HTTP_RANGE: bytes=0-16383
header: Content-Range: bytes 0-13052/13053
header: Content-Length: 13053