[gdal-dev] Virtual Filesystem Question(s)

Stephen Woodbridge woodbri at swoodbridge.com
Sat Feb 9 14:32:17 PST 2013


Just a followup to this, I have downloaded gdal-1.9.2 and get the same 
results. So no joy there.

-Steve

On 2/9/2013 4:28 PM, Stephen Woodbridge wrote:
> Hi Even,
>
> Frank pointed me to your blog entry on the GDAL virtual file system at:
>
> http://erouault.blogspot.com/2012/05/new-gdal-virtual-file-system-to-read.html
>
>
> This is most excellent. I am trying to get this to work using the
> /vsicurl/ option to connect to a php page that generates CSV output, But
> it is not sucessful :(
>
> I was hoping you could help me sort out how to get this to work.
>
> Running GDAL 1.9.0
>
> woodbri at mappy:/u/software/gdal-1.9.0$ ./config.status -V
> config.status
> configured by ./configure, generated by GNU Autoconf 2.68,
>    with options "'--prefix=/usr/local' '--mandir=/usr/local/share/man'
> '--includedir=/usr/local/include/gdal' '--with-threads' '--with-hdf5=no'
> '--with-grass=no' '--with-ecw=no' '--with-mrsid=no' '--with-jp2mrsid=no'
> '--with-libtiff=internal' '--with-geotiff=internal' '--with-jasper'
> '--with-netcdf' '--with-xerces' '--with-geos' '--with-sqlite3'
> '--with-curl' '--with-pg' '--with-mysql' '--with-perl' '--with-python'
> '--with-ecw' '--with-mrsid=/u/software/Geo_DSDK-7.0.0.2167'
> '--with-fgdb=/u/software/FileGDB_API' '--with-cfitsio=no'"
>
> I have created a file test.csv and copied the output of the php script
> into and created a test.vrt and ogrinfo is happy with it.
>
> I change test.vrt to
>
> <SrcDataSource>/vsicurl/http://example.com/ws/test-pg.php</SrcDataSource>
>
> This is happy:
>
> woodbri at mappy:~/work/mongo/ogr$ ogrinfo -ro test-pg.vrt
> INFO: Open of `test-pg.vrt'
>        using driver `VRT' successful.
> 1: test (Point)
>
> But this is not:
>
> woodbri at mappy:~/work/mongo/ogr$ ogrinfo -ro test-pg.vrt test
> INFO: Open of `test-pg.vrt'
>        using driver `VRT' successful.
> ERROR 1: Failed to open datasource
> `/vsicurl/http://example.com/ws/test-pg.php'.
>
> Layer name: test
> Geometry: Point
> Feature Count: 0
> Layer SRS WKT:
> GEOGCS["WGS 84",
>      DATUM["WGS_1984",
>          SPHEROID["WGS 84",6378137,298.257223563,
>              AUTHORITY["EPSG","7030"]],
>          TOWGS84[0,0,0,0,0,0,0],
>          AUTHORITY["EPSG","6326"]],
>      PRIMEM["Greenwich",0,
>          AUTHORITY["EPSG","8901"]],
>      UNIT["degree",0.0174532925199433,
>          AUTHORITY["EPSG","9108"]],
>      AUTHORITY["EPSG","4326"]]
>
> Doing some additional work like find a php script the simulates byte
> range downloading. and turning on some debuging, it looks like I have
> run into a but in OGR.
>
> So if you notice below, OGR gets the file size and then asks for more
> bytes then are in the file. The whole file is sent but OGR get the 200
> response code but (I guessing) because the size returned is smaller then
> the size requested, it repeats the request over and over.
>
> I'll grab the 1.9.2 tarball and see if I can build that and get better
> results. If your php request is doing a sql query (like I'm doing) then
> this is going to be problematic if the results exceed a single request,
> because you can not be assured that the subsequent request for the next
> part of the file will generate the exact same query results. It would be
> very helpful if there is a mode or virtual driver where OGR reads the
> whole data stream and buffers it into a tmp file or whatever then deals
> with it, because there is no way for the server to put the request into
> a tempfile and then relate it to subsequent requests from that client
> versus other clients and know when to clean things up. It makes more
> sense for the client to handle these issues because it knows when it is
> done with the resource.
>
> Any thoughts or ideas on all of this?
>
> Thanks,
>    -Steve
>
> woodbri at mappy:~/work/mongo/ogr$ ogrinfo --debug on -ro test-pg.vrt test
> OGR: OGROpen(test-pg.vrt/0x177c3a0) succeeded as VRT.
> INFO: Open of `test-pg.vrt'
>        using driver `VRT' successful.
> OGR: GetLayerCount() = 1
>
> VSICURL: GetFileList(/vsicurl/http://imaptools.com:8080/ws)
> VSICURL: File[0] = geoip.php, is_dir = 0, size = 0, time = 2013/02/08
> 15:44:00
> VSICURL: File[1] = test-pg.php, is_dir = 0, size = 0, time = 2013/02/09
> 15:55:00
> VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
> VSICURL: Got reponse_code=200
> VSICURL: GetFileSize(http://imaptools.com:8080/ws/test-pg.php)=13053
> response_code=200
> VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
> VSICURL: Got reponse_code=200
> VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
> VSICURL: Got reponse_code=200
> VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
> VSICURL: Got reponse_code=200
> VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
> VSICURL: Got reponse_code=200
> VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
> VSICURL: Got reponse_code=200
> VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
> VSICURL: Got reponse_code=200
> VSICURL: Downloading 0-16383 (http://imaptools.com:8080/ws/test-pg.php)...
> ^C
>
> I logged the php code and here is a partial output:
>
> header: Accept-Ranges: 0-13053
> request: HTTP_RANGE: bytes=0-16383
> header: Content-Range: bytes 0-13052/13053
> header: Content-Length: 13053
>
> header: Accept-Ranges: 0-13053
> request: HTTP_RANGE: bytes=0-16383
> header: Content-Range: bytes 0-13052/13053
> header: Content-Length: 13053
>
> header: Accept-Ranges: 0-13053
> request: HTTP_RANGE: bytes=0-16383
> header: Content-Range: bytes 0-13052/13053
> header: Content-Length: 13053
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev



More information about the gdal-dev mailing list