[gdal-dev] Virtual Filesystem Question(s)

Stephen Woodbridge woodbri at swoodbridge.com
Sat Feb 9 19:55:23 PST 2013


On 2/9/2013 8:24 PM, Stephen Woodbridge wrote:
> On 2/9/2013 6:41 PM, Even Rouault wrote:
>> Le dimanche 10 février 2013 00:23:37, Stephen Woodbridge a écrit :
>>> On 2/9/2013 6:03 PM, Jukka Rahkonen wrote:
>>>> Stephen Woodbridge <woodbri <at> swoodbridge.com> writes:
>>>>> Just a followup to this, I have downloaded gdal-1.9.2 and get the same
>>>>> results. So no joy there.
>>>>>
>>>>> -Steve
>>>>
>>>> Perhaps vsicurl_streaming that is also mentioned in the same blog is
>>>> something worth trying.
>>>
>>> Yeah, I saw that. I may or may not be able to build trunk on this
>>> machine depending on the dependencies that needs as more of the
>>> libraries are fairly dated.
>>>
>>> But more importantly, I think this is a bug. I'm happy to write it up
>>> but it would be nice to get some confirmation. At the moment, I'm review
>>> the source code to see if I can see a problem there.
>>
>> Well, I rather think that there's a bug in your PHP script, because
>> curl -r
>> 0-16383 http://www.google.fr works even if the returned file is < 16383
>>
>> whereas :
>>
>> curl -r 0-16383 http://example.com/ws/test-pg.php hangs
>>
>> Or it is a bug in libcurl itself.
>
> OK, bug in my code. So using the above command now works. But ogrinfo
> alternately hangs and then runs with this error:
>
> ERROR 1: Failed to open datasource
> `/vsicurl/http://example.com/ws/test-pg.php'.
>
> In gdb, it appears to be hanging if I ^C and bt in poll()
>
> (gdb) run -ro test-pg.vrt test
> Starting program: /usr/local/bin/ogrinfo -ro test-pg.vrt test
> [Thread debugging using libthread_db enabled]
> warning: Lowest section in /usr/lib/libicudata.so.38 is .hash at
> 0000000000000120
> INFO: Open of `test-pg.vrt'
>        using driver `VRT' successful.
> [New Thread 0x7f35a95aa710 (LWP 13410)]
> ^C
> Program received signal SIGINT, Interrupt.
> [Switching to Thread 0x7f35a95aa710 (LWP 13410)]
> 0x00007f35a32efb9f in poll () from /lib/libc.so.6
> (gdb) bt
> #0  0x00007f35a32efb9f in poll () from /lib/libc.so.6
> #1  0x00007f35a51b7296 in ?? () from /usr/lib/libcurl.so.4
> #2  0x00007f35a51afbc1 in ?? () from /usr/lib/libcurl.so.4
> #3  0x00007f35a51afefd in ?? () from /usr/lib/libcurl.so.4
> #4  0x00007f35a519ea3b in ?? () from /usr/lib/libcurl.so.4
> #5  0x00007f35a51ac80b in ?? () from /usr/lib/libcurl.so.4
> #6  0x00007f35a8ca7f5c in VSICurlFilesystemHandler::GetFileList (
>      this=0x17b0310,
>      pszDirname=0x17b28e8 "/vsicurl/http://example.com/ws",
>      pbGotFileList=0x17b2850) at cpl_vsil_curl.cpp:2314
> #7  0x00007f35a8cab19b in VSICurlFilesystemHandler::ReadDir
> (this=0x17b0310,
>      pszDirname=<value optimized out>, pbGotFileList=0x7fff120d38f8)
>      at cpl_vsil_curl.cpp:2461
> #8  0x00007f35a8caa88b in VSICurlFilesystemHandler::Stat (this=0x17b0310,
>      pszFilename=0x17b2730 "/vsicurl/http://example.com/ws/test-pg.php",
>      pStatBuf=<value optimized out>, nFlags=3) at cpl_vsil_curl.cpp:2367
> #9  0x00007f35a8cb1c39 in VSIStatExL (
>      pszFilename=0x17b2730 "/vsicurl/http://example.com/ws/test-pg.php",
>      psStatBuf=0x7fff120d39b0, nFlags=3) at cpl_vsil.cpp:285
> #10 0x00007f35a8e73f44 in OGRShapeDataSource::Open (this=0x17b27a0,
>      pszNewName=0x17b2730 "/vsicurl/http://example.com/ws/test-pg.php",
>      bUpdate=0, bTestOpen=1, bForceSingleFileDataSource=0)
>      at ogrshapedatasource.cpp:109
> #11 0x00007f35a8e74e82 in OGRShapeDriver::Open (this=<value optimized out>,
>      pszFilename=0x17b2730 "/vsicurl/http://example.com/ws/test-pg.php",
>      bUpdate=0) at ogrshapedriver.cpp:70
> #12 0x00007f35a8e7272c in OGRSFDriverRegistrar::Open (
>      pszName=0x17b2730 "/vsicurl/http://example.com/ws/test-pg.php",
>      bUpdate=0, ppoDriver=0x0) at ogrsfdriverregistrar.cpp:226
> #13 0x00007f35a8e966f7 in OGRVRTLayer::FullInitialize (this=0x17b1530)
>      at ogrvrtlayer.cpp:322
> #14 0x00007f35a8e97450 in OGRVRTLayer::GetLayerDefn (this=0x17b1530)
>      at ogrvrtlayer.cpp:1864
> #15 0x000000000040195d in ReportOnLayer (poLayer=0x7fff120d2f50,
>      pszWHERE=0x1 <Address 0x1 out of bounds>, poSpatialFilter=0x493c0)
>      at ogrinfo.cpp:360
> #16 0x00000000004023f6 in main (nArgc=<value optimized out>,
>      papszArgv=0x17b0ae0) at ogrinfo.cpp:316
> (gdb) c
> Continuing.
>
> ##### after a very long wait 3-5 minutes it times out with the following
>
> ERROR 1: Failed to open datasource
> `/vsicurl/http://example.com/ws/test-pg.php'.
>
> Layer name: test
> Geometry: 3D Point
> Feature Count: 0
> Layer SRS WKT:
> GEOGCS["WGS 84",
>      DATUM["WGS_1984",
>          SPHEROID["WGS 84",6378137,298.257223563,
>              AUTHORITY["EPSG","7030"]],
>          TOWGS84[0,0,0,0,0,0,0],
>          AUTHORITY["EPSG","6326"]],
>      PRIMEM["Greenwich",0,
>          AUTHORITY["EPSG","8901"]],
>      UNIT["degree",0.0174532925199433,
>          AUTHORITY["EPSG","9108"]],
>      AUTHORITY["EPSG","4326"]]
>
> Program exited normally.
>
> OK, got 1.10beta1 compiled I'll give that a try. This might just be my
> very old version of curl 7.18.2

Even,

Running with 1.10beta1 and the (hopefully fixed) test-pg.php

I am getting similar behavior, so maybe my curl is just broken.

woodbri at mappy:~/work/mongo/ogr$ ogrinfo -ro test-pg.vrt test
INFO: Open of `test-pg.vrt'
       using driver `VRT' successful.
ERROR 1: Failed to open datasource 
`/vsicurl/http://imaptools.com:8080/ws/test-pg.php'.

Layer name: test
Geometry: Point
Feature Count: 0
Layer SRS WKT:
GEOGCS["WGS 84",
     DATUM["WGS_1984",
         SPHEROID["WGS 84",6378137,298.257223563,
             AUTHORITY["EPSG","7030"]],
         TOWGS84[0,0,0,0,0,0,0],
         AUTHORITY["EPSG","6326"]],
     PRIMEM["Greenwich",0,
         AUTHORITY["EPSG","8901"]],
     UNIT["degree",0.0174532925199433,
         AUTHORITY["EPSG","9108"]],
     AUTHORITY["EPSG","4326"]]

And my debug logging reports:

Accept-Ranges: 0-13052
HTTP_RANGE: bytes=0-16383
Content-Range: bytes 0-13052/13053
Content-Length: 13053
Reading: start: 0; length: 13053
strlen(out2): 13053    <---- this is the size of the echo response text

But the good news is that your /vsicurl_streaming/ is working. Why is it 
trying to down a .csvt file?

woodbri at mappy:~/work/mongo/ogr$ ogrinfo test-pg.vrt -al -q -ro --debug on
OGR: OGROpen(test-pg.vrt/0x22ed650) succeeded as VRT.
OGR: GetLayerCount() = 1

VSICURL: Start download for http://imaptools.com:8080/ws/test-pg.php?id=12
VSICURL: Stop download for http://imaptools.com:8080/ws/test-pg.php?id=12
VSICURL: Start download for http://imaptools.com:8080/ws/test-pg.csvt
VSICURL: Stop download for http://imaptools.com:8080/ws/test-pg.csvt
OGR: 
OGROpen(CSV:/vsicurl_streaming/http://imaptools.com:8080/ws/test-pg.php?id=12/0x22ef680) 
succeeded as CSV.

Thanks,
   -Steve


> -Steve
>
>> I haven't checked the RFCs, but all tests I have done before against
>> Apache or
>> IIS servers show that asking for more than the file size work.
>>
>>>
>>> Thank you for your suggestion,
>>>     -Steve
>>>
>>> _______________________________________________
>>> gdal-dev mailing list
>>> gdal-dev at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev



More information about the gdal-dev mailing list