[gdal-dev] Virtual Filesystem Question(s)

[Stephen Woodbridge]

On 2/9/2013 6:41 PM, Even Rouault wrote:
> Le dimanche 10 février 2013 00:23:37, Stephen Woodbridge a écrit :
>> On 2/9/2013 6:03 PM, Jukka Rahkonen wrote:
>>> Stephen Woodbridge <woodbri <at> swoodbridge.com> writes:
>>>> Just a followup to this, I have downloaded gdal-1.9.2 and get the same
>>>> results. So no joy there.
>>>>
>>>> -Steve
>>>
>>> Perhaps vsicurl_streaming that is also mentioned in the same blog is
>>> something worth trying.
>>
>> Yeah, I saw that. I may or may not be able to build trunk on this
>> machine depending on the dependencies that needs as more of the
>> libraries are fairly dated.
>>
>> But more importantly, I think this is a bug. I'm happy to write it up
>> but it would be nice to get some confirmation. At the moment, I'm review
>> the source code to see if I can see a problem there.
>
> Well, I rather think that there's a bug in your PHP script, because curl -r
> 0-16383 http://www.google.fr works even if the returned file is < 16383
>
> whereas :
>
> curl -r 0-16383 http://example.com/ws/test-pg.php hangs
>
> Or it is a bug in libcurl itself.

OK, bug in my code. So using the above command now works. But ogrinfo 
alternately hangs and then runs with this error:

ERROR 1: Failed to open datasource 
`/vsicurl/http://example.com/ws/test-pg.php'.

In gdb, it appears to be hanging if I ^C and bt in poll()

(gdb) run -ro test-pg.vrt test
Starting program: /usr/local/bin/ogrinfo -ro test-pg.vrt test
[Thread debugging using libthread_db enabled]
warning: Lowest section in /usr/lib/libicudata.so.38 is .hash at 
0000000000000120
INFO: Open of `test-pg.vrt'
       using driver `VRT' successful.
[New Thread 0x7f35a95aa710 (LWP 13410)]
^C
Program received signal SIGINT, Interrupt.
[Switching to Thread 0x7f35a95aa710 (LWP 13410)]
0x00007f35a32efb9f in poll () from /lib/libc.so.6
(gdb) bt
#0  0x00007f35a32efb9f in poll () from /lib/libc.so.6
#1  0x00007f35a51b7296 in ?? () from /usr/lib/libcurl.so.4
#2  0x00007f35a51afbc1 in ?? () from /usr/lib/libcurl.so.4
#3  0x00007f35a51afefd in ?? () from /usr/lib/libcurl.so.4
#4  0x00007f35a519ea3b in ?? () from /usr/lib/libcurl.so.4
#5  0x00007f35a51ac80b in ?? () from /usr/lib/libcurl.so.4
#6  0x00007f35a8ca7f5c in VSICurlFilesystemHandler::GetFileList (
     this=0x17b0310,
     pszDirname=0x17b28e8 "/vsicurl/http://example.com/ws",
     pbGotFileList=0x17b2850) at cpl_vsil_curl.cpp:2314
#7  0x00007f35a8cab19b in VSICurlFilesystemHandler::ReadDir (this=0x17b0310,
     pszDirname=<value optimized out>, pbGotFileList=0x7fff120d38f8)
     at cpl_vsil_curl.cpp:2461
#8  0x00007f35a8caa88b in VSICurlFilesystemHandler::Stat (this=0x17b0310,
     pszFilename=0x17b2730 "/vsicurl/http://example.com/ws/test-pg.php",
     pStatBuf=<value optimized out>, nFlags=3) at cpl_vsil_curl.cpp:2367
#9  0x00007f35a8cb1c39 in VSIStatExL (
     pszFilename=0x17b2730 "/vsicurl/http://example.com/ws/test-pg.php",
     psStatBuf=0x7fff120d39b0, nFlags=3) at cpl_vsil.cpp:285
#10 0x00007f35a8e73f44 in OGRShapeDataSource::Open (this=0x17b27a0,
     pszNewName=0x17b2730 "/vsicurl/http://example.com/ws/test-pg.php",
     bUpdate=0, bTestOpen=1, bForceSingleFileDataSource=0)
     at ogrshapedatasource.cpp:109
#11 0x00007f35a8e74e82 in OGRShapeDriver::Open (this=<value optimized out>,
     pszFilename=0x17b2730 "/vsicurl/http://example.com/ws/test-pg.php",
     bUpdate=0) at ogrshapedriver.cpp:70
#12 0x00007f35a8e7272c in OGRSFDriverRegistrar::Open (
     pszName=0x17b2730 "/vsicurl/http://example.com/ws/test-pg.php",
     bUpdate=0, ppoDriver=0x0) at ogrsfdriverregistrar.cpp:226
#13 0x00007f35a8e966f7 in OGRVRTLayer::FullInitialize (this=0x17b1530)
     at ogrvrtlayer.cpp:322
#14 0x00007f35a8e97450 in OGRVRTLayer::GetLayerDefn (this=0x17b1530)
     at ogrvrtlayer.cpp:1864
#15 0x000000000040195d in ReportOnLayer (poLayer=0x7fff120d2f50,
     pszWHERE=0x1 <Address 0x1 out of bounds>, poSpatialFilter=0x493c0)
     at ogrinfo.cpp:360
#16 0x00000000004023f6 in main (nArgc=<value optimized out>,
     papszArgv=0x17b0ae0) at ogrinfo.cpp:316
(gdb) c
Continuing.

##### after a very long wait 3-5 minutes it times out with the following

ERROR 1: Failed to open datasource 
`/vsicurl/http://example.com/ws/test-pg.php'.

Layer name: test
Geometry: 3D Point
Feature Count: 0
Layer SRS WKT:
GEOGCS["WGS 84",
     DATUM["WGS_1984",
         SPHEROID["WGS 84",6378137,298.257223563,
             AUTHORITY["EPSG","7030"]],
         TOWGS84[0,0,0,0,0,0,0],
         AUTHORITY["EPSG","6326"]],
     PRIMEM["Greenwich",0,
         AUTHORITY["EPSG","8901"]],
     UNIT["degree",0.0174532925199433,
         AUTHORITY["EPSG","9108"]],
     AUTHORITY["EPSG","4326"]]

Program exited normally.

OK, got 1.10beta1 compiled I'll give that a try. This might just be my 
very old version of curl 7.18.2

-Steve

> I haven't checked the RFCs, but all tests I have done before against Apache or
> IIS servers show that asking for more than the file size work.
>
>>
>> Thank you for your suggestion,
>>     -Steve
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev