[gdal-dev] Proper usage of gdal vsi list

[Laurențiu Nicola]

I was wrong, GDAL actually has code to deal with the autoindex pages: https://github.com/OSGeo/gdal/blob/0f757d0/port/cpl_vsil_curl.cpp#L5024-L5340. It just doesn't work very well with your site:

>>> gdal.ReadDir("/vsicurl/https://cscie12.dce.harvard.edu/apache/autoindex/")
['client-server.gif', 'raspberry.gif', 'raspberry.jpg', 'raspberry.png', 'slide1.html', 'slide2.html', 'slide3.html']

>>> gdal.ReadDir("/vsicurl/https://nginx.org/packages/")
['aix', 'alpine', 'amzn', 'centos', 'debian', 'keys', 'rhel', 'sles', 'ubuntu', 'mainline']

>>> gdal.ReadDir("/vsicurl/https://cdn.star.nesdis.noaa.gov/GOES18/ABI/MESO/M1/GEOCOLOR/")
(nothing)

Laurentiu

On Tue, Oct 28, 2025, at 20:31, Scott via gdal-dev wrote:
> But, the docs say:
>
> gdal vsi list list files of GDAL Virtual File Systems (compressed, 
> network hosted, etc...): /vsimem, /vsizip, /vsitar, /vsicurl, ....
>
> ...so this implies http dir listings via /vsicurl. I may be interpreting 
> that incorrectly, though.
>
> Scott
>
> On 10/28/25 11:17, Laurențiu Nicola via gdal-dev wrote:
>> I'd be quite surprised if GDAL's vsicurl was able to understand a directory index.
>> 
>> Take https://cscie12.dce.harvard.edu/apache/autoindex/, for example, it's clear that's a directory with 7 files, but it also has a bunch of column sorting links you'd want to ignore.
>> 
>> To be fair, there aren't that many directory index formats, but I'm not sure that parsing them is a job for GDAL.
>> 
>> Laurentiu
>> 
>> On Tue, Oct 28, 2025, at 18:28, Scott via gdal-dev wrote:
>>> Using list with /vsicurl only returns results if I specify a file. No
>>> results when specifying a directory. /vsis3 works as expected. Am I
>>> doing this correctly?
>>>
>>> Works:
>>> gdal vsi list -lR
>>> "/vsicurl/https://cdn.star.nesdis.noaa.gov/GOES18/ABI/MESO/M1/GEOCOLOR/2025292000425_GOES18-ABI-MESO-GEOCOLOR-39N-119W-1000x1000.jpg"
>>>
>>> No workee:
>>> gdal vsi list -lR
>>> "/vsicurl/https://cdn.star.nesdis.noaa.gov/GOES18/ABI/MESO/M1/GEOCOLOR/"
>>>
>>> /vsicurl_streaming only returns directory name:
>>> gdal vsi list -lR
>>> "/vsicurl_streaming/https://cdn.star.nesdis.noaa.gov/GOES18/ABI/MESO/M1/GEOCOLOR/"
>>>
>>> --depth has no effect.
>>>
>>> Debian 12/GDAL 3.12.0beta1
>>>
>>> Thanks!
>>> Scott
>>>
>>> -- 
>>> www.postholer.com
>>>
>>> _______________________________________________
>>> gdal-dev mailing list
>>> gdal-dev at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev