[gdal-dev] Proper usage of gdal vsi list

[Scott]

Thanks for that! The sites you listed worked will with the gdal vsi list 
client.

Interestingly, the following worked with your sites as well, but not the 
NOAA site:

wget --spider -r --no-parent 
https://cdn.star.nesdis.noaa.gov/GOES19/ABI/MESO/18N-76W/GEOCOLOR/ 2>&1 
| grep "^--"


On 10/28/25 11:44, Laurențiu Nicola via gdal-dev wrote:
> I was wrong, GDAL actually has code to deal with the autoindex pages: https://github.com/OSGeo/gdal/blob/0f757d0/port/cpl_vsil_curl.cpp#L5024-L5340. It just doesn't work very well with your site:
> 
>>>> gdal.ReadDir("/vsicurl/https://cscie12.dce.harvard.edu/apache/autoindex/")
> ['client-server.gif', 'raspberry.gif', 'raspberry.jpg', 'raspberry.png', 'slide1.html', 'slide2.html', 'slide3.html']
> 
>>>> gdal.ReadDir("/vsicurl/https://nginx.org/packages/")
> ['aix', 'alpine', 'amzn', 'centos', 'debian', 'keys', 'rhel', 'sles', 'ubuntu', 'mainline']
> 
>>>> gdal.ReadDir("/vsicurl/https://cdn.star.nesdis.noaa.gov/GOES18/ABI/MESO/M1/GEOCOLOR/")
> (nothing)
> 
> Laurentiu
> 
> On Tue, Oct 28, 2025, at 20:31, Scott via gdal-dev wrote:
>> But, the docs say:
>>
>> gdal vsi list list files of GDAL Virtual File Systems (compressed,
>> network hosted, etc...): /vsimem, /vsizip, /vsitar, /vsicurl, ....
>>
>> ...so this implies http dir listings via /vsicurl. I may be interpreting
>> that incorrectly, though.
>>
>> Scott
>>
>> On 10/28/25 11:17, Laurențiu Nicola via gdal-dev wrote:
>>> I'd be quite surprised if GDAL's vsicurl was able to understand a directory index.
>>>
>>> Take https://cscie12.dce.harvard.edu/apache/autoindex/, for example, it's clear that's a directory with 7 files, but it also has a bunch of column sorting links you'd want to ignore.
>>>
>>> To be fair, there aren't that many directory index formats, but I'm not sure that parsing them is a job for GDAL.
>>>
>>> Laurentiu
>>>
>>> On Tue, Oct 28, 2025, at 18:28, Scott via gdal-dev wrote:
>>>> Using list with /vsicurl only returns results if I specify a file. No
>>>> results when specifying a directory. /vsis3 works as expected. Am I
>>>> doing this correctly?
>>>>
>>>> Works:
>>>> gdal vsi list -lR
>>>> "/vsicurl/https://cdn.star.nesdis.noaa.gov/GOES18/ABI/MESO/M1/GEOCOLOR/2025292000425_GOES18-ABI-MESO-GEOCOLOR-39N-119W-1000x1000.jpg"
>>>>
>>>> No workee:
>>>> gdal vsi list -lR
>>>> "/vsicurl/https://cdn.star.nesdis.noaa.gov/GOES18/ABI/MESO/M1/GEOCOLOR/"
>>>>
>>>> /vsicurl_streaming only returns directory name:
>>>> gdal vsi list -lR
>>>> "/vsicurl_streaming/https://cdn.star.nesdis.noaa.gov/GOES18/ABI/MESO/M1/GEOCOLOR/"
>>>>
>>>> --depth has no effect.
>>>>
>>>> Debian 12/GDAL 3.12.0beta1
>>>>
>>>> Thanks!
>>>> Scott
>>>>
>>>> -- 
>>>> www.postholer.com
>>>>
>>>> _______________________________________________
>>>> gdal-dev mailing list
>>>> gdal-dev at lists.osgeo.org
>>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>> _______________________________________________
>>> gdal-dev mailing list
>>> gdal-dev at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev