[gdal-dev] Proper usage of gdal vsi list

[Laurențiu Nicola]

I think GDAL doesn't parse it because the URL doesn't match the directory name on the page (/ma-cdn03/GOES/data/GOES18/ABI/MESO/M1/GEOCOLOR/).

On Tue, Oct 28, 2025, at 20:58, Scott via gdal-dev wrote:
> Thanks for that! The sites you listed worked will with the gdal vsi list 
> client.
>
> Interestingly, the following worked with your sites as well, but not the 
> NOAA site:
>
> wget --spider -r --no-parent 
> https://cdn.star.nesdis.noaa.gov/GOES19/ABI/MESO/18N-76W/GEOCOLOR/ 2>&1 
> | grep "^--"
>
>
> On 10/28/25 11:44, Laurențiu Nicola via gdal-dev wrote:
>> I was wrong, GDAL actually has code to deal with the autoindex pages: https://github.com/OSGeo/gdal/blob/0f757d0/port/cpl_vsil_curl.cpp#L5024-L5340. It just doesn't work very well with your site:
>> 
>>>>> gdal.ReadDir("/vsicurl/https://cscie12.dce.harvard.edu/apache/autoindex/")
>> ['client-server.gif', 'raspberry.gif', 'raspberry.jpg', 'raspberry.png', 'slide1.html', 'slide2.html', 'slide3.html']
>> 
>>>>> gdal.ReadDir("/vsicurl/https://nginx.org/packages/")
>> ['aix', 'alpine', 'amzn', 'centos', 'debian', 'keys', 'rhel', 'sles', 'ubuntu', 'mainline']
>> 
>>>>> gdal.ReadDir("/vsicurl/https://cdn.star.nesdis.noaa.gov/GOES18/ABI/MESO/M1/GEOCOLOR/")
>> (nothing)
>> 
>> Laurentiu
>> 
>> On Tue, Oct 28, 2025, at 20:31, Scott via gdal-dev wrote:
>>> But, the docs say:
>>>
>>> gdal vsi list list files of GDAL Virtual File Systems (compressed,
>>> network hosted, etc...): /vsimem, /vsizip, /vsitar, /vsicurl, ....
>>>
>>> ...so this implies http dir listings via /vsicurl. I may be interpreting
>>> that incorrectly, though.
>>>
>>> Scott
>>>
>>> On 10/28/25 11:17, Laurențiu Nicola via gdal-dev wrote:
>>>> I'd be quite surprised if GDAL's vsicurl was able to understand a directory index.
>>>>
>>>> Take https://cscie12.dce.harvard.edu/apache/autoindex/, for example, it's clear that's a directory with 7 files, but it also has a bunch of column sorting links you'd want to ignore.
>>>>
>>>> To be fair, there aren't that many directory index formats, but I'm not sure that parsing them is a job for GDAL.
>>>>
>>>> Laurentiu
>>>>
>>>> On Tue, Oct 28, 2025, at 18:28, Scott via gdal-dev wrote:
>>>>> Using list with /vsicurl only returns results if I specify a file. No
>>>>> results when specifying a directory. /vsis3 works as expected. Am I
>>>>> doing this correctly?
>>>>>
>>>>> Works:
>>>>> gdal vsi list -lR
>>>>> "/vsicurl/https://cdn.star.nesdis.noaa.gov/GOES18/ABI/MESO/M1/GEOCOLOR/2025292000425_GOES18-ABI-MESO-GEOCOLOR-39N-119W-1000x1000.jpg"
>>>>>
>>>>> No workee:
>>>>> gdal vsi list -lR
>>>>> "/vsicurl/https://cdn.star.nesdis.noaa.gov/GOES18/ABI/MESO/M1/GEOCOLOR/"
>>>>>
>>>>> /vsicurl_streaming only returns directory name:
>>>>> gdal vsi list -lR
>>>>> "/vsicurl_streaming/https://cdn.star.nesdis.noaa.gov/GOES18/ABI/MESO/M1/GEOCOLOR/"
>>>>>
>>>>> --depth has no effect.
>>>>>
>>>>> Debian 12/GDAL 3.12.0beta1
>>>>>
>>>>> Thanks!
>>>>> Scott
>>>>>
>>>>> -- 
>>>>> www.postholer.com
>>>>>
>>>>> _______________________________________________
>>>>> gdal-dev mailing list
>>>>> gdal-dev at lists.osgeo.org
>>>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>> _______________________________________________
>>>> gdal-dev mailing list
>>>> gdal-dev at lists.osgeo.org
>>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>
>>> _______________________________________________
>>> gdal-dev mailing list
>>> gdal-dev at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev