[gdal-dev] vsicurl and an issue with http 302 redirect in the middle of the file

[Even Rouault]

Jukka,

Running with CPL_CURL_VERBOSE=YES shows the following

 > GET 
/data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t15z.wrfsfcf01.grib2 
HTTP/1.1
Host: ftpprd.ncep.noaa.gov
Accept: */*
Range: bytes=84819968-84836351

* Mark bundle as not supporting multiuse
< HTTP/1.1  302 Your allowed limit has been reached. Please go to 
https://www.weather.gov/abusive-user-block  for more info
* no chunk, no close, no size. Assume close to signal end
<

So it seems that after a number of quickly consecutive GET requests to 
the server as here, the server no longer want to see us. The 302 HTTP 
error it sends however is not appropriate. 429 "Too many requests" would 
be more appropriate

GRIB files with lots of "messages" / bands like that one aren't really 
cloud friendly, as you have to skip in many parts of the file to get an 
inventory of it.

That said, I see that in that instance, the .grib files are accompanied 
by a sidecar file, like 
https://ftpprd.ncep.noaa.gov/data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t15z.wrfsfcf01.grib2.idx, 
which I believe is produced by the wgrib utility. We could conceptually 
try to make use of it, however it doesn't contain all metadata that are 
typically retrieved by the driver, so for gdalinfo itself that wouldn't 
be sufficient, but for just a Open() call that might be enough.

I've noted that https://github.com/OSGeo/gdal/issues/3799

Even


Le 05/05/2021 à 22:04, Rahkonen Jukka (MML) a écrit :
>
> Hi,
>
> Have a look at 
> https://gis.stackexchange.com/questions/395867/opening-a-grib-from-the-web-with-gdal-in-python-using-vsicurl-throws-error-on-m 
> <https://gis.stackexchange.com/questions/395867/opening-a-grib-from-the-web-with-gdal-in-python-using-vsicurl-throws-error-on-m>. 
> There gdalinfo fails with /vsicurl/ somewhere in the middle of the 
> grib2 file when the server sends http 302 as a response for a range 
> request. The file is OK as was tested by downloading it first totally 
> with curl. Could it be that the http “moved permanently” at that stage 
> makes /vsicurl/ to fail?
>
> I made also a few tests and it seems that the error may happen in 
> another http range, but it is always similar:
> VSICURL: Got response_code=302
>
> GRIB: ERROR: Ran out of file in Section 7
>
> -Jukka Rahkonen-
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20210505/d3d764da/attachment.html>