[gdal-dev] Rare read errors using /vsis3

[Even Rouault]

Tim,

as far as I can see this error message comes from the specific 
ReadMultiRange() method that has no retry capabilities. It should 
probably, although I'm unsure from an implementation point of view how 
that would work with the multi request curl API. I guess it can. Anway, 
just to tell that you're going to be out of luck there.

A  workaround would be to set GDAL_HTTP_MULTIRANGE=SERIAL to go back to 
the classic code path (obviously you'll loose the parallel network 
request aspect!), and there GDAL_HTTP_RETRY_CODES=ALL should hopefully 
work even for response_code=0

Even

Le 19/12/2024 à 20:13, Tim Harris via gdal-dev a écrit :
>
> I am using Python and translating small chunks of imagery out of S3 
> and occasionally run into errors like this:
>
> 2024-12-17 18:06:16.201 MST: ERROR 1: Request for 372390946-372700449 
> failed with response_code=0
>
> 2024-12-17 18:06:16.201 MST: ERROR 1: Request for 
> 2028508162-2028817665 failed with response_code=0
>
> 2024-12-17 18:06:16.201 MST: ERROR 1: Request for 
> 2030984194-2031293697 failed with response_code=0
>
> 2024-12-17 18:06:16.202 MST: ERROR 1: Request for 
> 1476778594-1477088097 failed with response_code=0
>
> 2024-12-17 18:06:16.202 MST: ERROR 1: Request for 374247970-374557473 
> failed with response_code=0
>
> Even with gdal.UseExceptions() and GDAL_HTTP_MAX_RETRY / 
> GDAL_HTTP_RETRY_DELAY set, it doesn’t seem to do anything other than 
> log this error and proceed. The end result is that my output file has 
> blank pixels in it.
>
> I have been staring at the code a bit to understand what it means, 
> particularly the response_code being 0. I see this in the libcurl docs:
>
> > The stored value is zero if no server response code has been received.
>
> I still don’t quite know why this happens but could be that some AWS 
> server closed a connection early or something. But is there any way to 
> force a retry? I see that GDAL_HTTP_RETRY_CODES was added recently. If 
> I set it to “ALL” will that ensure that a situation like this results 
> in a retry even on a closed connection? I am not set up to easily 
> reproduce this or even to see a stack trace, so I’m not sure what 
> leads to this error. It doesn’t happen very often, but it’s disruptive 
> when it does happen because it’s hard to catch.
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
Butcher of all kinds of standards, open or closed formats. At the end, this is just about bytes.
Mood of the day: "Bien entendu, on peut sauter sur sa chaise comme un cabri en disant : les standards ! les standards ! les standards ! Mais ça n’aboutit à rien et ça ne signifie rien." ~ dixit De Gaulle
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20241219/667a76e1/attachment.htm>