[gdal-dev] Rare read errors using /vsis3

[Tim Harris]

I am using Python and translating small chunks of imagery out of S3 and
occasionally run into errors like this:



2024-12-17 18:06:16.201 MST: ERROR 1: Request for 372390946-372700449
failed with response_code=0

2024-12-17 18:06:16.201 MST: ERROR 1: Request for 2028508162-2028817665
failed with response_code=0

2024-12-17 18:06:16.201 MST: ERROR 1: Request for 2030984194-2031293697
failed with response_code=0

2024-12-17 18:06:16.202 MST: ERROR 1: Request for 1476778594-1477088097
failed with response_code=0

2024-12-17 18:06:16.202 MST: ERROR 1: Request for 374247970-374557473
failed with response_code=0



Even with gdal.UseExceptions() and GDAL_HTTP_MAX_RETRY /
GDAL_HTTP_RETRY_DELAY set, it doesn’t seem to do anything other than log
this error and proceed. The end result is that my output file has blank
pixels in it.



I have been staring at the code a bit to understand what it means,
particularly the response_code being 0. I see this in the libcurl docs:



> The stored value is zero if no server response code has been received.



I still don’t quite know why this happens but could be that some AWS server
closed a connection early or something. But is there any way to force a
retry? I see that GDAL_HTTP_RETRY_CODES was added recently. If I set it to
“ALL” will that ensure that a situation like this results in a retry even
on a closed connection? I am not set up to easily reproduce this or even to
see a stack trace, so I’m not sure what leads to this error. It doesn’t
happen very often, but it’s disruptive when it does happen because it’s
hard to catch.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20241219/a8f506cc/attachment-0001.htm>