[gdal-dev] Memory utilization and GDAL_HTTP_MULTIRANGE

[Peter Schmitt]

Hi,

I have been experimenting with the GDAL_HTTP_MULTIRANGE config option added
in gdal-2.3.  I have noticed that setting this to YES requires fewer curl
requests, but utilizes a more memory compared to SERIAL. From a very simple
test (details below):

1. gdal-2.2.4:
        Maximum resident set size (kbytes): 45544
2. gdal-2.3.0 with GDAL_HTTP_MULTIRANGE=SERIAL
        Maximum resident set size (kbytes): 46120
3. gdal-2.3.0 with GDAL_HTTP_MULTIRANGE=YES
        Maximum resident set size (kbytes): 83828

The biggest thing I noticed from the log is that multiple HTTP connections
are left intact when GDAL_HTTP_MULTIRANGE=YES. I'm guessing this is where
the additional memory utilization comes from.  Is there any way to reduce
the memory footprint when GDAL_HTTP_MULTIRANGE=YES?

I came across this while trying to understand why I was seeing a much
higher memory footprint with a MapServer built with gdal >= 2.3 compared to
MapServer built with gdal <= 2.2.4, detailed here:
https://lists.osgeo.org/pipermail/mapserver-users/2019-April/081121.html

Thanks,
Pete

PS:  Here are my test details.  I ran a test with the following config
options:

GDAL_DISABLE_READDIR_ON_OPEN=YES
CPL_VSIL_CURL_ALLOWED_EXTENSIONS=tif
GDAL_CACHEMAX=0
VSI_CACHE=FALSE
CPL_VSIL_CURL_NON_CACHED="/vsicurl/https://s3.amazonaws.com"
CPL_CURL_VERBOSE=ON

and the following command:
/usr/bin/time -v gdal_translate -outsize 256 256 -projwin_srs EPSG:3857
-projwin 13894417.251632 -1012637.7505811 13895028.747858 -1013249.2468073
/vsicurl/
https://s3.amazonaws.com/pschmitt-test-public/b378087a-c2a5-43a0-abec-71fcfb051150_cog.tif
/out.tif | grep "Maximum resident set size"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20190419/90780c1c/attachment.html>