[gdal-dev] Memory utilization and GDAL_HTTP_MULTIRANGE
Peter Schmitt
pschmitt at gmail.com
Fri Apr 19 14:58:09 PDT 2019
Hi,
I have been experimenting with the GDAL_HTTP_MULTIRANGE config option added
in gdal-2.3. I have noticed that setting this to YES requires fewer curl
requests, but utilizes a more memory compared to SERIAL. From a very simple
test (details below):
1. gdal-2.2.4:
Maximum resident set size (kbytes): 45544
2. gdal-2.3.0 with GDAL_HTTP_MULTIRANGE=SERIAL
Maximum resident set size (kbytes): 46120
3. gdal-2.3.0 with GDAL_HTTP_MULTIRANGE=YES
Maximum resident set size (kbytes): 83828
The biggest thing I noticed from the log is that multiple HTTP connections
are left intact when GDAL_HTTP_MULTIRANGE=YES. I'm guessing this is where
the additional memory utilization comes from. Is there any way to reduce
the memory footprint when GDAL_HTTP_MULTIRANGE=YES?
I came across this while trying to understand why I was seeing a much
higher memory footprint with a MapServer built with gdal >= 2.3 compared to
MapServer built with gdal <= 2.2.4, detailed here:
https://lists.osgeo.org/pipermail/mapserver-users/2019-April/081121.html
Thanks,
Pete
PS: Here are my test details. I ran a test with the following config
options:
GDAL_DISABLE_READDIR_ON_OPEN=YES
CPL_VSIL_CURL_ALLOWED_EXTENSIONS=tif
GDAL_CACHEMAX=0
VSI_CACHE=FALSE
CPL_VSIL_CURL_NON_CACHED="/vsicurl/https://s3.amazonaws.com"
CPL_CURL_VERBOSE=ON
and the following command:
/usr/bin/time -v gdal_translate -outsize 256 256 -projwin_srs EPSG:3857
-projwin 13894417.251632 -1012637.7505811 13895028.747858 -1013249.2468073
/vsicurl/
https://s3.amazonaws.com/pschmitt-test-public/b378087a-c2a5-43a0-abec-71fcfb051150_cog.tif
/out.tif | grep "Maximum resident set size"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20190419/90780c1c/attachment.html>
More information about the gdal-dev
mailing list