[gdal-dev] ogr2ogr : limit memory usage

[Even Rouault]

On vendredi 13 octobre 2017 10:06:56 CEST Matthieu Lefort wrote:
> Hi list,
> I am using ogr2ogr to load big geojson files (about 1 Go each) into
> postgresql database. On my MacBook, (GDAL 1.11.3, released 2015/09/16),
> although it lasts about 20 minutes for each file, this works fine. I tried
> to do the same thing on a VPS (Debian Stretch 9.2, 8 Go RAM) but the
> process consumes all RAM in 20 seconds before stopping (out of memory).
> Install was made with apt-get install gdal-bin. (GDAL 2.1.2, released
> 2016/10/24)
> 
> Is there a way to control memory usage or to avoid at least using more
> memory than the available one ?

Matthieu,

no, this is a known issue ( see https://trac.osgeo.org/gdal/ticket/6540 ) which would require 
substantial changes in the geojson driver to be adressed. It needs to ingest the whole file in 
memory currently, parse it completely, and store the resulting features in RAM. So typically, 
you need 10 to 20 times more RAM than the file size (I wouldn't expect the memory usage to 
be substantially different between GDAL 1.11 and 2.1, but there has been indeed a small 
change to use the MEM driver underneath)

So for now, your best chance is to try splitting the features[] array into smaller files.

https://www.google.fr/search?q=splitting+geojson+files points to various hints on how to 
split geojson files.

Even


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20171013/608273c7/attachment.html>