[gdal-dev] ogr2ogr : limit memory usage

Matthieu Lefort mlefort at neogeo.fr
Fri Oct 13 03:48:50 PDT 2017


OK, I’ll try it. Thanks for theses informations.
Matthieu

> Le 13 oct. 2017 à 12:43, Even Rouault <even.rouault at spatialys.com> a écrit :
> 
> On vendredi 13 octobre 2017 10:06:56 CEST Matthieu Lefort wrote:
> > Hi list,
> > I am using ogr2ogr to load big geojson files (about 1 Go each) into
> > postgresql database. On my MacBook, (GDAL 1.11.3, released 2015/09/16),
> > although it lasts about 20 minutes for each file, this works fine. I tried
> > to do the same thing on a VPS (Debian Stretch 9.2, 8 Go RAM) but the
> > process consumes all RAM in 20 seconds before stopping (out of memory).
> > Install was made with apt-get install gdal-bin. (GDAL 2.1.2, released
> > 2016/10/24)
> >
> > Is there a way to control memory usage or to avoid at least using more
> > memory than the available one ?
> 
> Matthieu,
> 
> no, this is a known issue ( see https://trac.osgeo.org/gdal/ticket/6540 <https://trac.osgeo.org/gdal/ticket/6540> ) which would require substantial changes in the geojson driver to be adressed. It needs to ingest the whole file in memory currently, parse it completely, and store the resulting features in RAM. So typically, you need 10 to 20 times more RAM than the file size (I wouldn't expect the memory usage to be substantially different between GDAL 1.11 and 2.1, but there has been indeed a small change to use the MEM driver underneath)
> 
> So for now, your best chance is to try splitting the features[] array into smaller files.
> 
> https://www.google.fr/search?q=splitting+geojson+files <https://www.google.fr/search?q=splitting+geojson+files> points to various hints on how to split geojson files.
> 
> Even
> 
> 
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com <http://www.spatialys.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20171013/575cf364/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20171013/575cf364/attachment.sig>


More information about the gdal-dev mailing list