[gdal-dev] Using ogr2ogr with limited memory

[Even Rouault]

Le 28/09/2023 à 18:47, Scott via gdal-dev a écrit :
>
> I should have been more specific.
>
> One particular machine has 8GB of memory. When I try to do the most 
> simple ogr2ogr command on large files, the host runs out of memory 
> (vmstat shows this) and ogr2ogr terminates with 'Killed', nothing more.
>
> The data formats I have experienced this with are .fgb, .parquet and 
> .gpkg. The data files are 10's of GB.

As input ? as output? Which operating system ? Which GDAL version ? The 
output of "ogrinfo -al -so the_input" might also be helpful. An exact 
ogr2ogr command line invocation that triggers the issue would certainly 
be useful.  In general, most GDAL drivers and ogr2ogr itself operate in 
streaming mode with low RAM requirements, but there might be exceptions 
(some configurations of GeoJSON file may require full ingestion on 
reading for example).  I'm also aware of issues with RAM fragmentation 
due to how some memory allocators work, but they seem to be restricted 
to multithreaded uses 
(https://gdal.org/user/multithreading.html#ram-fragmentation-and-multi-threading), 
which current ogr2ogr shouldn't trigger

Even

>
> Thanks for the responses!
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.