[Gdal-dev] possible ogr2ogr memory leak

[Frank Warmerdam]

On Tue, 22 Mar 2005 14:59:07 +1100, Rhys Ickeringill
<rhysi at omnilink.com.au> wrote:
> During conversion of a large (~1Gb / 3 million record) MapInfo TAB file,
> memory usage peaked at around 0.5Gb. No coordinate system transformations
> were requested. Unless I have an almost impossibly complex geometry, this
> seems like abnormal behaviour to me. 

Rhys, 

How are you determining memory usage?  Some operation systems will
count disk file pages as part of the memory footprint of an application if
they remain in RAM for a while.  So, I tend not to take reports of memory
leaks based on system monitors very seriously. 

> The command line was:
> 
> ogr2ogr -f postgreSQL -lco DIM=2 -nln Roads -nlt MULTIPOLYGON -select
> "Road_Name,Road_Type" PG: "ROADS.TAB"
> 
> Do people agree that this sounds like abnormal behaviour? The only reason I
> can think that this would be expected is if the MITAB driver deliberately
> keeps records it has fetched loaded indefinately. Is this the case?

I would not expect any serious memory leaks.  The actual memory footprint
may depend on a variety of factors. 

I tried converting some polygonal mapinfo data into shapefiles, and 
run ogr2ogr under the valgrind utility which provides detailed reports
of heap memory still in use at the end of the application and there were
no reports that looked like they would scale up with the number of 
features.  So if there is a problem, it might be due to the specific nature
of your mapinfo file, or more likely on the postgres driver side of things.
I didn't try valgrind on a load to postgres since I suspect memory leak
testing would get messy there. 

I would suggest you try a few experiments (mapinfo to shape, shape to
postgres, etc) in an effort to isolate the which driver is likely responsible.
Also, do what you can to rule unhelpful system memory monitors. 

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent