[Qgis-developer] shapefile editing memory issue

Tue Jul 17 15:22:54 PDT 2012

Thanks for the quick response.  Since my previous post, I have modified the code to eliminate all the calls to startEditing() and commitChanges().  I thought the memory problems might have been related to memory leaks in those functions.  Now, I'm implementing changeAttributeValues() through the vector data provider to modify the attribute table directly without messing with the layer editing buffer, as recommended here:
http://lists.osgeo.org/pipermail/qgis-developer/2011-July/015385.html

However, some major memory leaks remain.  As recommended in the previous post, I implemented my plug-in under valgrind on a Linux machine.  Some output below highlights the changeAttributeValues() function as a possible culprit:  500,000 bytes 'definitely' lost with only a couple iterations of my algorithm.  It's easy to see how gobs of memory can be lost after many thousand iterations.  Any suggestions for a better way to update an attribute table repeatedly without losing so much memory?  Thanks, Kelly

==31858== 576,500 (23,040 direct, 553,460 indirect) bytes in 360 blocks are definitely lost in loss record 14,582 of 14,586
==31858==    at 0x4A059DC: operator new(unsigned long) (vg_replace_malloc.c:220)
==31858==    by 0x86CB51D: SHPReadOGRFeature(SHPInfo*, DBFInfo*, OGRFeatureDefn*, int, SHPObject*) (in /usr/lib64/libgdal.so.1.14.3)
==31858==    by 0x86A99D4: OGRShapeLayer::GetFeature(long) (in /usr/lib64/libgdal.so.1.14.3)
==31858==    by 0x19A72375: QgsOgrProvider::changeAttributeValues(QMap<int, QMap<int, QVariant> > const&) (in /usr/lib64/qgis/libogrprovider.so)
==31858==    by 0x1F002421: ??? (in /usr/lib64/python2.7/site-packages/qgis/core.so)
==31858==    by 0x1BFE665A: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0)
==31858==    by 0x1BFE771C: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0)
==31858==    by 0x1BFE771C: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0)
==31858==    by 0x1BFE771C: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0)
==31858==    by 0x1BFE804C: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0)
==31858==    by 0x1BF6EC61: ??? (in /usr/lib64/libpython2.7.so.1.0)
==31858==    by 0x1BF45FC2: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0)

-----Original Message-----
From: Tim Sutton [mailto:lists at linfiniti.com] 
Sent: Thursday, July 12, 2012 12:59 PM
To: Thorp, Kelly
Cc: qgis-developer at lists.osgeo.org
Subject: Re: [Qgis-developer] shapefile editing memory issue

Hi

On Thu, Jul 12, 2012 at 8:00 PM, Thorp, Kelly <Kelly.Thorp at ars.usda.gov> wrote:
> Hello List:
>
>
>
> I am developing a QGIS plug-in for the purpose of running simulation models
> for unique land units, as represented by polygons in a polygon shapefile.
> Essentially, the plug-in uses spatial information in the polygon layer to
> write a simulation model input file.  It then uses Python's 'subprocess'
> module to call the model executable and run the simulation.  It then reads
> an output file generated by the model, sets the polygon shapefile to be
> editable, changes some attributes based on the model output, and then
> commits those changes.  Then repeating this in a loop for each feature in
> the polygon shapefile.
>
>
>
> I've also included an optimization routine for the purpose of model
> calibration.  This essentially means doing the above procedures repeatedly
> tens or hundreds of thousands of times.   After each model simulation, the
> updated 'simulated' attribute values are compared with other attributes that
> contain 'observed' data.  The optimization then repeatedly adjusts the model
> parameters until simulated and observed data are in best agreement.
>
>
>
> This process can take many hours or days, while I am repeatedly editing the
> shapefile and commiting the edits.  However, I am not adding or removing any
> features or attributes.  Problem is the memory slowly creeps up and if the
> optimization takes too long, I'll run out of memory and QGIS will crash. I
> notice that if my optimization routine does finish, I can completely shut
> down my plugin's dialog window, but the memory usage remains high.  This
> makes me think the problem is not in my plug-in.  Any ideas for me?
>
>

You would probably need to run your plugin under a memory profiler
like valgrind to see where the leaks are occurring.

My apologies if that is too low level an answer.

Regards

Tim

>
> Thanks,
>
> Kelly
>
>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Kelly Robert Thorp, Ph.D., P.E.
>
> Research Agricultural Engineer
>
> USDA-ARS
>
> U.S. Arid Land Agricultural Research Center
>
> 21881 N Cardon Lane
>
> Maricopa, AZ  85138
>
> 520-316-6375 (Phone)
>
> 520-316-6330 (Fax)
>
> kelly.thorp at ars.usda.gov
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
>
>
>
>
>
> This electronic message contains information generated by the USDA solely
> for the intended recipients. Any unauthorized interception of this message
> or the use or disclosure of the information it contains may violate the law
> and subject the violator to civil or criminal penalties. If you believe you
> have received this message in error, please notify the sender and delete the
> email immediately.
>
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>

-- 
Tim Sutton - QGIS Project Steering Committee Member (Release  Manager)
==============================================
Please do not email me off-list with technical
support questions. Using the lists will gain
more exposure for your issues and the knowledge
surrounding your issue will be shared with all.

Visit http://linfiniti.com to find out about:
 * QGIS programming and support services
 * Mapserver and PostGIS based hosting plans
 * FOSS Consulting Services
Skype: timlinux
Irc: timlinux on #qgis at freenode.net
==============================================