[Qgis-developer] memory usage by attribute table

[Tom Elwertowski]

Tara Athan wrote:
> I frequently use a large polygon shapefile with +50,000 records in a dbf 
> file of 87Mb, which would crash v0.10, so that was the first thing I 
> checked in my new build.
> QGIS v0.11 is able to open the file (congratulations).

This is probably due to upgrading from Qt3 to Qt4 table classes.

> But when I open it in QGIS v0.11, I see a memory hit of 1.2GB, around 
> 13x the size of the file.

I would expect at hit of at least 3x to 4x. 13x seems a bit high but not totally unexpected since we are using generic table classes.

A custom data storage class can be written for Qt4 tables. It appears that this would be worthwhile for large files.

Another possible optimization for Qt4 tables is to load only the data visible on the screen. This can only be done for formats which support random access, however. SHP files qualify but other QGIS formats do not. If we go this route, we then need to maintain two table data storage classes: one for formats which support random access and another for formats which don't.

I removed the Qt3 classes so that QGIS can be built as a 64bit Mac application. After QGIS 1.0, the QGIS community should compare and prioritize this and other optimizations and new features for QGIS.

Tom