[gdal-dev] Optimizing access to shapefiles

[Yevgen Antymyrov]

On 07/19/2010 02:08 PM, Martin Dobias wrote:
> Hi,
>
> in order to speed up rendering in QGIS as a part of my GSoC project,
> I've took some time to profile reading of shapefiles in OGR. From the
> results I'd like to suggest some changes that significantly contribute
> to the speed of data retrieval. On a test shapefile of a road network
> (about 100 thousand polylines), I have seen 3-4 times faster retrieval
> when I've implemented the following changes:
>
> 1. allow users of OGR library set which fields they really need. Most
> of time is wasted by fetching all the attributes, but typically none
> or just one attribute is necessary when rendering. For that, I've
> added the following call:
> OGRLayer::SetDesiredFields(int numFields, int* fields);
> The user passes an array of ints, each item tells whether the field
> should be fetched (1) or not (0). The numFields tells the size of the
> array. If numFields<  0 then the layer will return all fields (default
> behavior). The driver implementation then just before fetching a field
> checks whether to fetch the field or not. This optimization could be
> easily used in any driver, I've implemented it only for shapefiles.
> The speedup will vary depending on the size of the attribute table and
> number of desired fields. On my test shapefile containing 16 fields,
> the data has been fetched up to 3x faster when no fields were set as
> desired.

I have a couple of questions.

Comparing to other Shapefile tools like Arc* and TatukGIS, the OGR-based tool Quantum GIS 
has really bad performance because it just does too much. Do you think your solution can 
be applied to Qgis? I mean when displaying geometry, it can call SetDesiredFields() with 
all 0s. And only when user opens Attribute Table, Qgis can re-get all/selected fields. 
Will this be possible with your patch?


-- 
Yevgen