[gdal-dev] Optimizing access to shapefiles

[Chaitanya kumar CH]

Martin,

These changes look quite promising. Please add two tickets at
http://trac.osgeo.org/gdal/newticket , one for each topic, and attach the
patch.

On Mon, Jul 19, 2010 at 5:38 PM, Martin Dobias <wonder.sk at gmail.com> wrote:

> Hi,
>
> in order to speed up rendering in QGIS as a part of my GSoC project,
> I've took some time to profile reading of shapefiles in OGR. From the
> results I'd like to suggest some changes that significantly contribute
> to the speed of data retrieval. On a test shapefile of a road network
> (about 100 thousand polylines), I have seen 3-4 times faster retrieval
> when I've implemented the following changes:
>
> 1. allow users of OGR library set which fields they really need. Most
> of time is wasted by fetching all the attributes, but typically none
> or just one attribute is necessary when rendering. For that, I've
> added the following call:
> OGRLayer::SetDesiredFields(int numFields, int* fields);
> The user passes an array of ints, each item tells whether the field
> should be fetched (1) or not (0). The numFields tells the size of the
> array. If numFields < 0 then the layer will return all fields (default
> behavior). The driver implementation then just before fetching a field
> checks whether to fetch the field or not. This optimization could be
> easily used in any driver, I've implemented it only for shapefiles.
> The speedup will vary depending on the size of the attribute table and
> number of desired fields. On my test shapefile containing 16 fields,
> the data has been fetched up to 3x faster when no fields were set as
> desired.
>
> 2. reuse allocated memory. When a new shape is going to be read within
> shapelib, new OGRShape object and its coordinate arrays are allocated.
> By reusing one such temporary OGRShape object within a layer together
> with the coordinate arrays (only allowing them to grow - to
> accommodate larger shapes), I have obtained further speedup of about
> 30%.
>
> I'm attaching patch for both cases. I'd like to hear from you if there
> is interest in making the OGR library faster using the suggested
> strategies. I don't expect the patch to be applied as-is since it is
> kind of a quick hack, though I hope it can serve well for a
> discussion.
>
> If there is interest, I think of some further optimizations by reusing
> OGRFeature instances and possibly also geometries - I expect further
> performance improvement of 10-20% in read access to features for all
> drivers.
>
> Regards
> Martin
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>



-- 
Best regards,
Chaitanya kumar CH.
/tʃaɪθənjə/ /kʊmɑr/
+91-9494447584
17.2416N 80.1426E
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20100719/1a62f714/attachment.html