[Qgis-developer] memory layer performance while adding many features?

[Raymond Nijssen]

Hi Denis,

Adding all features at once, or 1000 by 1000, speeds up enormously. It 
only takes a few minutes, at a steady speed of 480 fpm. Where adding the 
same dataset 1 by 1 took about 5 hours.

Thanks for pointing this out!

Raymond

On 30-07-15 09:34, Denis Rouzaud wrote:
>  From your code [0] , I would see 2 options:
>
> * add features all at once on the provider using
> QgsVectorDataProvider::addFeatures (the method you currently use)
>
> * work on the layer level rather than the provider, which I believe is
> the recommended way for plugins. Make the layer editable, and add
> features to the edit buffer and commit at once. Or maybe commit 1000 by
> 1000.
>
> I hope this helps.
>
>
>
>
> [0]
> https://github.com/opengeogroep/AERIUS-QGIS-plugins/blob/master/ImaerReader/imaer_reader.py#L228
>
>
> On 07/30/2015 09:20 AM, Raymond Nijssen wrote:
>> Hi Denis,
>>
>> The code is in the ImaerReader plugin in the qgis repo and in github:
>>
>> https://github.com/opengeogroep/AERIUS-QGIS-plugins
>>
>> I'm adding the features one by one.
>>
>> Regards,
>> Raymond
>>
>> On 30-07-15 08:46, Denis Rouzaud wrote:
>>> Hi Raymond,
>>>
>>> Can you show us the code doing this?
>>> Do you commit features one by one ar all at once?
>>>
>>> Best wishes,
>>> Denis
>>>
>>> On 07/29/2015 09:04 AM, Raymond Nijssen wrote:
>>>> Dear developers, ;)
>>>>
>>>> A plugin of mine imports data from a gml file into a memory layer.
>>>> Works fine for tiny gml files, but takes forever on huge ones. Sounds
>>>> plausible maybe, but the relation is not linear.
>>>>
>>>> So I did did some tests and found out that reading and parsing the gml
>>>> and creating the features always goes at the same speed of about 400
>>>> features a second (fps). But when adding these to my memory layer, the
>>>> process slows down tremendously.
>>>>
>>>> I output the fps for every 1000 features and it gave me this graph:
>>>>
>>>> http://terglobo.nl/downloads/memory-layer-performance.png
>>>>
>>>> At the end of the importing process I'm processing just 3 features per
>>>> second.
>>>>
>>>> Is this expected behaviour? Is a memory layer not meant for something
>>>> like this? Or is this a bug?
>>>>
>>>> In the source code I found that for every feature I add to a memory
>>>> layer the index is updated. Could that be inefficient?
>>>>
>>>> Hoping anyone can explain!
>>>>
>>>> Regards,
>>>> Raymond
>>>>
>>>>
>>>
>>>
>>
>
>

-- 
Terglobo
Ampèrestraat 110
5223 CT 's-Hertogenbosch
06-25314983