[Qgis-developer] Will Processing in QGIS 3 support parallelization?

[Neumann, Andreas]

Hi Nyall, 

Interesting. 

Your "thinking aloud" - is this something that would have to wait for
QGIS 4x or could this be interested in 3.x already? Just wondering ... 

>From a user point of view it would be very interesting to have
processing within a layer in update mode without having to create
separate new layers. Also, of course, the parallelization option (for
certain algorithms where feasible). 

Andreas 

On 2017-01-06 00:11, Nyall Dawson wrote:

> On 3 January 2017 at 18:19, Anita Graser <anitagraser at gmx.at> wrote: 
> 
>> Hi,
>> 
>> I'm currently writing a script that could benefit a lot from
>> parallelization. Will Processing in QGIS 3 provide parallelization support?
>> Or is it better to look into how to achieve parallelization independent of
>> Processing, e.g. using Python's multiprocessing library?
> 
> Depends what you're after. When processing is ported to the new task
> manager framework then algorithms will be able to run in parallel
> (where possible). Eg a buffer for one layer can run while a transform
> occurs on another layer.
> 
> If you're after parallelization *within* a single algorithm (Eg
> buffering features using multiple threads) then I'm unaware of any
> plans in place to handle this.
> 
> * That said.... read on for some thinking aloud....
> 
> I think when we port the guts of processing over to c++ then this will
> become relatively straightforward. I'd see this happening:
> - algorithms which operate feature-by-feature inherit a special
> algorithm subclass (say "QgsFeatureBasedAlgorithm" or something) and
> override some base class "QgsFeature processFeature( QgsFeature )"
> method. Eg a buffer alg will implement this to buffer the passed
> feature's geometry and return a new modified feature.
> - QgsFeatureBasedAlgorithm could take advantage of something like
> QtConcurrent::mappedReduced to call processFeature on multiple threads
> and use the result function to write out the results for each
> processFeature call. Nice and (theoretically) easy way to gain
> multithreaded algorithms, and it would be simple to adapt many
> existing algorithms to this (buffer, centroid, transform, translate,
> .... Basically anything which operates on each feature in isolation).
> 
> The side benefit of this refactoring would allow something I've wanted
> for a while - a way for processing algorithms to modify a selection
> inside a layer "in place". Eg select a bunch of polygons, run the
> buffer alg on the selection (not sure of the best UI to expose this!)
> and each selected feature will be buffered. Currently there's no easy
> way to do this in QGIS - you've got to run the alg on a selection and
> get a new layer, then delete the selection, and finally copy features
> back from the new output layer to the source layer. Yuck.
> 
> Nyall
> 
>> Thank you and best wishes,
>> Anita
>> 
>> _______________________________________________
>> Qgis-developer mailing list
>> Qgis-developer at lists.osgeo.org
>> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20170106/1e3855af/attachment.html>