[GRASS-user] v.net parallelisation issues

[Moritz Lennert]

On 13/02/15 13:40, Mark Wynter wrote:
> Hi Moritz
>
> With the second approach  (the code I shared in my post), I have 3500 discrete jobs, and I set the number of batches equal to the number of CPUs.  Each batch job is despatched to a cpu, where it then pulls from a queue of job id’s that are processed in serial within each batch job.  The thinking behind this approach was to allocate jobs across available CPUs as separate batch processes.
>
> The other and preferred approach is to launch 1 batch job, and then GNU parallel draws down from the list of 3500 jobs, assigning jobs to worker functions as CPUs become available.  This code pattern I’ve had much success with parallelising PostGIS queries etc.
>
> As you have suspected, I get no benefit from additional CPUs.

Are you sure the problem is CPU-bound ?

>
> Unfortunately I don’t have time on my side, and parallelisation is critical.  A fallback is to spin up a cluster of 16 x 2 CPU machines and pre-allocate job-ids to machines, and then write the results back to the master node - but this is not ideal and pathway I am reticent about going down.
>
> Do you know anyone who may have attempted to parallelise v.net?

No. Personally I don't have any experience with this.
You are specifically speaking about v.net.distance, here, or ?

>
> I guess the most important question right now is - is it possible to do poor man’s parallelisation with v.net?   Anyone?

The one who knows the insides of these modules best is Markus Metz.

Moritz