[pgrouting-users] pgRouting and multi-core

[Stephen Woodbridge]

On 7/23/2013 9:43 AM, Yaron Lev wrote:
> thanks a lot for your in-depth explanation, it helped me a lot and indeed i
> was referring to the c/c++ part.
>
> the basic question i am  trying to understand, how beneficial it will be to
> increase the DB instance size (cpu wise) (i am using AWS).
>
> just to be clear, and re-iterating, each request has two basic parts:
>
> 1. grab data, and build the graph on the db - this is done on the
> PostgreSQL db, and therefor will benefit from multi-core environment.

correct

> 2. solving and returning the answer  -
> here i am a little lost. PostgreSQL allow integration of C, i would expect
> then, that these extensions(pgRouting code) will also benefit from
> multi-core environment.
> (i.e each request will use a different thread if needed, and on different
> core if needed)

yes, each request runs in a separate db instance or thread.

> so in short, will part 2, also benefit from increasing the instance size(i
> understand it consumes roughly 30% of resources needed to complete the
> request)?

The C/C++ code *mostly* does its own memory allocation and does not rely 
on the memory pooling that postgresql provides. some parts of the 
progress to use this but the graphs and solution memory is allocated 
outside the postgresql memory pooling.

In general if you are not running a lot of other processes on the server 
you can allocate more memory to postgresql, how much memory the graphs 
use depends on the typical size of a graph. I have never tried to figure 
this out. I would go with the standard memory sizing suggestions you can 
find for postgresql tuning. If you need a number to start with, on an 
8GB memory system you might try 4GB in shared buffers and watch it under 
load with top to see how memory is being used and how much swap is 
getting used, how large the processes are getting, etc. and then make 
adjustments from there.

Also you can optimize your data storage by clustering on the edge table 
spatial index. This will minimize IO when selecting edges for a graph 
because edges that are spatially close together will get clustered on 
pages, and once you need one edge on a page it is likely you will need 
multiple edges on that page and page caching is done in the shared buffers.

-steve

> again, many thanks for your help.
>
>
>
>
> On Tue, Jul 23, 2013 at 3:39 PM, Stephen Woodbridge <woodbri at swoodbridge.com
>> wrote:
>
>> On 7/23/2013 7:55 AM, Yaron Lev wrote:
>>
>>> Hi,
>>>
>>> i was looking for some information about weather or not pgRouting is
>>> utilizing more than one core if available.
>>>
>>> from the limited testing i made, and from googling (i found very little
>>> refrences), i think pgRouting does not support multi-core processing as of
>>> now.
>>>
>>> i will appreciate if anyone can confirm it, or give any information about
>>> it.
>>>
>>
>> PgRouting is built on top of Postgresql so to the extent that postgresql
>> can use multiple cores to process multiple requests we do. I you are asking
>> if the routing algorithm itself is run in threads or using something like
>> parallel graph algorithms? the answer is no.
>>
>> Also you should be aware the solving the graph is less then 50% of the
>> cost of answering a query because we have to query the database for the
>> edges, build the graph, solve the graph and return the results back to the
>> client. selecting the edges and building the graph likely take 2/3rds of
>> the processing time for a given query. The edge selection is probably done
>> in a separate core/thread/process by the database from that that the rest
>> of the processing is taking place in.
>>
>> -Steve
>>
>>
>> ______________________________**_________________
>> Pgrouting-users mailing list
>> Pgrouting-users at lists.osgeo.**org <Pgrouting-users at lists.osgeo.org>
>> http://lists.osgeo.org/**mailman/listinfo/pgrouting-**users<http://lists.osgeo.org/mailman/listinfo/pgrouting-users>
>>
>
>
>
> _______________________________________________
> Pgrouting-users mailing list
> Pgrouting-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/pgrouting-users
>