[pgrouting-dev] withPoints(): possible memory leak?

[Clemens Raffler]

Dear pgrouting dev team,

I am hitting a problem with the pgr_withPoints function family and 
wanted to ask you for further guidance and/or advice on how to deal with 
it.

First I would like to outline  the task I am working on:
I am calculating an origin destination matrix for two sets of points 
(eg. The start_points and end_points), so I am particularly interested 
in retrieving the cost between each startpoint and endpoint using the 
pgr_withPointsCost() function. As a cost factor I am normally using 
precalculated time-costs (seconds) as cost and reverse_cost depicting 
the (real) time a cyclist needs to traverse an edge (I will call those 
costs real_cost). In order to model a cyclists route choice more 
accurately I am introducing a multiplicator for costs on edges that seem 
unpleasant to ride on. Costs on those edges get multiplied by 100 in 
order to retrieve a more realistic routing output. As a result of this 
extra modelling, the pgr_withPointsCost() function aggregates costs 
based on the multiplied cost attributes and do not reflect the initial 
time (eg. real_cost) along the path properly.

My approach to solve this problem would be to join and aggregate the 
real time costs (which is stored in the input graph table) costs along 
the manipulated routing output involves the use of pgr_withPoints() 
function. In more detail: This function allows me to first store the 
individual path elements of the routes in a result table. Then I would 
like to join the real cost based on the edge attributes of the routing 
output and the input graph table and group by start_pid and end_pid 
while applying sum(real_cost).

But when changing the pgr_withPointsCost() function to pgr_withPoints() 
function I repeatedly run into some heavy memory leakage (different 
errors that seem to occur in the pgr_withPoints() function). I can give 
you some details on the tests I did:

1)    Testrun with create table as pgr_withPointsCost() using a graph 
with ~50000 edges, ~4000 Start and End pids: completes without errors 
(although 99% of memory is used).
2)    Testrun with create table as pgr_withPoints() using a graph with 
~50000 edges, ~4000 Start and End pids: ERROR std::bad_alloc Hint: 
Working with directed Graph.
3)    Testrun with create table as pgr_withPoints() using a graph with 
~50000 edges, 100 Start and ~4000 End pids: ERROR invalid memory alloc 
request size 3474796248 Where: SQL-Funktion »pgr_withpoints«
4)    Testrun with create table as pgr_withPoints() using a graph with 
~50000 edges, 10 Start and ~4000 End pids:  completes without errors (50 
sec)

I checked the whole issues history of pgrouting on github and found 
tests with way more startpoints (around 80000, but also using 
pgr_withPointsCost() and way more RAM) – maybe it is related: 
https://github.com/pgRouting/pgrouting/issues/694#issuecomment-288035720

Are you familiar with this kind of behaviour of pgr_withPoints(), which 
ultimately calls just pgr_dijkstra()?  Is this a memory leak or do I 
just not have enough RAM (24GB)? Do you have any hints on how to solve 
this issue or do you experience similar problems? I would like to avoid 
cutting the query into smaller chuncs of start_points and iterate over 
them as it is very time inefficient to run such queries.

I am currently running Postgresql Version 10.3, 64 bit, PostGIS v. 2.4 
and pgrouting 2.6.0, release/2.6.1.59.0 on a Windows 10, 64bit Machine 
with 24GB of RAM. I will also try updating a test system to the current 
postgres, postgis and pgrouting versions and run the query again.

I would be glad if you could have a look into this.

Best regards,
Clemens