[pgrouting-dev] out of memory with kdijkstra

[Stephen Woodbridge]

Ok, this is great detail so please post a ticket when you have a chance. 
It clearly sounds like this is a serious memory leak, I took a quick 
look at the source but didn't see anything obvious. Unfortunately, I 
will not have any time in the next month to look at this.

You can't do this on a production machine because it needs to shut down 
the server, but this script runs the backend process under valgrind 
which is a very handy memory checker.

change the path to the postgresql stuff to match your system.

#!/bin/bash

sudo service postgresql stop

echo "SELECT pgr_vidstodmatrix(vertex_ids, 'edge_table', bbox) as dm 
FROM dm_groups" | sudo -u postgres valgrind --leak-check=yes -v 
--undef-value-errors=yes --track-origins=yes --log-file=valgrindlog 
/usr/lib/postgresql/9.3/bin/postgres --single -D 
/etc/postgresql/9.3/main/ your_database_name

sudo service postgresql start


-Steve

On 12/24/2014 1:25 PM, yobiSource wrote:
> Thanks for the response.
>
> I have collected all the data in a Gist:
> https://gist.github.com/yobiSource/ab024968aeb390452825
>
> in short:
> Ubuntu 12.04 x64
> Mem/Swap: 16GB/16GB
> PostgreSQL 9.3.5 (PostGis 2.1.4)
> shared_buffers = 3GB
> edge_table (OSM data) count=1733763 size=246MB
> smin=1 smax=1606592 tmin=1 tmax=1606590 gmin=1 gmax=1733763
>
> The query [1] uses the (optimized) function pgr_vidstodmatrix from dev.
> After 5:20 hours, the most memory and swap is in use.
> The memory is released only when the connection is closed, even when the
> execution is terminated!
>
> I'm creating a ticket after Christmas.
>
> I noticed in k_targets_boost_wrapper line 36-49 possibly a solution but
> i can't compile on the production-server.
>
> [1] https://gist.github.com/yobiSource/ab024968aeb390452825#file-sql
> [2]
> https://github.com/pgRouting/pgrouting/blob/develop/src/kdijkstra/src/k_targets_boost_wrapper.cpp
>
>
> Am 23.12.2014 um 02:31 schrieb Stephen Woodbridge:
>> On 12/22/2014 8:06 PM, yobiSource wrote:
>>> Hi,
>>>
>>> is there any progress in memory management? [1]
>>>
>>> I create 5000 times a distance matrix (pgr_kdijkstra) with ~300 points
>>> and run oom. (>20 GB)
>>>
>>> I avoid that by just split the data and open multiple connections (one
>>> after another) but it is a bit complicated.
>>>
>>>
>>> [1]
>>> https://github.com/pgRouting/pgrouting/issues?q=is%3Aissue+is%3Aopen+memory
>>>
>>>
>>>
>>
>> Hi yobi,
>>
>> I can appreciate how frustrating issues are like this. They are for the
>> developers also. Maybe you should open a ticket for this issue and
>> decribe what you are doing in some detail. Ideally getting a small
>> database dump and a query that reproduces problem would be very helpful.
>>
>> While there are other commands that have memory issues, they are most
>> likely not related to your OOM issue. Information you should post in the
>> ticket includes:
>>
>> OS: Windows, Linux
>>
>> Hardware: 32/64 bit, Memory, amount of swapfile space
>>
>> How many shared_buffers in your postgresql.conf file?
>>
>> How many edges in your graph?
>>     select count(*) from edge_table;
>>
>> edge and node ids:
>>     select min(source) as smin, max(source) as smaxm, min(target) as
>> tmin, max(target) as tmax, min(gid) as gmin, max(gid) as gmax
>>      from edge_table;
>>
>> What is your query that is running out on memory?
>>
>> Describe the steps to reproduce the problem.
>>
>> Then you can refer to the ticket number when asking about the issue and
>> all the data is in one place.
>>
>> As far as progress, we have been tied up on other projects because we
>> get very little funding for pgrouting, but with some more detailed
>> information we might be able to help you work around the issue.
>>
>> Also you might want to look at the tools in github branch develop in
>> src/common/sql/pgrouting_conversion_tools.sql which has a lot of tools
>> for building distance matrices.
>>
>> Also src/kdijkstra/src/k_targets_sp.c which has some new stuff that is
>> not in the tagged 2.0.0 release.
>>
>> -Steve
>> _______________________________________________
>> pgrouting-dev mailing list
>> pgrouting-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/pgrouting-dev
>
>