[pgrouting-dev] out of memory with kdijkstra

[yobiSource]

I was able to run over the holidays that script.
But I had to abort after 66:30 hours.
log output: 
https://gist.github.com/yobiSource/ab024968aeb390452825#file-valgrind

Am 25.12.2014 um 05:09 schrieb Stephen Woodbridge:
> Ok, this is great detail so please post a ticket when you have a chance.
> It clearly sounds like this is a serious memory leak, I took a quick
> look at the source but didn't see anything obvious. Unfortunately, I
> will not have any time in the next month to look at this.
>
> You can't do this on a production machine because it needs to shut down
> the server, but this script runs the backend process under valgrind
> which is a very handy memory checker.
>
> change the path to the postgresql stuff to match your system.
>
> #!/bin/bash
>
> sudo service postgresql stop
>
> echo "SELECT pgr_vidstodmatrix(vertex_ids, 'edge_table', bbox) as dm
> FROM dm_groups" | sudo -u postgres valgrind --leak-check=yes -v
> --undef-value-errors=yes --track-origins=yes --log-file=valgrindlog
> /usr/lib/postgresql/9.3/bin/postgres --single -D
> /etc/postgresql/9.3/main/ your_database_name
>
> sudo service postgresql start
>
>
> -Steve
>
> On 12/24/2014 1:25 PM, yobiSource wrote:
>> Thanks for the response.
>>
>> I have collected all the data in a Gist:
>> https://gist.github.com/yobiSource/ab024968aeb390452825
>>
>> in short:
>> Ubuntu 12.04 x64
>> Mem/Swap: 16GB/16GB
>> PostgreSQL 9.3.5 (PostGis 2.1.4)
>> shared_buffers = 3GB
>> edge_table (OSM data) count=1733763 size=246MB
>> smin=1 smax=1606592 tmin=1 tmax=1606590 gmin=1 gmax=1733763
>>
>> The query [1] uses the (optimized) function pgr_vidstodmatrix from dev.
>> After 5:20 hours, the most memory and swap is in use.
>> The memory is released only when the connection is closed, even when the
>> execution is terminated!
>>
>> I'm creating a ticket after Christmas.
>>
>> I noticed in k_targets_boost_wrapper line 36-49 possibly a solution but
>> i can't compile on the production-server.
>>
>> [1] https://gist.github.com/yobiSource/ab024968aeb390452825#file-sql
>> [2]
>> https://github.com/pgRouting/pgrouting/blob/develop/src/kdijkstra/src/k_targets_boost_wrapper.cpp
>>
>>
>>
>> Am 23.12.2014 um 02:31 schrieb Stephen Woodbridge:
>>> On 12/22/2014 8:06 PM, yobiSource wrote:
>>>> Hi,
>>>>
>>>> is there any progress in memory management? [1]
>>>>
>>>> I create 5000 times a distance matrix (pgr_kdijkstra) with ~300 points
>>>> and run oom. (>20 GB)
>>>>
>>>> I avoid that by just split the data and open multiple connections (one
>>>> after another) but it is a bit complicated.
>>>>
>>>>
>>>> [1]
>>>> https://github.com/pgRouting/pgrouting/issues?q=is%3Aissue+is%3Aopen+memory
>>>>
>>>>
>>>>
>>>>
>>>
>>> Hi yobi,
>>>
>>> I can appreciate how frustrating issues are like this. They are for the
>>> developers also. Maybe you should open a ticket for this issue and
>>> decribe what you are doing in some detail. Ideally getting a small
>>> database dump and a query that reproduces problem would be very helpful.
>>>
>>> While there are other commands that have memory issues, they are most
>>> likely not related to your OOM issue. Information you should post in the
>>> ticket includes:
>>>
>>> OS: Windows, Linux
>>>
>>> Hardware: 32/64 bit, Memory, amount of swapfile space
>>>
>>> How many shared_buffers in your postgresql.conf file?
>>>
>>> How many edges in your graph?
>>>     select count(*) from edge_table;
>>>
>>> edge and node ids:
>>>     select min(source) as smin, max(source) as smaxm, min(target) as
>>> tmin, max(target) as tmax, min(gid) as gmin, max(gid) as gmax
>>>      from edge_table;
>>>
>>> What is your query that is running out on memory?
>>>
>>> Describe the steps to reproduce the problem.
>>>
>>> Then you can refer to the ticket number when asking about the issue and
>>> all the data is in one place.
>>>
>>> As far as progress, we have been tied up on other projects because we
>>> get very little funding for pgrouting, but with some more detailed
>>> information we might be able to help you work around the issue.
>>>
>>> Also you might want to look at the tools in github branch develop in
>>> src/common/sql/pgrouting_conversion_tools.sql which has a lot of tools
>>> for building distance matrices.
>>>
>>> Also src/kdijkstra/src/k_targets_sp.c which has some new stuff that is
>>> not in the tagged 2.0.0 release.
>>>
>>> -Steve
>>> _______________________________________________
>>> pgrouting-dev mailing list
>>> pgrouting-dev at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/pgrouting-dev
>>
>>
>
> _______________________________________________
> pgrouting-dev mailing list
> pgrouting-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/pgrouting-dev


-- 
Cheers
yobiSource