[pgrouting-dev] out of memory with kdijkstra

[yobiSource]

Thanks for the response.

I have collected all the data in a Gist:
https://gist.github.com/yobiSource/ab024968aeb390452825

in short:
Ubuntu 12.04 x64
Mem/Swap: 16GB/16GB
PostgreSQL 9.3.5 (PostGis 2.1.4)
shared_buffers = 3GB
edge_table (OSM data) count=1733763 size=246MB
smin=1 smax=1606592 tmin=1 tmax=1606590 gmin=1 gmax=1733763

The query [1] uses the (optimized) function pgr_vidstodmatrix from dev. 
After 5:20 hours, the most memory and swap is in use.
The memory is released only when the connection is closed, even when the 
execution is terminated!

I'm creating a ticket after Christmas.

I noticed in k_targets_boost_wrapper line 36-49 possibly a solution but 
i can't compile on the production-server.

[1] https://gist.github.com/yobiSource/ab024968aeb390452825#file-sql
[2] 
https://github.com/pgRouting/pgrouting/blob/develop/src/kdijkstra/src/k_targets_boost_wrapper.cpp

Am 23.12.2014 um 02:31 schrieb Stephen Woodbridge:
> On 12/22/2014 8:06 PM, yobiSource wrote:
>> Hi,
>>
>> is there any progress in memory management? [1]
>>
>> I create 5000 times a distance matrix (pgr_kdijkstra) with ~300 points
>> and run oom. (>20 GB)
>>
>> I avoid that by just split the data and open multiple connections (one
>> after another) but it is a bit complicated.
>>
>>
>> [1]
>> https://github.com/pgRouting/pgrouting/issues?q=is%3Aissue+is%3Aopen+memory
>>
>>
>
> Hi yobi,
>
> I can appreciate how frustrating issues are like this. They are for the
> developers also. Maybe you should open a ticket for this issue and
> decribe what you are doing in some detail. Ideally getting a small
> database dump and a query that reproduces problem would be very helpful.
>
> While there are other commands that have memory issues, they are most
> likely not related to your OOM issue. Information you should post in the
> ticket includes:
>
> OS: Windows, Linux
>
> Hardware: 32/64 bit, Memory, amount of swapfile space
>
> How many shared_buffers in your postgresql.conf file?
>
> How many edges in your graph?
>     select count(*) from edge_table;
>
> edge and node ids:
>     select min(source) as smin, max(source) as smaxm, min(target) as
> tmin, max(target) as tmax, min(gid) as gmin, max(gid) as gmax
>      from edge_table;
>
> What is your query that is running out on memory?
>
> Describe the steps to reproduce the problem.
>
> Then you can refer to the ticket number when asking about the issue and
> all the data is in one place.
>
> As far as progress, we have been tied up on other projects because we
> get very little funding for pgrouting, but with some more detailed
> information we might be able to help you work around the issue.
>
> Also you might want to look at the tools in github branch develop in
> src/common/sql/pgrouting_conversion_tools.sql which has a lot of tools
> for building distance matrices.
>
> Also src/kdijkstra/src/k_targets_sp.c which has some new stuff that is
> not in the tagged 2.0.0 release.
>
> -Steve
> _______________________________________________
> pgrouting-dev mailing list
> pgrouting-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/pgrouting-dev


-- 
Cheers
yobiSource