[pgrouting-users] "invalid memory alloc request size" on large graphs

[Imre Samu]

>  ... `pgr_connectedComponents` for example) on the subsequent graph,
>  I am facing a malloc issue : invalid memory alloc request size
1080000000.
> My machine has a bit more than 10GB of free RAM.

Try:

1.) set the highest "work_mem" for the session and close all other running
programs ..

SET work_mem = '8GB';
SELECT pgr_createTopology( ... )
-- set back for the default value;
SET work_mem = '1GB';

2.) Linux:  Enable Swap space (  ~ 20GB  or  2x of your RAM )
https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04

3.) optimize table variables storage spaces ( to less space )
    - change your bigint variables to integer
    for (about 40M edges) an integer  [  -2147483648 to +2147483647 ]
should enought
    https://www.postgresql.org/docs/current/datatype-numeric.html

Regards,
 Imre

Phyks <phyks+pgrouting at phyks.me> ezt írta (időpont: 2023. jan. 21., Szo,
14:06):

> Hi,
>
> I'm trying to use pgRouting on a quite large graph (about 40M edges). My
> graph structure is expected to be quite simple and not very connected
> (being mostly a simple tree structure).
>
> I came across the discussion on speeding up `pgr_createTopology` for
> large networks
> (
> https://github.com/pgRouting/pgrouting/discussions/2255#discussioncomment-4739376)
>
> and am trying to put it in practice for my use case. Moving from
> `pgr_createTopology` to `pgr_extractVertices` gives a huge increase in
> processing time and memory requirements.
>
> However, when trying to run `pgr_*` (`pgr_dijkstra`, `pgr_contraction`
> or `pgr_connectedComponents` for example) on the subsequent graph, I am
> facing a malloc issue : invalid memory alloc request size 1080000000.
>
> This is easily worked around with `pgr_dijkstra` since routing is mainly
> local and the edges table can easily be filtered with a bounding box.
> This is not as easy for `pgr_contraction` or `pgr_connectedComponents`
> where tiling the computation (through bbox) requires some extra
> post-processing.
>
> I'm not sure about the units of the `malloc` error message, but it seems
> to me this is about 1GB allocation which should be no issue on the
> machine and setup I am running. For the record, my postgresql is having
> either a basic configuration (out of the box install) or a pgtune-d one,
> same issue in both cases. My machine has a bit more than 10GB of free RAM.
>
> Would anyone have more insights on this issue which could help me better
> understand and work around it? This is particularly frustrating since
> computations on a quarter of the table (about 10M edges) work perfectly
> fine and in a very reasonable amount of time.
>
> Thank you in advance,
> Best,
> _______________________________________________
> Pgrouting-users mailing list
> Pgrouting-users at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/pgrouting-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/pgrouting-users/attachments/20230121/7a5689e3/attachment.htm>