[pgrouting-users] "invalid memory alloc request size" on large graphs

[Phyks]

Hi,

I'm trying to use pgRouting on a quite large graph (about 40M edges). My 
graph structure is expected to be quite simple and not very connected 
(being mostly a simple tree structure).

I came across the discussion on speeding up `pgr_createTopology` for 
large networks 
(https://github.com/pgRouting/pgrouting/discussions/2255#discussioncomment-4739376) 
and am trying to put it in practice for my use case. Moving from 
`pgr_createTopology` to `pgr_extractVertices` gives a huge increase in 
processing time and memory requirements.

However, when trying to run `pgr_*` (`pgr_dijkstra`, `pgr_contraction` 
or `pgr_connectedComponents` for example) on the subsequent graph, I am 
facing a malloc issue : invalid memory alloc request size 1080000000.

This is easily worked around with `pgr_dijkstra` since routing is mainly 
local and the edges table can easily be filtered with a bounding box. 
This is not as easy for `pgr_contraction` or `pgr_connectedComponents` 
where tiling the computation (through bbox) requires some extra 
post-processing.

I'm not sure about the units of the `malloc` error message, but it seems 
to me this is about 1GB allocation which should be no issue on the 
machine and setup I am running. For the record, my postgresql is having 
either a basic configuration (out of the box install) or a pgtune-d one, 
same issue in both cases. My machine has a bit more than 10GB of free RAM.

Would anyone have more insights on this issue which could help me better 
understand and work around it? This is particularly frustrating since 
computations on a quarter of the table (about 10M edges) work perfectly 
fine and in a very reasonable amount of time.

Thank you in advance,
Best,