[PostGIS] #5798: Out of memory: Killed process

[PostGIS]

#5798: Out of memory: Killed process
--------------------------------+---------------------------
  Reporter:  Lars Aksel Opsahl  |      Owner:  strk
      Type:  defect             |     Status:  new
  Priority:  medium             |  Milestone:  PostGIS 3.5.1
 Component:  topology           |    Version:  3.4.x
Resolution:                     |   Keywords:
--------------------------------+---------------------------
Comment (by Lars Aksel Opsahl):

 I tested with ST_RemEdgeNewFace instead off ST_RemEdgeModFace and it was
 the same behavior. One thing notiched was that did see a lot off LWLock
 for pg_stat_statements, so I removed pg_stat_statements, but the memory
 problem continued.

 I went back and checked some old scripts and am I not sure that I haves
 tested wth 200 paralell threads on this server before.

 With 200 threds in parallel I run the 1300 jobs with memory usage around
 100 GB and I do finish most the jobs yes, but not all before running out
 off memory.

 When running with 60 threads in parallel the memory usage is around 60 GB
 for the same amount off jobs, but all jobs are done. When we check the
 logs we find this :

 Some jobs takes a long time (almost 3 hours) and needs a lot off memory to
 remove edges.
 Done clean small polygons for face_edges_to_remove_list size 7196 for
 face_table_name tmp_dyrkbarjord_10_c3_037_689.face at 2024-10-17
 00:00:39.899873+02 used_time: 8681.734

 While the job below remove twice as many faces in about 100 seconds.
 Done clean small polygons for face_edges_to_remove_list size 16502 for
 face_table_name tmp_dyrkbarjord_10_c3_037_725.face at 2024-10-16
 21:38:11.523127+02 used_time: 105.92985

 But we are able move this 1309 topologies which involves around 5 million
 faces and more than 100 millions edges into a single topology with 900.000
 thousand faces (There are still some errors in the result that I have to
 check out).

 So I think you are right there are no clear memory leaks but some times
 topology needs a lot memory and time.

 So I close this one.
-- 
Ticket URL: <https://trac.osgeo.org/postgis/ticket/5798#comment:10>
PostGIS <http://trac.osgeo.org/postgis/>
The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project.