[postgis-tickets] [PostGIS] #4737: Reduce memory usage during MVT encoding

[PostGIS]

#4737: Reduce memory usage during MVT encoding
--------------------------+---------------------------
  Reporter:  Algunenano   |      Owner:  Algunenano
      Type:  enhancement  |     Status:  assigned
  Priority:  medium       |  Milestone:  PostGIS 3.1.0
 Component:  postgis      |    Version:  master
Resolution:               |   Keywords:
--------------------------+---------------------------

Comment (by Algunenano):

 Some extra improvements on top of this:

 If we use protobuf classes instead of our own in the hash tables. Instead
 of doing an intermediate class and later create a VectorTile__Tile__Value,
 use VectorTile__Tile__Value directly. No noticeable impact in memory usage
 (expected) but I do see an impact on performance, an extra ~7% improvement
 on top of the previous improvement (5-10%) from using less memory.

 Also, for parallel queries there are a couple of interesting changes:

 * Use a temporal memory context (child of the aggregation context) to
 process rows (in pgis_asmvt_transfn). This context is then deleted when
 pgis_asmvt_serialfn is called as we no longer need all the extra buffer.
 This ensures a cleanup is done and helps reduce the memory in the
 main/leader backend during the intermediate steps.

 * When combining tiles, do shallow copies and not full copies. Since both
 sources (mvt_agg_context) and destination (also a mvt_agg_context) live in
 the same memory context (aggcontext) we can keep pointers to them so long
 we don't free the sources before we generate the final buffer.


 I have everything more or less done and I need to do some more testing
 (mainly because we don't have parallelism test for mvts so I've done them
 manually for now) but with all the changes together for a really big MVT:
 - Before: Query time: ~5.3s. Peak memory usage¹: ~6GB
 - Now: Query time: ~3.7s. Peak memory usage¹: ~3.6GB

 Attaching the current measurements (master == current, protoc8 == with the
 changes described in the description and this comment).

 1 - The memory usage is sum of the memory ussed by all active PG backends
 done via psrecord.

-- 
Ticket URL: <https://trac.osgeo.org/postgis/ticket/4737#comment:1>
PostGIS <http://trac.osgeo.org/postgis/>
The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project.