[postgis-tickets] [PostGIS] #4737: Reduce memory usage during MVT encoding

[PostGIS]

#4737: Reduce memory usage during MVT encoding
-------------------------+---------------------------
 Reporter:  Algunenano   |      Owner:  Algunenano
     Type:  enhancement  |     Status:  assigned
 Priority:  medium       |  Milestone:  PostGIS 3.1.0
Component:  postgis      |    Version:  master
 Keywords:               |
-------------------------+---------------------------
 During the last months I've received multiple reports of excesive memory
 usage of Postgres during MVT generation.

 Although these situations are extreme, as MVTs are not meant for retrieval
 of dozens or even hundreds of MBs at one, they are hard to deal with since
 it's normal for the OOM killer to do its job and kill the leader backend
 id, which triggers a database restart.

 I'm still investigating how the memory contexts work during the
 aggregation (since some memory seems to not go away), but one thing I've
 discovered is that the current (spec'd) proto is not very memory efficient
 as it's declaring  `message Value` as being able to hold values of
 different types at once (so it can hold a string, a float, a double, an
 int, ..., all at the same time). This is prohibited by the spec, but it's
 just a comment in the .proto so it's not enforced.
 Since protobuf-c 1.1+, you can use the `oneof` clause to avoid this, which
 trasnsforms the resulting C code into a union, which is way more efficient
 in terms of memory. I've tested this and I've seen a 20-25% decrease in
 memory usage when generating them (`encode_values`).

 What I still not sure about is the best way to apply this. I can either
 change the .proto and raise the minimum requirement for protobuf-c, or I
 can generate the resulting .c|.h and push them in the repo (so they are
 stable and not generated every time).

-- 
Ticket URL: <https://trac.osgeo.org/postgis/ticket/4737>
PostGIS <http://trac.osgeo.org/postgis/>
The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project.