[postgis-tickets] [PostGIS] #4737: Reduce memory usage during MVT encoding
PostGIS
trac at osgeo.org
Fri Aug 7 02:33:52 PDT 2020
#4737: Reduce memory usage during MVT encoding
-------------------------+---------------------------
Reporter: Algunenano | Owner: Algunenano
Type: enhancement | Status: assigned
Priority: medium | Milestone: PostGIS 3.1.0
Component: postgis | Version: master
Keywords: |
-------------------------+---------------------------
During the last months I've received multiple reports of excesive memory
usage of Postgres during MVT generation.
Although these situations are extreme, as MVTs are not meant for retrieval
of dozens or even hundreds of MBs at one, they are hard to deal with since
it's normal for the OOM killer to do its job and kill the leader backend
id, which triggers a database restart.
I'm still investigating how the memory contexts work during the
aggregation (since some memory seems to not go away), but one thing I've
discovered is that the current (spec'd) proto is not very memory efficient
as it's declaring `message Value` as being able to hold values of
different types at once (so it can hold a string, a float, a double, an
int, ..., all at the same time). This is prohibited by the spec, but it's
just a comment in the .proto so it's not enforced.
Since protobuf-c 1.1+, you can use the `oneof` clause to avoid this, which
trasnsforms the resulting C code into a union, which is way more efficient
in terms of memory. I've tested this and I've seen a 20-25% decrease in
memory usage when generating them (`encode_values`).
What I still not sure about is the best way to apply this. I can either
change the .proto and raise the minimum requirement for protobuf-c, or I
can generate the resulting .c|.h and push them in the repo (so they are
stable and not generated every time).
--
Ticket URL: <https://trac.osgeo.org/postgis/ticket/4737>
PostGIS <http://trac.osgeo.org/postgis/>
The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project.
More information about the postgis-tickets
mailing list