[Qgis-developer] twkb (Tiny WKB) in QGIS?

[Nicklas Avén]

Hallo all

I have seen this question raised before some time ago on QGIS list.
That made me glad :-)

But I think maybe time is more right now, since twkb is suppoerted in a
released PostGIS version.

For anyone not knowing what twkb is:
TWKB is a geometry format with similarities to wkb, but uses compressen
the same way as MapBox Vector Tiles.

I started to work with this format maybe 4 years ago. 
I added encoding support in PostGIS trunk some time after PostGIS 2.1
release.

I will not tell the whole history here, but last spring, in 2015 the
format went through quite a big face lift. Since CartoDB was interested
in using the format between their database servers and rendering
servers, Paul Ramsey looked into the format and did some great
improvements. He wrote a blog post about the result for CartoDB
http://blog.cartodb.com/smaller-faster/

Also Javier Santana at CartoDB should be mentioned here since he was
part of the discussions.

Then with PostGIS 2.2 we now have encoding and decoding of twkb in
PostGIS.

I have all the time thought that twkb should be a good fit for the
communication between PostGIS and QGIS.

There is some things to consider though.
TWKB doesn't preserve the precision in the coordinates. Because of that
it is not suitable for writing back to the db.

But the increased size (often about 7 times) compared to wkb would give
a real boost to QGIS. Especially Encoding and decoding is fast. For
example, writing a table inside PostGIS to a new table like:
CREATE TABLE foo AS
SELECT ST_Binary(geom) FROM table1;

is slower than encoding and writing as twkb
CREATE TABLE foo AS
SELECT ST_AsTWKB(geom) FROM table1;

But, as mentioned, then decimals is removed. 
The encoder in PostGIS takes a second argument that defines how many
decimals that should be included. Default is 0 for meter based
projections and 6 or 5 (I don't remember) for 4326. The precision value
can also be negative which results in reounded integer values, like -1
means 10 meters precision (on meter based projections)

So here is my suggestion how to handle things.

As an option for people using PostGIS >= 2.2 as backend, they can switch
to twkb as transportation format. When zoomed out QGIS asks PostGIS for
0 or -1 decimals precision. When zooming in QGIS requests more
precision. But meter precision is enough for quite deep zooming.

So, the editing issue. When the user opens a layer for editing QGIS will
have to switch to wkb format. If not the whole geoemtry that is edited
will loose precision when sent back. 

It could be possible of course to edit and accept less precision. For
GIS-professionals that know how accurate their data is, they know that
storing 10 decimals is no gain. They could decide that their data set
only have cm precision and work both ways with twkb with 2 decimals.

The compression is worse with 2 decimals than 0 decimals of course, but
still smaller than wkb. So one approach could be that there is an option
to decide with what precision to edit and use twkb as return format too.


But as a first step I think decoding at QGIS as a client is a big step
forward.

I don't have the skills to write this in C++ myself, but everything
needed exists in C, both in PostGIS and in a small standalone library
that I have been playing with https://github.com/nicklasaven/twkbC.

But to make things as fast as possible the decoding should of course be
done directly from twkb into QGIS internal format, with no other
representations in between. 

So, if anyone is interested, I will do all that I can to help. But I
don't think it would be much faster to get someone on board that knows
the QGIS code base and know C++, than me trying to do things myself.

So, what do you think?
Could this be interesting?
If so, what next? This shouldn't be a very big thing if we find a way to
work. 

I guess if it is interesting it is a candidate for QGIS 3.0


Best Regards

Nicklas Avén