[Qgis-developer] twkb (Tiny WKB) in QGIS?

[kimaidou]

Hi Nicklas,

This is indeed a great idea, and will go on further on the step already
done here [1] with PostGIS 2.2 backend. QGIS 2.14 now uses
ST_RemoveRepeatedPoints as described by one of the optimisations we can use
with PostGIS

As you said, we could only use this format for rendering purpose, and not
for spatial analyses or writing back to the PostgreSQL server.


I do not personaly know how much work is needed to achieve this, but this
would be a great step further to lower bandwich usage and improve
performance.

Cheers
Michaël

[1] https://github.com/qgis/QGIS/pull/2410

2016-03-02 9:24 GMT+01:00 Nicklas Avén <nicklas.aven at jordogskog.no>:

> Hallo all
>
> I have seen this question raised before some time ago on QGIS list.
> That made me glad :-)
>
> But I think maybe time is more right now, since twkb is suppoerted in a
> released PostGIS version.
>
> For anyone not knowing what twkb is:
> TWKB is a geometry format with similarities to wkb, but uses compressen
> the same way as MapBox Vector Tiles.
>
> I started to work with this format maybe 4 years ago.
> I added encoding support in PostGIS trunk some time after PostGIS 2.1
> release.
>
> I will not tell the whole history here, but last spring, in 2015 the
> format went through quite a big face lift. Since CartoDB was interested
> in using the format between their database servers and rendering
> servers, Paul Ramsey looked into the format and did some great
> improvements. He wrote a blog post about the result for CartoDB
> http://blog.cartodb.com/smaller-faster/
>
> Also Javier Santana at CartoDB should be mentioned here since he was
> part of the discussions.
>
> Then with PostGIS 2.2 we now have encoding and decoding of twkb in
> PostGIS.
>
> I have all the time thought that twkb should be a good fit for the
> communication between PostGIS and QGIS.
>
> There is some things to consider though.
> TWKB doesn't preserve the precision in the coordinates. Because of that
> it is not suitable for writing back to the db.
>
> But the increased size (often about 7 times) compared to wkb would give
> a real boost to QGIS. Especially Encoding and decoding is fast. For
> example, writing a table inside PostGIS to a new table like:
> CREATE TABLE foo AS
> SELECT ST_Binary(geom) FROM table1;
>
> is slower than encoding and writing as twkb
> CREATE TABLE foo AS
> SELECT ST_AsTWKB(geom) FROM table1;
>
> But, as mentioned, then decimals is removed.
> The encoder in PostGIS takes a second argument that defines how many
> decimals that should be included. Default is 0 for meter based
> projections and 6 or 5 (I don't remember) for 4326. The precision value
> can also be negative which results in reounded integer values, like -1
> means 10 meters precision (on meter based projections)
>
> So here is my suggestion how to handle things.
>
> As an option for people using PostGIS >= 2.2 as backend, they can switch
> to twkb as transportation format. When zoomed out QGIS asks PostGIS for
> 0 or -1 decimals precision. When zooming in QGIS requests more
> precision. But meter precision is enough for quite deep zooming.
>
> So, the editing issue. When the user opens a layer for editing QGIS will
> have to switch to wkb format. If not the whole geoemtry that is edited
> will loose precision when sent back.
>
> It could be possible of course to edit and accept less precision. For
> GIS-professionals that know how accurate their data is, they know that
> storing 10 decimals is no gain. They could decide that their data set
> only have cm precision and work both ways with twkb with 2 decimals.
>
> The compression is worse with 2 decimals than 0 decimals of course, but
> still smaller than wkb. So one approach could be that there is an option
> to decide with what precision to edit and use twkb as return format too.
>
>
> But as a first step I think decoding at QGIS as a client is a big step
> forward.
>
> I don't have the skills to write this in C++ myself, but everything
> needed exists in C, both in PostGIS and in a small standalone library
> that I have been playing with https://github.com/nicklasaven/twkbC.
>
> But to make things as fast as possible the decoding should of course be
> done directly from twkb into QGIS internal format, with no other
> representations in between.
>
> So, if anyone is interested, I will do all that I can to help. But I
> don't think it would be much faster to get someone on board that knows
> the QGIS code base and know C++, than me trying to do things myself.
>
> So, what do you think?
> Could this be interesting?
> If so, what next? This shouldn't be a very big thing if we find a way to
> work.
>
> I guess if it is interesting it is a candidate for QGIS 3.0
>
>
> Best Regards
>
> Nicklas Avén
>
>
>
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20160302/f412ac74/attachment.html>