[OSGeo-Standards] An idea about a compressed binary geometry format
Nicklas Avén
nicklas.aven at jordogskog.no
Wed Aug 21 15:03:35 PDT 2013
Sorry, I meant to post to list last time.
You are absolutely right that it should be investigated if it maybe is
the same thing that we are working on from different angels.
I haven't known about topoJSON that long, I actually heard about it soon
after I started to work on TWKB some months ago.
One thing I think is very promising with TWKB is how efficient the
packing is in PostGIS.
I like that the geoemtries gets packed at the same time as it is read
from the disc.
some numbers from testing on my laptop.
I have a table with roads of Norway. As wkb the geometries uses about
360 mb.
If i create wkb-geometris of the roads and saves it to a new table like:
SELECT ST_AsBinary(geom) from roads_table;
it uses about 6500 ms.
If I instead creates TWKB-geometris like:
SELECT ST_AsTWKB(geom,0,id,'NDR') from roads_table;
it uses 3500 ms and is compressed to 60 mb.
Then we have removed all decimals and only have meter precission.
If we instead saves 2 decimals:
SELECT ST_AsTWKB(geom,2,id,'NDR') from roads_table;
it uses 3900 ms and 104 mb.
My point is that even with internal handling in the database and fast
ssd-discs the disc writing is the bottleneck, not the calculation of
TWKB.
So, I think that TWKB also could be an interesting format for instance
as transporting geometries from a database to a desktop-client like
QGIS.
Then it would be ok to tune the needed amount of decimals depending of
zoom level.
For sending to remote clients I think it is more interesting to send
TWKB with "enough" precision and cache them at the client.
Since I met GIS from the foresters perspective I like the possibility of
using maps off line.
This became a long and messy post. Sorry about that, but now it is time
for bed.
Thanks for the interest.
How can we build some sort of community around development of something
like TWKB? How to make decisions in a good way and so on?
Best Regards
Nicklas
On Wed, 2013-08-21 at 18:22 -0300, P Kishor wrote:
> I should have added in my original email that, of course, TopoJSON is
> not binary. For real binary, it would have to be something like BSON,
> which is only MongoDB specific, for now at least. TopoJSON reduces the
> size of GeoJSON via topology which removes redundant geometry.
> However, it does involve packing and unpacking to and from GeoJSON
> because, as of now, viz libraries such as OpenLayers and Leaflet have
> direct support only for GeoJSON. So, my question again -- instead of
> inventing TWKB (not that there is anything wrong with inventing new
> wheels), why not explore and extend support for TopoJSON?
>
>
> On Wed, Aug 21, 2013 at 6:16 PM, Nicklas Avén
> <nicklas.aven at jordogskog.no> wrote:
> Hallo Puneet
>
> You mean in the picture I see with TWKB?
> I don't know all details abput topoJSON, but from what I
> understand it
> involves a lot of great ideas.
>
> I think a format like TWKB should borrow a lot from topoJSON.
>
> One thing is that I think TWKB is shall support topologies
> too.
>
> The difficult thing is to make it very small for small
> geometries and at
> the same time competent to handle complex geometries.
> I think TWKB can be very efficient for that.
>
> /Nicklas
>
>
> On Tue, 2013-08-20 at 21:06 -0300, Puneet Kishor wrote:
> > Where does TopoJSON fit in this picture?
> >
> >
> >
> >
> >
> > --
> > Puneet Kishor
> > Policy Coordinator for Science and Data
> > Creative Commons
> >
> > On Aug 20, 2013, at 9:01 PM, Jody Garnett
> <jody.garnett at gmail.com>
> > wrote:
> >
> >
> > > Sounds interesting, any alternative to BinaryXML would be
> welcome.
> > >
> > >
> > > The header of Binary XML was very complicated, extending
> into
> > > information such as endian order. I got the impression it
> was set up
> > > around with a server bias. The result was a specification
> which was
> > > difficult to implement as a client, and has seen only one
> > > implementation in open source products I am aware of (in
> the gvSig
> > > community).
> > >
> > >
> > >
> > >
> > > On Tue, Aug 20, 2013 at 6:27 AM, Nicklas Avén
> > > <nicklas.aven at jordogskog.no> wrote:
> > > Hallo
> > >
> > > I do not know if this list is the right place so
> that will
> > > be my first
> > > question.
> > >
> > > I have a naive idea about a compressed binary
> format for
> > > geometries.
> > >
> > > I have done some work to sketch what it is about.
> > >
> > > The name so far is TWKB (Tiny WKB)
> > >
> > > Can this list be the right place to discuss this
> format. It
> > > is just an
> > > embryo, and my hope is that it can be something
> designed by
> > > and hard
> > > connected to the FOSS4G-community.
> > >
> > >
> > > What is done is:
> > >
> > > A draft of a spec:
> > >
> https://github.com/nicklasaven/TWKB/blob/master/twkb.md
> > >
> > > including ideas from Evan Rouault and Oliver
> Tonnhofer
> > >
> > >
> > > Demos:
> > > Streaming TWKB from PostGIS to Leaflet via
> websocket.
> > > http://178.79.156.122/twkb_node/
> > >
> > > Streaming TWKB-aggregates (tiles created on the
> fly) from
> > > PostGIS to
> > > Leaflet
> > > http://178.79.156.122/twkb_agg
> > >
> > > A page to compare twkb in timing and size with
> geoJSON
> > > directly from
> > > PostGIS.
> > > http://178.79.156.122/twkb_test
> > >
> > >
> > > Examples of implementation.
> > > The PostGIS part is in the trunk:
> > > http://svn.osgeo.org/postgis/trunk/
> > >
> > > The client parts is exemplified here:
> > > https://github.com/nicklasaven/twkb_web
> > >
> > >
> > > So, first, can this list host discussions about
> TWKB (or
> > > whatever name
> > > it gets)?
> > > If not where should it go?
> > >
> > > Second, any feedback is very welcome
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Nicklas Avén
> > >
> > > _______________________________________________
> > > Standards mailing list
> > > Standards at lists.osgeo.org
> > > http://lists.osgeo.org/mailman/listinfo/standards
> > >
> > >
> > > _______________________________________________
> > > Standards mailing list
> > > Standards at lists.osgeo.org
> > > http://lists.osgeo.org/mailman/listinfo/standards
> > >
>
>
>
>
>
>
>
> --
> Puneet Kishor punkish.org
> Policy Coordinator for Science and Data
> Creative Commons
More information about the Standards
mailing list