[gdal-dev] Vector Tiles in OGR

Mon Feb 1 14:48:32 PST 2016

All,

I¹d like to share a few thoughts about the current state of vector tiles.
I believe that, with vector data, there is a chance to get rid of some of
the problems that current mapping tools face in practice, especially in
regards to projections. And this chance should be used in my opinion,
instead of doing things the "old way³, as with MVT.

The current specification and implementation of MVT is both depending on a
certain projection when the tiles are created, and with the
relative-to-screen coordinates per tile it also depends on how the
renderer is implemented (e.g. Commonly in other OpenGL applications, as in
our OpenGL based map, input data, screen scale and perspective
transformation are mostly handled in the GL shaders).
The dependency on a specific projection also means that none of the issues
occurring in practice when dealing with map projections gets fixed. Tom
MacWright from MapBox gave his feedback in the comment section at [1]
about why MapBox has limited support for other projections. He nails it
mostly, but I draw different conclusions. It is well possible to separate
data from representation with vector tiles, and thus a general purpose
vector tile format must deal with.

This isn¹t a rant against MapBox Vector Tiles or Mapbox. This isn¹t an
easy thing. I just want to raise awareness that MVT is, currently, a
particular implementation for a particular rendering engine.
 It works well for Mapbox maps, and it would be nice to have conversion
support for MVT. But I think that, without major braking changes, this
should not be considered to be a general purpose vector format at all.
This seems to be what some are hoping for.

There is enough interest from our side that we¹d be interested in
contributing (maybe monetary or with code, or both) to define and
implement a better general purpose tile format.

~Ben

[1] https://vis4.net/blog/posts/no-more-mercator-tiles/

Am 01.02.16, 20:16 schrieb "gdal-dev on behalf of Stefan Keller" unter
<gdal-dev-bounces at lists.osgeo.org on behalf of sfkeller at gmail.com>:

>That's good news!
>We'll be happy to add Python bindings to such a generic C++ library
>for Mapbox vector tiles.
>We have still some time (say few weeks?) left here to decide to
>contribute either to these bindings or to Mapzen's pure Python lib.
>
>One reason why I'm insisting on Python (bindings) - besides easier
>testing with QGIS - is the following:
>AFAIK with vector tiles we have to deal with quite a new kind of
>feature provider:
>One which contains "zoom levels" and one which "suddenly" changes
>feature class at different zoom levels.
>If a client like QGIS consumes vector tiles from OGR in a one-shot
>call, how should the client deal with such "zoom level" dependent
>issues?
>So my thinking is, to consume only the most appropriate zoom level,
>probably only within current map canvas...
>
>:Stefan
>
>2016-02-01 18:27 GMT+01:00 Flippmoke <flippmoke at gmail.com>:
>>
>> Stefan,
>>
>> I am not aware of any libraries besides mapnik vector tile that have
>>been attempting to implement v2 specification. I think mapzen's
>>implementation might be best? We have been wanting to add vector tiles
>>to the python mapnik bindings for quite some time but we don't have the
>>time to do it right now.
>>
>> I will try to think this week about how we could make a generic C++
>>library quickly for Mapbox vector tiles.
>>
>> Thanks,
>>
>> Blake Thompson
>>
>>> On Feb 1, 2016, at 1:06 AM, Stefan Keller <sfkeller at gmail.com> wrote:
>>>
>>> Hi Blake
>>>
>>> It's clear to me that a good generic c++ library for encoding and
>>> decoding vector tiles is needed.
>>> On the other hand, I'd like to experiment with vector tiles and
>>> related QGIS parallel loading issues for which Python is more
>>> convenient.
>>>
>>> It seems that Mapzen's [1] Python implementation is further along.
>>> Or do you know if Jesse (or somebody) is still working on Mapbox's
>>>variant [2] ?
>>>
>>> :Stefan
>>>
>>> [1] https://github.com/mapzen/mapbox-vector-tile
>>> [2] https://github.com/mapbox/vector-tile-py
>>>
>>>
>>> 2016-01-31 23:22 GMT+01:00 Flippmoke <flippmoke at gmail.com>:
>>>> All,
>>>>
>>>> The first thing that is going to be required likely is a good generic
>>>>c++ library for encoding and decoding vector tiles IMO.
>>>>
>>>> I have spent a lot of time working on developing the mapnik vector
>>>>tile library to be a solid implementation, especially around all the
>>>>work done for 2.0 of Mapbox vector tile specification. I would love to
>>>>eventually help make this a generalized implementation so that it is
>>>>portable to a library like GDAL.
>>>>
>>>> We have talked about doing something like this at Mapbox but will
>>>>require a good deal of time, so it hasn't been pushed forward yet. The
>>>>other issues that come to mind to me is that some of the libraries
>>>>that mapnik vector tile depends upon most likely need good custom
>>>>implementations if we wanted to make it more generic.
>>>>
>>>> The reason for this is that I would prefer for it to be a header only
>>>>library with no dependencies. Adding more dependencies for GDAL seems
>>>>like it could be a mess.
>>>>
>>>> Blake Thompson
>>>>
>>>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <sfkeller at gmail.com>
>>>>>wrote:
>>>>>
>>>>> Hi Petr, Blake, Even and everybody
>>>>>
>>>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles.
>>>>> I'd already asked Even a similar question.
>>>>> I'm advising now an intern to implement such a MVT Vector Tiles
>>>>>reader
>>>>> client in Python if possible as part of a QGIS plugin.
>>>>> And I'm interested not to do redundant implementations.
>>>>> Are there any news on this matter?
>>>>>
>>>>> :Stefan
>>>>> Prof. at HSR and leader of Geometa Lab
>>>>>
>>>>> @Even: Regarding OGR MVT development can you send to me (directly) a
>>>>> contact name of the group which won the bidding?
>>>>>
>>>>>
>>>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal <petr.pridal at klokantech.com>:
>>>>>> Hi everybody,
>>>>>>
>>>>>> thanks for the comments. Regarding the technical details:
>>>>>>
>>>>>>> Each Tile is its own discrete dataset therefore the hardest part
>>>>>>>would be
>>>>>>> merging of features into the original features across tiles. You
>>>>>>>could use
>>>>>>> some complicated merging technique, but that could be very
>>>>>>>expensive.
>>>>>>
>>>>>>
>>>>>> The driver could require "id" parameter, which specifies name of an
>>>>>> attribute with unique identifier on the features shared across the
>>>>>>tiles.
>>>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique
>>>>>>non-zero
>>>>>> "osm_id" in place. The first version of driver could read only
>>>>>>features with
>>>>>> such ID. The identifier does not have to be always filled, but
>>>>>>often it is,
>>>>>> especially in deeper zoom levels derived purely from OpenStreetMap
>>>>>>- but
>>>>>> yeh, a completely general stitching would be much harder.
>>>>>>
>>>>>> The clipping + merging of shapes itself can be implemented with
>>>>>>underlaying
>>>>>> GEOS library functions.
>>>>>>
>>>>>> BTW It is possible to examine vector tiles data in pure JavaScript
>>>>>>X-Ray
>>>>>> viewer (no WebGL support on browser required) made in OpenLayers3:
>>>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>>>>>>
>>>>>> or with WebGL-enabled webbrowser with higher performance MapBox GL
>>>>>>JS
>>>>>> powered X-Ray:
>>>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>>>>>>
>>>>>> A debug viewer for each vector tile (made in OL3) decoded in pure
>>>>>> JavaScript:
>>>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>>>>>>
>>>>>> The OGR driver could be also practical for the unsimplified /
>>>>>>ungeneralised
>>>>>> MBTiles with OSM data available at:
>>>>>>http://osmlab.github.io/osm-qa-tiles/
>>>>>>
>>>>>>>
>>>>>>> If you wanted to simply view vector tile data it could get
>>>>>>>confusing as
>>>>>>> well since most vector tiles need a buffer when they are used for
>>>>>>> visualization programs. This is important for things like labeling
>>>>>>>on maps
>>>>>>> etc. Therefore, you could get a large amount of overlapping data
>>>>>>>on tiles if
>>>>>>> you loaded them all in at once.
>>>>>>
>>>>>>
>>>>>> Clipping on border of each tile must be applied before stitching
>>>>>>the vector
>>>>>> features together  - so the buffer is removed. Buffer is there only
>>>>>>for
>>>>>> visualisation and labeling.
>>>>>>
>>>>>>>
>>>>>>> As far as writing a simple decoder -- the mapnik-vector-tile
>>>>>>>decoder isn't
>>>>>>> currently able to decode vector tiles into a form not supported by
>>>>>>>mapnik.
>>>>>>> Therefore, OGR would not be able to easily use this.
>>>>>>
>>>>>>
>>>>>> Correct. It can't be used directly (GDAL can't link against), but
>>>>>>the
>>>>>> relevant code can be extracted and reused directly. For basic
>>>>>>decoding
>>>>>> critical is the Protobuf library (which is already in GDAL) and
>>>>>> 
>>>>>>https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector
>>>>>>_tile.proto
>>>>>> + the code for converting pixel coordinates in features to relevant
>>>>>>geo
>>>>>> coordinates and GEOS-based merging of features.
>>>>>>
>>>>>>> We plan to write a generalized encoder and decoder eventually so
>>>>>>>that any
>>>>>>> other library can plugin with out the mapnik requirement.
>>>>>>
>>>>>>
>>>>>> This would be amazing.
>>>>>>
>>>>>>> If you have any questions specifically about Mapnik Vector Tile or
>>>>>>>the
>>>>>>> Mapbox Vector Tile Specification feel free to ask me. I would note
>>>>>>>that you
>>>>>>> guys will want to update your vector tiles once the V2
>>>>>>>specification changes
>>>>>>> are into Mapnik-Vector-Tile.
>>>>>>
>>>>>>
>>>>>> Cool! Thanks. Our world tiles are generated via Docker containers
>>>>>>on a
>>>>>> cluster of computers - using Mapnik via Tilelive directly - so
>>>>>>upgrades on
>>>>>> Mapnik-Vector-Tile will propagate to the open-source rendering
>>>>>>software.
>>>>>> I expect V2 spec will mean also new MapBox Streets V7, right?
>>>>>>
>>>>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I
>>>>>>>don't
>>>>>>> see
>>>>>>> mention of that neither in the MVT spec (
>>>>>>> https://github.com/mapbox/vector-
>>>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>>>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ),
>>>>>>>although
>>>>>>> I
>>>>>>> guess it is just a matter of storing a MVT blob in the tile_data
>>>>>>>column.
>>>>>>
>>>>>>
>>>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles
>>>>>>inside -
>>>>>> and Mapnik reads them directly when rasterising. Storing the vector
>>>>>>tiles in
>>>>>> SQLite make sense - and it simplifies deployment, and you can then
>>>>>>easily
>>>>>> have complete world on each node of a cluster of tileservers. If
>>>>>>you don't
>>>>>> want to upgrade your base map too often, this approach is pretty
>>>>>>efficient
>>>>>> and removes the centralised database as typical bottleneck.
>>>>>>
>>>>>> I did some tests on vector tiles vs raster in MBTiles and
>>>>>>documented these
>>>>>> in README.md at and
>>>>>>https://github.com/klokantech/vector-tiles-sample before
>>>>>> we started to work on OSM2VectorTiles project.
>>>>>> There are also sample data and offline MapBox GL JS viewer in that
>>>>>>repo.
>>>>>> Vector tiles can be hosted also "unpacked", just as files in
>>>>>>folders on a
>>>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does
>>>>>>by
>>>>>> default for raster tiles.
>>>>>>
>>>>>> If OGR driver is implemented, the primary source should be probably
>>>>>>reading
>>>>>> from an URL via curl.
>>>>>>
>>>>>> Reading PBFs from an MBTiles (or another SQLite) is just the most
>>>>>>practical
>>>>>> portable alternative to it.
>>>>>>
>>>>>> Technically people may store the tiles in different ways - for
>>>>>>example
>>>>>> Wikimedia guys (http://maps.wikimedia.org/ and
>>>>>> https://github.com/kartotherian/) use Cassandra for tile storage, I
>>>>>>guess
>>>>>> MapBox internally is also on a different storage for production
>>>>>>servers ...
>>>>>> for transfers and streaming tilesets there is the  - but tiles are
>>>>>>ALWAYS
>>>>>> exposed via HTTP.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Petr
>>>>>> --
>>>>>> Petr Pridal, Ph.D.
>>>>>> CEO
>>>>>>
>>>>>> Klokan Technologies GmbH
>>>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland
>>>>>> Tel: +41 (0)41 511 26 12
>>>>>> Email: info at klokantech.com
>>>>>> Web: http://www.klokantech.com/
>>>>>>
>>>>>> _______________________________________________
>>>>>> gdal-dev mailing list
>>>>>> gdal-dev at lists.osgeo.org
>>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>_______________________________________________
>gdal-dev mailing list
>gdal-dev at lists.osgeo.org
>http://lists.osgeo.org/mailman/listinfo/gdal-dev