[gdal-dev] Vector Tiles in OGR

Stefan Keller sfkeller at gmail.com
Mon Feb 1 15:27:19 PST 2016 

Hi Ben

Thanks for explaining - Petr and I are the maintainers of OSM2Vectortiles.
I like projections (and also GeoPackage :-) ).
But reprojection is solved here, since OGR/QGIS can transform on the fly.
This thread is about Petr's question about a reader "for mapbox-like
vector tiles stored in MBTiles or downloaded from a url...".

So I'd like to come back to the issue:
1. of a generic MVT decoder/enoder library (C++ or Python) => Blake?
2. and of a "feature provider" in an interactive environment (like
QGIS) or a file/stream converter (like OGR) => anybody?

:Stefan



2016-02-02 0:04 GMT+01:00 Stadin, Benjamin
<Benjamin.Stadin at heidelberg-mobil.com>:
> Hi Stefan,
>
> A vector tile may contain data for several zoom levels. But what data is
> shown at which zoom level is defined by the styling, not by the data
> itself. It is handled on client side, by the web (or native) tile
> renderer.
>
> The path is the same as with mercator tiles (zoomLevel/x/y):
> https://example.com/17/65535/43602.mvt
>
>
> The tile for zoom level 17 may contain features of class „building“, and
> the style definition may contain a rule that filters type „building“ for
> zoomLevels <= 18.
>
> ~Ben
>
> Am 01.02.16, 20:16 schrieb "gdal-dev on behalf of Stefan Keller" unter
> <gdal-dev-bounces at lists.osgeo.org on behalf of sfkeller at gmail.com>:
>
>>That's good news!
>>We'll be happy to add Python bindings to such a generic C++ library
>>for Mapbox vector tiles.
>>We have still some time (say few weeks?) left here to decide to
>>contribute either to these bindings or to Mapzen's pure Python lib.
>>
>>One reason why I'm insisting on Python (bindings) - besides easier
>>testing with QGIS - is the following:
>>AFAIK with vector tiles we have to deal with quite a new kind of
>>feature provider:
>>One which contains "zoom levels" and one which "suddenly" changes
>>feature class at different zoom levels.
>>If a client like QGIS consumes vector tiles from OGR in a one-shot
>>call, how should the client deal with such "zoom level" dependent
>>issues?
>>So my thinking is, to consume only the most appropriate zoom level,
>>probably only within current map canvas...
>>
>>:Stefan
>>
>>2016-02-01 18:27 GMT+01:00 Flippmoke <flippmoke at gmail.com>:
>>>
>>> Stefan,
>>>
>>> I am not aware of any libraries besides mapnik vector tile that have
>>>been attempting to implement v2 specification. I think mapzen's
>>>implementation might be best? We have been wanting to add vector tiles
>>>to the python mapnik bindings for quite some time but we don't have the
>>>time to do it right now.
>>>
>>> I will try to think this week about how we could make a generic C++
>>>library quickly for Mapbox vector tiles.
>>>
>>> Thanks,
>>>
>>> Blake Thompson
>>>
>>>> On Feb 1, 2016, at 1:06 AM, Stefan Keller <sfkeller at gmail.com> wrote:
>>>>
>>>> Hi Blake
>>>>
>>>> It's clear to me that a good generic c++ library for encoding and
>>>> decoding vector tiles is needed.
>>>> On the other hand, I'd like to experiment with vector tiles and
>>>> related QGIS parallel loading issues for which Python is more
>>>> convenient.
>>>>
>>>> It seems that Mapzen's [1] Python implementation is further along.
>>>> Or do you know if Jesse (or somebody) is still working on Mapbox's
>>>>variant [2] ?
>>>>
>>>> :Stefan
>>>>
>>>> [1] https://github.com/mapzen/mapbox-vector-tile
>>>> [2] https://github.com/mapbox/vector-tile-py
>>>>
>>>>
>>>> 2016-01-31 23:22 GMT+01:00 Flippmoke <flippmoke at gmail.com>:
>>>>> All,
>>>>>
>>>>> The first thing that is going to be required likely is a good generic
>>>>>c++ library for encoding and decoding vector tiles IMO.
>>>>>
>>>>> I have spent a lot of time working on developing the mapnik vector
>>>>>tile library to be a solid implementation, especially around all the
>>>>>work done for 2.0 of Mapbox vector tile specification. I would love to
>>>>>eventually help make this a generalized implementation so that it is
>>>>>portable to a library like GDAL.
>>>>>
>>>>> We have talked about doing something like this at Mapbox but will
>>>>>require a good deal of time, so it hasn't been pushed forward yet. The
>>>>>other issues that come to mind to me is that some of the libraries
>>>>>that mapnik vector tile depends upon most likely need good custom
>>>>>implementations if we wanted to make it more generic.
>>>>>
>>>>> The reason for this is that I would prefer for it to be a header only
>>>>>library with no dependencies. Adding more dependencies for GDAL seems
>>>>>like it could be a mess.
>>>>>
>>>>> Blake Thompson
>>>>>
>>>>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <sfkeller at gmail.com>
>>>>>>wrote:
>>>>>>
>>>>>> Hi Petr, Blake, Even and everybody
>>>>>>
>>>>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles.
>>>>>> I'd already asked Even a similar question.
>>>>>> I'm advising now an intern to implement such a MVT Vector Tiles
>>>>>>reader
>>>>>> client in Python if possible as part of a QGIS plugin.
>>>>>> And I'm interested not to do redundant implementations.
>>>>>> Are there any news on this matter?
>>>>>>
>>>>>> :Stefan
>>>>>> Prof. at HSR and leader of Geometa Lab
>>>>>>
>>>>>> @Even: Regarding OGR MVT development can you send to me (directly) a
>>>>>> contact name of the group which won the bidding?
>>>>>>
>>>>>>
>>>>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal <petr.pridal at klokantech.com>:
>>>>>>> Hi everybody,
>>>>>>>
>>>>>>> thanks for the comments. Regarding the technical details:
>>>>>>>
>>>>>>>> Each Tile is its own discrete dataset therefore the hardest part
>>>>>>>>would be
>>>>>>>> merging of features into the original features across tiles. You
>>>>>>>>could use
>>>>>>>> some complicated merging technique, but that could be very
>>>>>>>>expensive.
>>>>>>>
>>>>>>>
>>>>>>> The driver could require "id" parameter, which specifies name of an
>>>>>>> attribute with unique identifier on the features shared across the
>>>>>>>tiles.
>>>>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique
>>>>>>>non-zero
>>>>>>> "osm_id" in place. The first version of driver could read only
>>>>>>>features with
>>>>>>> such ID. The identifier does not have to be always filled, but
>>>>>>>often it is,
>>>>>>> especially in deeper zoom levels derived purely from OpenStreetMap
>>>>>>>- but
>>>>>>> yeh, a completely general stitching would be much harder.
>>>>>>>
>>>>>>> The clipping + merging of shapes itself can be implemented with
>>>>>>>underlaying
>>>>>>> GEOS library functions.
>>>>>>>
>>>>>>> BTW It is possible to examine vector tiles data in pure JavaScript
>>>>>>>X-Ray
>>>>>>> viewer (no WebGL support on browser required) made in OpenLayers3:
>>>>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>>>>>>>
>>>>>>> or with WebGL-enabled webbrowser with higher performance MapBox GL
>>>>>>>JS
>>>>>>> powered X-Ray:
>>>>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>>>>>>>
>>>>>>> A debug viewer for each vector tile (made in OL3) decoded in pure
>>>>>>> JavaScript:
>>>>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>>>>>>>
>>>>>>> The OGR driver could be also practical for the unsimplified /
>>>>>>>ungeneralised
>>>>>>> MBTiles with OSM data available at:
>>>>>>>http://osmlab.github.io/osm-qa-tiles/
>>>>>>>
>>>>>>>>
>>>>>>>> If you wanted to simply view vector tile data it could get
>>>>>>>>confusing as
>>>>>>>> well since most vector tiles need a buffer when they are used for
>>>>>>>> visualization programs. This is important for things like labeling
>>>>>>>>on maps
>>>>>>>> etc. Therefore, you could get a large amount of overlapping data
>>>>>>>>on tiles if
>>>>>>>> you loaded them all in at once.
>>>>>>>
>>>>>>>
>>>>>>> Clipping on border of each tile must be applied before stitching
>>>>>>>the vector
>>>>>>> features together  - so the buffer is removed. Buffer is there only
>>>>>>>for
>>>>>>> visualisation and labeling.
>>>>>>>
>>>>>>>>
>>>>>>>> As far as writing a simple decoder -- the mapnik-vector-tile
>>>>>>>>decoder isn't
>>>>>>>> currently able to decode vector tiles into a form not supported by
>>>>>>>>mapnik.
>>>>>>>> Therefore, OGR would not be able to easily use this.
>>>>>>>
>>>>>>>
>>>>>>> Correct. It can't be used directly (GDAL can't link against), but
>>>>>>>the
>>>>>>> relevant code can be extracted and reused directly. For basic
>>>>>>>decoding
>>>>>>> critical is the Protobuf library (which is already in GDAL) and
>>>>>>>
>>>>>>>https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector
>>>>>>>_tile.proto
>>>>>>> + the code for converting pixel coordinates in features to relevant
>>>>>>>geo
>>>>>>> coordinates and GEOS-based merging of features.
>>>>>>>
>>>>>>>> We plan to write a generalized encoder and decoder eventually so
>>>>>>>>that any
>>>>>>>> other library can plugin with out the mapnik requirement.
>>>>>>>
>>>>>>>
>>>>>>> This would be amazing.
>>>>>>>
>>>>>>>> If you have any questions specifically about Mapnik Vector Tile or
>>>>>>>>the
>>>>>>>> Mapbox Vector Tile Specification feel free to ask me. I would note
>>>>>>>>that you
>>>>>>>> guys will want to update your vector tiles once the V2
>>>>>>>>specification changes
>>>>>>>> are into Mapnik-Vector-Tile.
>>>>>>>
>>>>>>>
>>>>>>> Cool! Thanks. Our world tiles are generated via Docker containers
>>>>>>>on a
>>>>>>> cluster of computers - using Mapnik via Tilelive directly - so
>>>>>>>upgrades on
>>>>>>> Mapnik-Vector-Tile will propagate to the open-source rendering
>>>>>>>software.
>>>>>>> I expect V2 spec will mean also new MapBox Streets V7, right?
>>>>>>>
>>>>>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I
>>>>>>>>don't
>>>>>>>> see
>>>>>>>> mention of that neither in the MVT spec (
>>>>>>>> https://github.com/mapbox/vector-
>>>>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>>>>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ),
>>>>>>>>although
>>>>>>>> I
>>>>>>>> guess it is just a matter of storing a MVT blob in the tile_data
>>>>>>>>column.
>>>>>>>
>>>>>>>
>>>>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles
>>>>>>>inside -
>>>>>>> and Mapnik reads them directly when rasterising. Storing the vector
>>>>>>>tiles in
>>>>>>> SQLite make sense - and it simplifies deployment, and you can then
>>>>>>>easily
>>>>>>> have complete world on each node of a cluster of tileservers. If
>>>>>>>you don't
>>>>>>> want to upgrade your base map too often, this approach is pretty
>>>>>>>efficient
>>>>>>> and removes the centralised database as typical bottleneck.
>>>>>>>
>>>>>>> I did some tests on vector tiles vs raster in MBTiles and
>>>>>>>documented these
>>>>>>> in README.md at and
>>>>>>>https://github.com/klokantech/vector-tiles-sample before
>>>>>>> we started to work on OSM2VectorTiles project.
>>>>>>> There are also sample data and offline MapBox GL JS viewer in that
>>>>>>>repo.
>>>>>>> Vector tiles can be hosted also "unpacked", just as files in
>>>>>>>folders on a
>>>>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does
>>>>>>>by
>>>>>>> default for raster tiles.
>>>>>>>
>>>>>>> If OGR driver is implemented, the primary source should be probably
>>>>>>>reading
>>>>>>> from an URL via curl.
>>>>>>>
>>>>>>> Reading PBFs from an MBTiles (or another SQLite) is just the most
>>>>>>>practical
>>>>>>> portable alternative to it.
>>>>>>>
>>>>>>> Technically people may store the tiles in different ways - for
>>>>>>>example
>>>>>>> Wikimedia guys (http://maps.wikimedia.org/ and
>>>>>>> https://github.com/kartotherian/) use Cassandra for tile storage, I
>>>>>>>guess
>>>>>>> MapBox internally is also on a different storage for production
>>>>>>>servers ...
>>>>>>> for transfers and streaming tilesets there is the  - but tiles are
>>>>>>>ALWAYS
>>>>>>> exposed via HTTP.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Petr
>>>>>>> --
>>>>>>> Petr Pridal, Ph.D.
>>>>>>> CEO
>>>>>>>
>>>>>>> Klokan Technologies GmbH
>>>>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland
>>>>>>> Tel: +41 (0)41 511 26 12
>>>>>>> Email: info at klokantech.com
>>>>>>> Web: http://www.klokantech.com/
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> gdal-dev mailing list
>>>>>>> gdal-dev at lists.osgeo.org
>>>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>_______________________________________________
>>gdal-dev mailing list
>>gdal-dev at lists.osgeo.org
>>http://lists.osgeo.org/mailman/listinfo/gdal-dev
>

More information about the gdal-dev mailing list