[gdal-dev] Vector Tiles in OGR

Mon Feb 1 15:59:30 PST 2016

Hi Stefan,

"If there would be OGR driver for vector tiles it would make a similar
resource extremely useful also for other purposes in GIS tools - not only
for tileservers and direct visualisation.“

I was thinking you want to do more than just rendering. Then at least, as
Blake also mentioned, the features need to be merged. And I think some
degenerate geometries are to be expected for a few cases. You will also
have multiple polygons cut along tile bounds for what should be a single
polygon. And the introduced points to close a polygon’s path at the tile
bounds may add another inaccuracy when you later reproject the data.

It depends what your goal is. Serious map editing isn’t possible.

~Ben

Am 02.02.16, 00:27 schrieb "Stefan Keller" unter <sfkeller at gmail.com>:

>Hi Ben
>
>Thanks for explaining - Petr and I are the maintainers of OSM2Vectortiles.
>I like projections (and also GeoPackage :-) ).
>But reprojection is solved here, since OGR/QGIS can transform on the fly.
>This thread is about Petr's question about a reader "for mapbox-like
>vector tiles stored in MBTiles or downloaded from a url...".
>
>So I'd like to come back to the issue:
>1. of a generic MVT decoder/enoder library (C++ or Python) => Blake?
>2. and of a "feature provider" in an interactive environment (like
>QGIS) or a file/stream converter (like OGR) => anybody?
>
>:Stefan
>
>
>
>2016-02-02 0:04 GMT+01:00 Stadin, Benjamin
><Benjamin.Stadin at heidelberg-mobil.com>:
>> Hi Stefan,
>>
>> A vector tile may contain data for several zoom levels. But what data is
>> shown at which zoom level is defined by the styling, not by the data
>> itself. It is handled on client side, by the web (or native) tile
>> renderer.
>>
>> The path is the same as with mercator tiles (zoomLevel/x/y):
>> https://example.com/17/65535/43602.mvt
>>
>>
>> The tile for zoom level 17 may contain features of class „building“, and
>> the style definition may contain a rule that filters type „building“ for
>> zoomLevels <= 18.
>>
>> ~Ben
>>
>> Am 01.02.16, 20:16 schrieb "gdal-dev on behalf of Stefan Keller" unter
>> <gdal-dev-bounces at lists.osgeo.org on behalf of sfkeller at gmail.com>:
>>
>>>That's good news!
>>>We'll be happy to add Python bindings to such a generic C++ library
>>>for Mapbox vector tiles.
>>>We have still some time (say few weeks?) left here to decide to
>>>contribute either to these bindings or to Mapzen's pure Python lib.
>>>
>>>One reason why I'm insisting on Python (bindings) - besides easier
>>>testing with QGIS - is the following:
>>>AFAIK with vector tiles we have to deal with quite a new kind of
>>>feature provider:
>>>One which contains "zoom levels" and one which "suddenly" changes
>>>feature class at different zoom levels.
>>>If a client like QGIS consumes vector tiles from OGR in a one-shot
>>>call, how should the client deal with such "zoom level" dependent
>>>issues?
>>>So my thinking is, to consume only the most appropriate zoom level,
>>>probably only within current map canvas...
>>>
>>>:Stefan
>>>
>>>2016-02-01 18:27 GMT+01:00 Flippmoke <flippmoke at gmail.com>:
>>>>
>>>> Stefan,
>>>>
>>>> I am not aware of any libraries besides mapnik vector tile that have
>>>>been attempting to implement v2 specification. I think mapzen's
>>>>implementation might be best? We have been wanting to add vector tiles
>>>>to the python mapnik bindings for quite some time but we don't have the
>>>>time to do it right now.
>>>>
>>>> I will try to think this week about how we could make a generic C++
>>>>library quickly for Mapbox vector tiles.
>>>>
>>>> Thanks,
>>>>
>>>> Blake Thompson
>>>>
>>>>> On Feb 1, 2016, at 1:06 AM, Stefan Keller <sfkeller at gmail.com> wrote:
>>>>>
>>>>> Hi Blake
>>>>>
>>>>> It's clear to me that a good generic c++ library for encoding and
>>>>> decoding vector tiles is needed.
>>>>> On the other hand, I'd like to experiment with vector tiles and
>>>>> related QGIS parallel loading issues for which Python is more
>>>>> convenient.
>>>>>
>>>>> It seems that Mapzen's [1] Python implementation is further along.
>>>>> Or do you know if Jesse (or somebody) is still working on Mapbox's
>>>>>variant [2] ?
>>>>>
>>>>> :Stefan
>>>>>
>>>>> [1] https://github.com/mapzen/mapbox-vector-tile
>>>>> [2] https://github.com/mapbox/vector-tile-py
>>>>>
>>>>>
>>>>> 2016-01-31 23:22 GMT+01:00 Flippmoke <flippmoke at gmail.com>:
>>>>>> All,
>>>>>>
>>>>>> The first thing that is going to be required likely is a good
>>>>>>generic
>>>>>>c++ library for encoding and decoding vector tiles IMO.
>>>>>>
>>>>>> I have spent a lot of time working on developing the mapnik vector
>>>>>>tile library to be a solid implementation, especially around all the
>>>>>>work done for 2.0 of Mapbox vector tile specification. I would love
>>>>>>to
>>>>>>eventually help make this a generalized implementation so that it is
>>>>>>portable to a library like GDAL.
>>>>>>
>>>>>> We have talked about doing something like this at Mapbox but will
>>>>>>require a good deal of time, so it hasn't been pushed forward yet.
>>>>>>The
>>>>>>other issues that come to mind to me is that some of the libraries
>>>>>>that mapnik vector tile depends upon most likely need good custom
>>>>>>implementations if we wanted to make it more generic.
>>>>>>
>>>>>> The reason for this is that I would prefer for it to be a header
>>>>>>only
>>>>>>library with no dependencies. Adding more dependencies for GDAL seems
>>>>>>like it could be a mess.
>>>>>>
>>>>>> Blake Thompson
>>>>>>
>>>>>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <sfkeller at gmail.com>
>>>>>>>wrote:
>>>>>>>
>>>>>>> Hi Petr, Blake, Even and everybody
>>>>>>>
>>>>>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles.
>>>>>>> I'd already asked Even a similar question.
>>>>>>> I'm advising now an intern to implement such a MVT Vector Tiles
>>>>>>>reader
>>>>>>> client in Python if possible as part of a QGIS plugin.
>>>>>>> And I'm interested not to do redundant implementations.
>>>>>>> Are there any news on this matter?
>>>>>>>
>>>>>>> :Stefan
>>>>>>> Prof. at HSR and leader of Geometa Lab
>>>>>>>
>>>>>>> @Even: Regarding OGR MVT development can you send to me (directly)
>>>>>>>a
>>>>>>> contact name of the group which won the bidding?
>>>>>>>
>>>>>>>
>>>>>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal
>>>>>>><petr.pridal at klokantech.com>:
>>>>>>>> Hi everybody,
>>>>>>>>
>>>>>>>> thanks for the comments. Regarding the technical details:
>>>>>>>>
>>>>>>>>> Each Tile is its own discrete dataset therefore the hardest part
>>>>>>>>>would be
>>>>>>>>> merging of features into the original features across tiles. You
>>>>>>>>>could use
>>>>>>>>> some complicated merging technique, but that could be very
>>>>>>>>>expensive.
>>>>>>>>
>>>>>>>>
>>>>>>>> The driver could require "id" parameter, which specifies name of
>>>>>>>>an
>>>>>>>> attribute with unique identifier on the features shared across the
>>>>>>>>tiles.
>>>>>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique
>>>>>>>>non-zero
>>>>>>>> "osm_id" in place. The first version of driver could read only
>>>>>>>>features with
>>>>>>>> such ID. The identifier does not have to be always filled, but
>>>>>>>>often it is,
>>>>>>>> especially in deeper zoom levels derived purely from OpenStreetMap
>>>>>>>>- but
>>>>>>>> yeh, a completely general stitching would be much harder.
>>>>>>>>
>>>>>>>> The clipping + merging of shapes itself can be implemented with
>>>>>>>>underlaying
>>>>>>>> GEOS library functions.
>>>>>>>>
>>>>>>>> BTW It is possible to examine vector tiles data in pure JavaScript
>>>>>>>>X-Ray
>>>>>>>> viewer (no WebGL support on browser required) made in OpenLayers3:
>>>>>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>>>>>>>>
>>>>>>>> or with WebGL-enabled webbrowser with higher performance MapBox GL
>>>>>>>>JS
>>>>>>>> powered X-Ray:
>>>>>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>>>>>>>>
>>>>>>>> A debug viewer for each vector tile (made in OL3) decoded in pure
>>>>>>>> JavaScript:
>>>>>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>>>>>>>>
>>>>>>>> The OGR driver could be also practical for the unsimplified /
>>>>>>>>ungeneralised
>>>>>>>> MBTiles with OSM data available at:
>>>>>>>>http://osmlab.github.io/osm-qa-tiles/
>>>>>>>>
>>>>>>>>>
>>>>>>>>> If you wanted to simply view vector tile data it could get
>>>>>>>>>confusing as
>>>>>>>>> well since most vector tiles need a buffer when they are used for
>>>>>>>>> visualization programs. This is important for things like
>>>>>>>>>labeling
>>>>>>>>>on maps
>>>>>>>>> etc. Therefore, you could get a large amount of overlapping data
>>>>>>>>>on tiles if
>>>>>>>>> you loaded them all in at once.
>>>>>>>>
>>>>>>>>
>>>>>>>> Clipping on border of each tile must be applied before stitching
>>>>>>>>the vector
>>>>>>>> features together  - so the buffer is removed. Buffer is there
>>>>>>>>only
>>>>>>>>for
>>>>>>>> visualisation and labeling.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> As far as writing a simple decoder -- the mapnik-vector-tile
>>>>>>>>>decoder isn't
>>>>>>>>> currently able to decode vector tiles into a form not supported
>>>>>>>>>by
>>>>>>>>>mapnik.
>>>>>>>>> Therefore, OGR would not be able to easily use this.
>>>>>>>>
>>>>>>>>
>>>>>>>> Correct. It can't be used directly (GDAL can't link against), but
>>>>>>>>the
>>>>>>>> relevant code can be extracted and reused directly. For basic
>>>>>>>>decoding
>>>>>>>> critical is the Protobuf library (which is already in GDAL) and
>>>>>>>>
>>>>>>>>https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vect
>>>>>>>>or
>>>>>>>>_tile.proto
>>>>>>>> + the code for converting pixel coordinates in features to
>>>>>>>>relevant
>>>>>>>>geo
>>>>>>>> coordinates and GEOS-based merging of features.
>>>>>>>>
>>>>>>>>> We plan to write a generalized encoder and decoder eventually so
>>>>>>>>>that any
>>>>>>>>> other library can plugin with out the mapnik requirement.
>>>>>>>>
>>>>>>>>
>>>>>>>> This would be amazing.
>>>>>>>>
>>>>>>>>> If you have any questions specifically about Mapnik Vector Tile
>>>>>>>>>or
>>>>>>>>>the
>>>>>>>>> Mapbox Vector Tile Specification feel free to ask me. I would
>>>>>>>>>note
>>>>>>>>>that you
>>>>>>>>> guys will want to update your vector tiles once the V2
>>>>>>>>>specification changes
>>>>>>>>> are into Mapnik-Vector-Tile.
>>>>>>>>
>>>>>>>>
>>>>>>>> Cool! Thanks. Our world tiles are generated via Docker containers
>>>>>>>>on a
>>>>>>>> cluster of computers - using Mapnik via Tilelive directly - so
>>>>>>>>upgrades on
>>>>>>>> Mapnik-Vector-Tile will propagate to the open-source rendering
>>>>>>>>software.
>>>>>>>> I expect V2 spec will mean also new MapBox Streets V7, right?
>>>>>>>>
>>>>>>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I
>>>>>>>>>don't
>>>>>>>>> see
>>>>>>>>> mention of that neither in the MVT spec (
>>>>>>>>> https://github.com/mapbox/vector-
>>>>>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>>>>>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ),
>>>>>>>>>although
>>>>>>>>> I
>>>>>>>>> guess it is just a matter of storing a MVT blob in the tile_data
>>>>>>>>>column.
>>>>>>>>
>>>>>>>>
>>>>>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector
>>>>>>>>Tiles
>>>>>>>>inside -
>>>>>>>> and Mapnik reads them directly when rasterising. Storing the
>>>>>>>>vector
>>>>>>>>tiles in
>>>>>>>> SQLite make sense - and it simplifies deployment, and you can then
>>>>>>>>easily
>>>>>>>> have complete world on each node of a cluster of tileservers. If
>>>>>>>>you don't
>>>>>>>> want to upgrade your base map too often, this approach is pretty
>>>>>>>>efficient
>>>>>>>> and removes the centralised database as typical bottleneck.
>>>>>>>>
>>>>>>>> I did some tests on vector tiles vs raster in MBTiles and
>>>>>>>>documented these
>>>>>>>> in README.md at and
>>>>>>>>https://github.com/klokantech/vector-tiles-sample before
>>>>>>>> we started to work on OSM2VectorTiles project.
>>>>>>>> There are also sample data and offline MapBox GL JS viewer in that
>>>>>>>>repo.
>>>>>>>> Vector tiles can be hosted also "unpacked", just as files in
>>>>>>>>folders on a
>>>>>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does
>>>>>>>>by
>>>>>>>> default for raster tiles.
>>>>>>>>
>>>>>>>> If OGR driver is implemented, the primary source should be
>>>>>>>>probably
>>>>>>>>reading
>>>>>>>> from an URL via curl.
>>>>>>>>
>>>>>>>> Reading PBFs from an MBTiles (or another SQLite) is just the most
>>>>>>>>practical
>>>>>>>> portable alternative to it.
>>>>>>>>
>>>>>>>> Technically people may store the tiles in different ways - for
>>>>>>>>example
>>>>>>>> Wikimedia guys (http://maps.wikimedia.org/ and
>>>>>>>> https://github.com/kartotherian/) use Cassandra for tile storage,
>>>>>>>>I
>>>>>>>>guess
>>>>>>>> MapBox internally is also on a different storage for production
>>>>>>>>servers ...
>>>>>>>> for transfers and streaming tilesets there is the  - but tiles are
>>>>>>>>ALWAYS
>>>>>>>> exposed via HTTP.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Petr
>>>>>>>> --
>>>>>>>> Petr Pridal, Ph.D.
>>>>>>>> CEO
>>>>>>>>
>>>>>>>> Klokan Technologies GmbH
>>>>>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland
>>>>>>>> Tel: +41 (0)41 511 26 12
>>>>>>>> Email: info at klokantech.com
>>>>>>>> Web: http://www.klokantech.com/
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> gdal-dev mailing list
>>>>>>>> gdal-dev at lists.osgeo.org
>>>>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>_______________________________________________
>>>gdal-dev mailing list
>>>gdal-dev at lists.osgeo.org
>>>http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>