[gdal-dev] Vector Tiles in OGR

Mon Feb 1 11:25:58 PST 2016

Hi Blake,

There is TileMaker [1]. It implements a MVT writer for v1 only as it
seems, but looks lightweight enough as a good base for a portable library.

Ben

[1] 
https://github.com/systemed/tilemaker/blob/master/src/write_geometry.cpp

Am 01.02.16, 18:27 schrieb "gdal-dev on behalf of Flippmoke" unter
<gdal-dev-bounces at lists.osgeo.org on behalf of flippmoke at gmail.com>:

>
>Stefan,
>
>I am not aware of any libraries besides mapnik vector tile that have been
>attempting to implement v2 specification. I think mapzen's implementation
>might be best? We have been wanting to add vector tiles to the python
>mapnik bindings for quite some time but we don't have the time to do it
>right now. 
>
>I will try to think this week about how we could make a generic C++
>library quickly for Mapbox vector tiles.
>
>Thanks,
>
>Blake Thompson
>
>> On Feb 1, 2016, at 1:06 AM, Stefan Keller <sfkeller at gmail.com> wrote:
>> 
>> Hi Blake
>> 
>> It's clear to me that a good generic c++ library for encoding and
>> decoding vector tiles is needed.
>> On the other hand, I'd like to experiment with vector tiles and
>> related QGIS parallel loading issues for which Python is more
>> convenient.
>> 
>> It seems that Mapzen's [1] Python implementation is further along.
>> Or do you know if Jesse (or somebody) is still working on Mapbox's
>>variant [2] ?
>> 
>> :Stefan
>> 
>> [1] https://github.com/mapzen/mapbox-vector-tile
>> [2] https://github.com/mapbox/vector-tile-py
>> 
>> 
>> 2016-01-31 23:22 GMT+01:00 Flippmoke <flippmoke at gmail.com>:
>>> All,
>>> 
>>> The first thing that is going to be required likely is a good generic
>>>c++ library for encoding and decoding vector tiles IMO.
>>> 
>>> I have spent a lot of time working on developing the mapnik vector
>>>tile library to be a solid implementation, especially around all the
>>>work done for 2.0 of Mapbox vector tile specification. I would love to
>>>eventually help make this a generalized implementation so that it is
>>>portable to a library like GDAL.
>>> 
>>> We have talked about doing something like this at Mapbox but will
>>>require a good deal of time, so it hasn't been pushed forward yet. The
>>>other issues that come to mind to me is that some of the libraries that
>>>mapnik vector tile depends upon most likely need good custom
>>>implementations if we wanted to make it more generic.
>>> 
>>> The reason for this is that I would prefer for it to be a header only
>>>library with no dependencies. Adding more dependencies for GDAL seems
>>>like it could be a mess.
>>> 
>>> Blake Thompson
>>> 
>>>> On Jan 31, 2016, at 10:10 AM, Stefan Keller <sfkeller at gmail.com>
>>>>wrote:
>>>> 
>>>> Hi Petr, Blake, Even and everybody
>>>> 
>>>> I'm (besides Petr) the other maintainer of OSM2VectorTiles.
>>>> I'd already asked Even a similar question.
>>>> I'm advising now an intern to implement such a MVT Vector Tiles reader
>>>> client in Python if possible as part of a QGIS plugin.
>>>> And I'm interested not to do redundant implementations.
>>>> Are there any news on this matter?
>>>> 
>>>> :Stefan
>>>> Prof. at HSR and leader of Geometa Lab
>>>> 
>>>> @Even: Regarding OGR MVT development can you send to me (directly) a
>>>> contact name of the group which won the bidding?
>>>> 
>>>> 
>>>> 2016-01-06 15:24 GMT+01:00 Petr Pridal <petr.pridal at klokantech.com>:
>>>>> Hi everybody,
>>>>> 
>>>>> thanks for the comments. Regarding the technical details:
>>>>> 
>>>>>> Each Tile is its own discrete dataset therefore the hardest part
>>>>>>would be
>>>>>> merging of features into the original features across tiles. You
>>>>>>could use
>>>>>> some complicated merging technique, but that could be very
>>>>>>expensive.
>>>>> 
>>>>> 
>>>>> The driver could require "id" parameter, which specifies name of an
>>>>> attribute with unique identifier on the features shared across the
>>>>>tiles.
>>>>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique
>>>>>non-zero
>>>>> "osm_id" in place. The first version of driver could read only
>>>>>features with
>>>>> such ID. The identifier does not have to be always filled, but often
>>>>>it is,
>>>>> especially in deeper zoom levels derived purely from OpenStreetMap -
>>>>>but
>>>>> yeh, a completely general stitching would be much harder.
>>>>> 
>>>>> The clipping + merging of shapes itself can be implemented with
>>>>>underlaying
>>>>> GEOS library functions.
>>>>> 
>>>>> BTW It is possible to examine vector tiles data in pure JavaScript
>>>>>X-Ray
>>>>> viewer (no WebGL support on browser required) made in OpenLayers3:
>>>>> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>>>>> 
>>>>> or with WebGL-enabled webbrowser with higher performance MapBox GL JS
>>>>> powered X-Ray:
>>>>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>>>>> 
>>>>> A debug viewer for each vector tile (made in OL3) decoded in pure
>>>>> JavaScript:
>>>>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>>>>> 
>>>>> The OGR driver could be also practical for the unsimplified /
>>>>>ungeneralised
>>>>> MBTiles with OSM data available at:
>>>>>http://osmlab.github.io/osm-qa-tiles/
>>>>> 
>>>>>> 
>>>>>> If you wanted to simply view vector tile data it could get
>>>>>>confusing as
>>>>>> well since most vector tiles need a buffer when they are used for
>>>>>> visualization programs. This is important for things like labeling
>>>>>>on maps
>>>>>> etc. Therefore, you could get a large amount of overlapping data on
>>>>>>tiles if
>>>>>> you loaded them all in at once.
>>>>> 
>>>>> 
>>>>> Clipping on border of each tile must be applied before stitching the
>>>>>vector
>>>>> features together  - so the buffer is removed. Buffer is there only
>>>>>for
>>>>> visualisation and labeling.
>>>>> 
>>>>>> 
>>>>>> As far as writing a simple decoder -- the mapnik-vector-tile
>>>>>>decoder isn't
>>>>>> currently able to decode vector tiles into a form not supported by
>>>>>>mapnik.
>>>>>> Therefore, OGR would not be able to easily use this.
>>>>> 
>>>>> 
>>>>> Correct. It can't be used directly (GDAL can't link against), but the
>>>>> relevant code can be extracted and reused directly. For basic
>>>>>decoding
>>>>> critical is the Protobuf library (which is already in GDAL) and
>>>>> 
>>>>>https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector_
>>>>>tile.proto
>>>>> + the code for converting pixel coordinates in features to relevant
>>>>>geo
>>>>> coordinates and GEOS-based merging of features.
>>>>> 
>>>>>> We plan to write a generalized encoder and decoder eventually so
>>>>>>that any
>>>>>> other library can plugin with out the mapnik requirement.
>>>>> 
>>>>> 
>>>>> This would be amazing.
>>>>> 
>>>>>> If you have any questions specifically about Mapnik Vector Tile or
>>>>>>the
>>>>>> Mapbox Vector Tile Specification feel free to ask me. I would note
>>>>>>that you
>>>>>> guys will want to update your vector tiles once the V2
>>>>>>specification changes
>>>>>> are into Mapnik-Vector-Tile.
>>>>> 
>>>>> 
>>>>> Cool! Thanks. Our world tiles are generated via Docker containers on
>>>>>a
>>>>> cluster of computers - using Mapnik via Tilelive directly - so
>>>>>upgrades on
>>>>> Mapnik-Vector-Tile will propagate to the open-source rendering
>>>>>software.
>>>>> I expect V2 spec will mean also new MapBox Streets V7, right?
>>>>> 
>>>>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I
>>>>>>don't
>>>>>> see
>>>>>> mention of that neither in the MVT spec (
>>>>>> https://github.com/mapbox/vector-
>>>>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>>>>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ),
>>>>>>although
>>>>>> I
>>>>>> guess it is just a matter of storing a MVT blob in the tile_data
>>>>>>column.
>>>>> 
>>>>> 
>>>>> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles
>>>>>inside -
>>>>> and Mapnik reads them directly when rasterising. Storing the vector
>>>>>tiles in
>>>>> SQLite make sense - and it simplifies deployment, and you can then
>>>>>easily
>>>>> have complete world on each node of a cluster of tileservers. If you
>>>>>don't
>>>>> want to upgrade your base map too often, this approach is pretty
>>>>>efficient
>>>>> and removes the centralised database as typical bottleneck.
>>>>> 
>>>>> I did some tests on vector tiles vs raster in MBTiles and documented
>>>>>these
>>>>> in README.md at and
>>>>>https://github.com/klokantech/vector-tiles-sample before
>>>>> we started to work on OSM2VectorTiles project.
>>>>> There are also sample data and offline MapBox GL JS viewer in that
>>>>>repo.
>>>>> Vector tiles can be hosted also "unpacked", just as files in folders
>>>>>on a
>>>>> standard web server - exactly like GDAL2Tiles or MapTiler.com does by
>>>>> default for raster tiles.
>>>>> 
>>>>> If OGR driver is implemented, the primary source should be probably
>>>>>reading
>>>>> from an URL via curl.
>>>>> 
>>>>> Reading PBFs from an MBTiles (or another SQLite) is just the most
>>>>>practical
>>>>> portable alternative to it.
>>>>> 
>>>>> Technically people may store the tiles in different ways - for
>>>>>example
>>>>> Wikimedia guys (http://maps.wikimedia.org/ and
>>>>> https://github.com/kartotherian/) use Cassandra for tile storage, I
>>>>>guess
>>>>> MapBox internally is also on a different storage for production
>>>>>servers ...
>>>>> for transfers and streaming tilesets there is the  - but tiles are
>>>>>ALWAYS
>>>>> exposed via HTTP.
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Petr
>>>>> --
>>>>> Petr Pridal, Ph.D.
>>>>> CEO
>>>>> 
>>>>> Klokan Technologies GmbH
>>>>> Hofnerstrasse 98, 6314 Unterageri, Switzerland
>>>>> Tel: +41 (0)41 511 26 12
>>>>> Email: info at klokantech.com
>>>>> Web: http://www.klokantech.com/
>>>>> 
>>>>> _______________________________________________
>>>>> gdal-dev mailing list
>>>>> gdal-dev at lists.osgeo.org
>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>_______________________________________________
>gdal-dev mailing list
>gdal-dev at lists.osgeo.org
>http://lists.osgeo.org/mailman/listinfo/gdal-dev