[gdal-dev] Vector Tiles in OGR

[Stefan Keller]

Hi Petr, Blake, Even and everybody

I'm (besides Petr) the other maintainer of OSM2VectorTiles.
I'd already asked Even a similar question.
I'm advising now an intern to implement such a MVT Vector Tiles reader
client in Python if possible as part of a QGIS plugin.
And I'm interested not to do redundant implementations.
Are there any news on this matter?

:Stefan
Prof. at HSR and leader of Geometa Lab

@Even: Regarding OGR MVT development can you send to me (directly) a
contact name of the group which won the bidding?


2016-01-06 15:24 GMT+01:00 Petr Pridal <petr.pridal at klokantech.com>:
> Hi everybody,
>
> thanks for the comments. Regarding the technical details:
>
>> Each Tile is its own discrete dataset therefore the hardest part would be
>> merging of features into the original features across tiles. You could use
>> some complicated merging technique, but that could be very expensive.
>
>
> The driver could require "id" parameter, which specifies name of an
> attribute with unique identifier on the features shared across the tiles.
> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique non-zero
> "osm_id" in place. The first version of driver could read only features with
> such ID. The identifier does not have to be always filled, but often it is,
> especially in deeper zoom levels derived purely from OpenStreetMap - but
> yeh, a completely general stitching would be much harder.
>
> The clipping + merging of shapes itself can be implemented with underlaying
> GEOS library functions.
>
> BTW It is possible to examine vector tiles data in pure JavaScript X-Ray
> viewer (no WebGL support on browser required) made in OpenLayers3:
> http://klokantech.github.io/ol3-sandbox/vector/xray.html
>
> or with WebGL-enabled webbrowser with higher performance MapBox GL JS
> powered X-Ray:
> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html
>
> A debug viewer for each vector tile (made in OL3) decoded in pure
> JavaScript:
> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html
>
> The OGR driver could be also practical for the unsimplified / ungeneralised
> MBTiles with OSM data available at: http://osmlab.github.io/osm-qa-tiles/
>
>>
>> If you wanted to simply view vector tile data it could get confusing as
>> well since most vector tiles need a buffer when they are used for
>> visualization programs. This is important for things like labeling on maps
>> etc. Therefore, you could get a large amount of overlapping data on tiles if
>> you loaded them all in at once.
>
>
> Clipping on border of each tile must be applied before stitching the vector
> features together  - so the buffer is removed. Buffer is there only for
> visualisation and labeling.
>
>>
>> As far as writing a simple decoder -- the mapnik-vector-tile decoder isn't
>> currently able to decode vector tiles into a form not supported by mapnik.
>> Therefore, OGR would not be able to easily use this.
>
>
> Correct. It can't be used directly (GDAL can't link against), but the
> relevant code can be extracted and reused directly. For basic decoding
> critical is the Protobuf library (which is already in GDAL) and
> https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector_tile.proto
> + the code for converting pixel coordinates in features to relevant geo
> coordinates and GEOS-based merging of features.
>
>> We plan to write a generalized encoder and decoder eventually so that any
>> other library can plugin with out the mapnik requirement.
>
>
> This would be amazing.
>
>> If you have any questions specifically about Mapnik Vector Tile or the
>> Mapbox Vector Tile Specification feel free to ask me. I would note that you
>> guys will want to update your vector tiles once the V2 specification changes
>> are into Mapnik-Vector-Tile.
>
>
> Cool! Thanks. Our world tiles are generated via Docker containers on a
> cluster of computers - using Mapnik via Tilelive directly - so upgrades on
> Mapnik-Vector-Tile will propagate to the open-source rendering software.
> I expect V2 spec will mean also new MapBox Streets V7, right?
>
>> I see you mention MVT in MBTiles. Is it a recognized practice ? I don't
>> see
>> mention of that neither in the MVT spec (
>> https://github.com/mapbox/vector-
>> tile-spec/tree/master/2.0 ) nor the MBTiles one (
>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ), although
>> I
>> guess it is just a matter of storing a MVT blob in the tile_data column.
>
>
> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles inside -
> and Mapnik reads them directly when rasterising. Storing the vector tiles in
> SQLite make sense - and it simplifies deployment, and you can then easily
> have complete world on each node of a cluster of tileservers. If you don't
> want to upgrade your base map too often, this approach is pretty efficient
> and removes the centralised database as typical bottleneck.
>
> I did some tests on vector tiles vs raster in MBTiles and documented these
> in README.md at and https://github.com/klokantech/vector-tiles-sample before
> we started to work on OSM2VectorTiles project.
> There are also sample data and offline MapBox GL JS viewer in that repo.
> Vector tiles can be hosted also "unpacked", just as files in folders on a
> standard web server - exactly like GDAL2Tiles or MapTiler.com does by
> default for raster tiles.
>
> If OGR driver is implemented, the primary source should be probably reading
> from an URL via curl.
>
> Reading PBFs from an MBTiles (or another SQLite) is just the most practical
> portable alternative to it.
>
> Technically people may store the tiles in different ways - for example
> Wikimedia guys (http://maps.wikimedia.org/ and
> https://github.com/kartotherian/) use Cassandra for tile storage, I guess
> MapBox internally is also on a different storage for production servers ...
> for transfers and streaming tilesets there is the  - but tiles are ALWAYS
> exposed via HTTP.
>
> Regards,
>
> Petr
> --
> Petr Pridal, Ph.D.
> CEO
>
> Klokan Technologies GmbH
> Hofnerstrasse 98, 6314 Unterageri, Switzerland
> Tel: +41 (0)41 511 26 12
> Email: info at klokantech.com
> Web: http://www.klokantech.com/
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev