[gdal-dev] Vector Tiles in OGR

[Stefan Keller]

Hi Even,

We - Martin Boos (new in Cc) and I - are definitely thinking about
implementing an option for spatial bounds.
Youmay know e.g. this WFS option of QGIS?
 „[_] Only request features overlapping the view extent“

What would you prefer: The same behaviour as for the WFS provider, or
taking a fixed bounds which aren't following users pan/zoom actions
(unless the user explicitly resets the bounds)?

:Stefan

@Blake: We are very interested in you opinion regarding handling of
keys "crs" and "bounds" now as well as part of the future TileJSON
spec.


---------- Forwarded message ----------
From: Even Rouault <even.rouault at spatialys.com>
Date: 2017-04-17 11:44 GMT+02:00
Subject: Re: [gdal-dev] Vector Tiles in OGR
To: Stefan Keller <sfkeller at gmail.com>
Cc: Gdal-Dev <gdal-dev at lists.osgeo.org>, Flippmoke
<flippmoke at gmail.com>, Even Rouault <even.rouault at mines-paris.org>,
petr.pridal at klokantech.com


Hi Stefan,


> P.S. @Even: Any news on a GDAL/OGR MVT reader?

No activity on this front as far as I know. There was a possible
funding opportunity recently for a project that would have involved
integating such a driver with QGIS, but the odds for it to come true
seems to be very low according to the latest updates I got.

And your plugin might actually solve the needs of this project !

Quick question: from my test of the plugin, I see it converts the
whole mbtiles and optionally merges geometry chunks of the same
feature spread over several tiles into a single geometry, right ? In
the preliminary discussions for the project I mentionned, it was not
clear if there was a need for doing just a decoding & rendering of
tiles that intersect the window of interest (could make sense if using
very large mbtiles or using remote tiles), instead of a whole mbtiles
conversion. Partial decoding, with merging of the tiles intersecting
the window of interest, seems to be a bit at odds with the (implicit
?) assumptions of the QGIS vector layers that assume that the geometry
of a feature will always be the same when it is returned, and not
clamped to the window of interest.

Even



>

> 2016-01-31 23:22 GMT+01:00 Flippmoke <flippmoke at gmail.com>:

> > All,

> >

> > The first thing that is going to be required likely is a good generic c++

> > library for encoding and decoding vector tiles IMO.

> >

> > I have spent a lot of time working on developing the mapnik vector tile

> > library to be a solid implementation, especially around all the work done

> > for 2.0 of Mapbox vector tile specification. I would love to eventually

> > help make this a generalized implementation so that it is portable to a

> > library like GDAL.

> >

> > We have talked about doing something like this at Mapbox but will require

> > a good deal of time, so it hasn't been pushed forward yet. The other

> > issues that come to mind to me is that some of the libraries that mapnik

> > vector tile depends upon most likely need good custom implementations if

> > we wanted to make it more generic.

> >

> > The reason for this is that I would prefer for it to be a header only

> > library with no dependencies. Adding more dependencies for GDAL seems

> > like it could be a mess.

> >

> > Blake Thompson

> >

> >> On Jan 31, 2016, at 10:10 AM, Stefan Keller <sfkeller at gmail.com> wrote:

> >>

> >> Hi Petr, Blake, Even and everybody

> >>

> >> I'm (besides Petr) the other maintainer of OSM2VectorTiles.

> >> I'd already asked Even a similar question.

> >> I'm advising now an intern to implement such a MVT Vector Tiles reader

> >> client in Python if possible as part of a QGIS plugin.

> >> And I'm interested not to do redundant implementations.

> >> Are there any news on this matter?

> >>

> >> :Stefan

> >>

> >> Prof. at HSR and leader of Geometa Lab

> >>

> >> @Even: Regarding OGR MVT development can you send to me (directly) a

> >> contact name of the group which won the bidding?

> >>

> >> 2016-01-06 15:24 GMT+01:00 Petr Pridal <petr.pridal at klokantech.com>:

> >>> Hi everybody,

> >>>

> >>> thanks for the comments. Regarding the technical details:

> >>>> Each Tile is its own discrete dataset therefore the hardest part would

> >>>> be

> >>>> merging of features into the original features across tiles. You could

> >>>> use

> >>>> some complicated merging technique, but that could be very expensive.

> >>>

> >>> The driver could require "id" parameter, which specifies name of an

> >>> attribute with unique identifier on the features shared across the

> >>> tiles.

> >>> Both MapBox vector tiles and OSM2VectorTiles tiles have an unique

> >>> non-zero

> >>> "osm_id" in place. The first version of driver could read only features

> >>> with such ID. The identifier does not have to be always filled, but

> >>> often it is, especially in deeper zoom levels derived purely from

> >>> OpenStreetMap - but yeh, a completely general stitching would be much

> >>> harder.

> >>>

> >>> The clipping + merging of shapes itself can be implemented with

> >>> underlaying

> >>> GEOS library functions.

> >>>

> >>> BTW It is possible to examine vector tiles data in pure JavaScript X-Ray

> >>> viewer (no WebGL support on browser required) made in OpenLayers3:

> >>> http://klokantech.github.io/ol3-sandbox/vector/xray.html

> >>>

> >>> or with WebGL-enabled webbrowser with higher performance MapBox GL JS

> >>> powered X-Ray:

> >>> http://klokantech.github.io/mapbox-gl-js-offline-example/xray.html

> >>>

> >>> A debug viewer for each vector tile (made in OL3) decoded in pure

> >>> JavaScript:

> >>> http://klokantech.github.io/ol3-sandbox/vector/tile-inspector.html

> >>>

> >>> The OGR driver could be also practical for the unsimplified /

> >>> ungeneralised

> >>> MBTiles with OSM data available at:

> >>> http://osmlab.github.io/osm-qa-tiles/

> >>>

> >>>> If you wanted to simply view vector tile data it could get confusing as

> >>>> well since most vector tiles need a buffer when they are used for

> >>>> visualization programs. This is important for things like labeling on

> >>>> maps

> >>>> etc. Therefore, you could get a large amount of overlapping data on

> >>>> tiles if you loaded them all in at once.

> >>>

> >>> Clipping on border of each tile must be applied before stitching the

> >>> vector

> >>> features together - so the buffer is removed. Buffer is there only for

> >>> visualisation and labeling.

> >>>

> >>>> As far as writing a simple decoder -- the mapnik-vector-tile decoder

> >>>> isn't

> >>>> currently able to decode vector tiles into a form not supported by

> >>>> mapnik.

> >>>> Therefore, OGR would not be able to easily use this.

> >>>

> >>> Correct. It can't be used directly (GDAL can't link against), but the

> >>> relevant code can be extracted and reused directly. For basic decoding

> >>> critical is the Protobuf library (which is already in GDAL) and

> >>> https://github.com/mapbox/mapnik-vector-tile/blob/master/proto/vector_ti

> >>> le.proto + the code for converting pixel coordinates in features to

> >>> relevant geo coordinates and GEOS-based merging of features.

> >>>

> >>>> We plan to write a generalized encoder and decoder eventually so that

> >>>> any

> >>>> other library can plugin with out the mapnik requirement.

> >>>

> >>> This would be amazing.

> >>>

> >>>> If you have any questions specifically about Mapnik Vector Tile or the

> >>>> Mapbox Vector Tile Specification feel free to ask me. I would note that

> >>>> you

> >>>> guys will want to update your vector tiles once the V2 specification

> >>>> changes are into Mapnik-Vector-Tile.

> >>>

> >>> Cool! Thanks. Our world tiles are generated via Docker containers on a

> >>> cluster of computers - using Mapnik via Tilelive directly - so upgrades

> >>> on

> >>> Mapnik-Vector-Tile will propagate to the open-source rendering software.

> >>> I expect V2 spec will mean also new MapBox Streets V7, right?

> >>>

> >>>> I see you mention MVT in MBTiles. Is it a recognized practice ? I don't

> >>>> see

> >>>> mention of that neither in the MVT spec (

> >>>> https://github.com/mapbox/vector-

> >>>> tile-spec/tree/master/2.0 ) nor the MBTiles one (

> >>>> https://github.com/mapbox/mbtiles-spec/blob/master/1.2/spec.md ),

> >>>> although

> >>>> I

> >>>> guess it is just a matter of storing a MVT blob in the tile_data

> >>>> column.

> >>>

> >>> Yes. MapBox Studio Classic generates such MBTiles with Vector Tiles

> >>> inside - and Mapnik reads them directly when rasterising. Storing the

> >>> vector tiles in SQLite make sense - and it simplifies deployment, and

> >>> you can then easily have complete world on each node of a cluster of

> >>> tileservers. If you don't want to upgrade your base map too often, this

> >>> approach is pretty efficient and removes the centralised database as

> >>> typical bottleneck.

> >>>

> >>> I did some tests on vector tiles vs raster in MBTiles and documented

> >>> these

> >>> in README.md at and https://github.com/klokantech/vector-tiles-sample

> >>> before we started to work on OSM2VectorTiles project.

> >>> There are also sample data and offline MapBox GL JS viewer in that repo.

> >>> Vector tiles can be hosted also "unpacked", just as files in folders on

> >>> a

> >>> standard web server - exactly like GDAL2Tiles or MapTiler.com does by

> >>> default for raster tiles.

> >>>

> >>> If OGR driver is implemented, the primary source should be probably

> >>> reading

> >>> from an URL via curl.

> >>>

> >>> Reading PBFs from an MBTiles (or another SQLite) is just the most

> >>> practical

> >>> portable alternative to it.

> >>>

> >>> Technically people may store the tiles in different ways - for example

> >>> Wikimedia guys (http://maps.wikimedia.org/ and

> >>> https://github.com/kartotherian/) use Cassandra for tile storage, I

> >>> guess

> >>> MapBox internally is also on a different storage for production servers

> >>> ...

> >>> for transfers and streaming tilesets there is the - but tiles are

> >>> ALWAYS

> >>> exposed via HTTP.

> >>>

> >>> Regards,

> >>>

> >>> Petr

> >>> --

> >>> Petr Pridal, Ph.D.

> >>> CEO

> >>>

> >>> Klokan Technologies GmbH

> >>> Hofnerstrasse 98, 6314 Unterageri, Switzerland

> >>> Tel: +41 (0)41 511 26 12

> >>> Email: info at klokantech.com

> >>> Web: http://www.klokantech.com/

> >>>

> >>> _______________________________________________

> >>> gdal-dev mailing list

> >>> gdal-dev at lists.osgeo.org

> >>> http://lists.osgeo.org/mailman/listinfo/gdal-dev





--

Spatialys - Geospatial professional services

http://www.spatialys.com