[Tilecache] Patch for WMTS in TileCache

[Christopher Schmidt]

On Fri, Sep 11, 2009 at 04:30:12PM +0930, Jim Groffen wrote:
> Hello Chris,
> 
> I apologize for the size of this email, but I'm trying to be complete. I've responded to your questions below and further on talk about an idea for incorporating your feedback.
> 
> 1 Currently, TileCache is designed to support accessing any tile set
>   with any cache backend. Since all tiling schemes essentially work the
>   same -- specifically, they all work on a fixed grid, with a set of
>   scales and reslutions that allows a multilevel fixed grid, typically
>   with the same corners at all levels of the grid -- it is easiest to
>   transform this grid into a fixed set.
> 
>   It seems like the WMTS layer does not do this, and invents its own
>   cache instead.
> 
>   Is the cache set up by WMTS different enough that this is *really*
>   neccesary?
> 
> TileCache does work with WMTS layers using disk cache and should work with any other cache too (I didn't test all of them). It does this by turning a WMTS layer definition into many TileCache compatible layers. Each TileCache layer is a unique combination of the style, dimensions and tile matrix sets for a layer. Before WMTS came along if you wanted the same layer with two different styles in TileCache you would make two different layers. That's how the WMTS implementation deals with it. Around line 118 of Service.py is where, during configuration file reading, a WMTS layer is found and a call to TileCache.Layers.WMTS.load_wmts_layer is made.
> 
> Given the following WMTS layer in the config file:
> 
> [basic]
> type=WMTS
> ...
> ...
> title=Basic
> tile_matrix_sets=WholeWorld_CRS_84,World84-90_CRS_84
> 
> If was ask for a TMS capabilities document we see two layers, one for each
> tile matrix set. The name for each layer is a string join("_") of layer name,
> style if present, dimensions if present and tile matrix set. Every possible
> combination is treated as its own layer because they will be cached
> separately on disk. An example TMS capabilities file is attached.

Okay, this is a valuable piece of information. This might lead to a
better implementation suggestion; thinking through it.

> It is true that when using a cache of type WMTS, unless passed a WMTSTile
> instance it will raise a TileCacheException. WMTSTile instances are at the
> moment only coming from a WMTS layer, because they need to have a tile matrix
> set. If the layer associated to the tile has style or dimension information
> then that gets used too.

I see (upon reading more carefully) that the WMTS disk cache is
apparently designed to support some sort of WMTS behavior where files
are read directly from disk, and that's why it exists. I was concerned
on reading it, but am no longer; objections to this are withdrawn. 

> 2 It appears that WMTS has been developed as a 'Layer'. Layers are ways
>   of accessing backend services -- so the MapServer layer accesses a
>   MapServer service, the WMS layer accesses a WMS service, etc. WMTS
>   does not appear to be this (since you're pointing at my WMS for your
>   demo config). Instead, it appears that WMTS should be a *service*
>   (TileCache/Services/ directory); the configuration parameters that
>   you are using to define tilesets should potentially be added to *any*
>   layer, and these configuration parameters are used to return metadata
>   and tile requests whenever a request comes in in the WMTS service
>   format.
> 
>   As it is now, presumably WMTS doesn't work with Mapnik, MapServer,
>   GDAL, etc. backends. This ireally counter to the design of TileCache,
>   and I'm sorry that was unclear.
> 
>   Note that a WMTS layer -- that is, a layer class which requested
>   tiles from a WMTS service -- is not out of the question, and could be
>   implemented -- as could a TMS layer, for example, but I don't think
>   that either of those is what has been built here.
> 
> In my current implementation when a WMTS layer is seen in the config,
> TileCache.Layers.WMTS interprets it into many WMS layers. In other
> words TileCache.Layers.WMTS provides support for WMTS layer
> definitions in the config. This is as you say supporting WMTS for only
> one backend service and not all backend services.

So, what The WMTS layer object actually does, as I understand it, is
create a *set* of TileCache layers; each of those TileCache layers is
(currently) a WMS layer. Is that correct? This makes me wonder if 
something that could (Thought not neccesarily 'should') be done is to
change the code to instead be a utility which takes a set of WMTS
configuration and turns it into a TileCache config.

My expectations for WMTS was that the only thing that would be needed
would be a 'Service' class. This Service class would take in requests,
and return the correct tile. That class appears to exist; thank you for
writing it. It returns the MEtaData, maps tile requests to internal
tiles, etc. (Somehow, in my first readthrough, I totally missed this.) 

It appears there is additional code in the form of the Tile and Layer,
as well as the TileMatrix Layer object. I am less certain how these are
meant to work, and concerned that people might not want to use them the
way that they are currently designed. (For example, wanting to use WMTS
against a Mapnik backend.) If, instead of a class which 'magically'
creates multiple layers, there was a pre-processing step which generated
the TileCache configs, it seems like it's possible that if someone
wanted to (as I suggest) use Mapnik instead of WMS, they could tweak the
output config.

In order to accomodate for the WMTS specific layer properties, the base
layer could be extended; for example, you could add a "wmts_layerset"
key/value param. Then all layers which had the same wmts_layerset would
get put together in the output capabilities. 

> Also: when a WMTS request comes in it goes to the WMTS Service, which
> in turn will hand it on to the WMTS Layer if it needs to get the tile
> from the backend. It would be possible to have the WMTS Service go to
> any layer to handle the backend as other Service implementations do,
> but it would mean pushing WMTS specific implementation bits down to
> the backends. The reason I didn't do this is because right now
> "Layers.WMS" knows how to handle tiles the TileCache way where
> Layers.WMTS knows how to handle tiles the WMTS way. They are different
> and I can't consolidate them easily. I'll get into that after I've
> explained TileMatrixSets

Layers.WMS takes a tile (layername, x, y, z) and fetches it from a
server. This is the part of the Layers.WMTS in 'renderTile'. This method
looks pretty much the same as the WMS layer to me. The additional
configuration properties on the layer look like something that could
potentially be pushed as options into the base Layer. 

This leaves a couple things in the Layer.WMTS:
 * GetFeatureInfo. In my opinion, this should be removed. TileCache
   is really not designed to be fetching and caching feature data,
   in my opinion; or if it is, then it should be something we work on
   more thoroughly than just pushing it into this one layer type.
 * Dimensions? Not entirely sure about this, but they just look like
   extra KV Pairs, right?
 * The "WMTSTile" stuff, which could be a Layer-level option -- "When 
   creating Tiles, use this subclass." 

> 3 I really think that -- althogh your effort was admirable -- I feel like
>   a lot of this work was simply done in the wrong place/way.
> 
> Having as little impact on the existing TileCache implementation as
> possible and isolating the WMTS code has lead me to not implement WMTS
> the way it should have been?

While that's certainly a possibility, I don't think that's actually the
case; but even if it was, I think we can both admit that 'changing as
little code as possible when adding a major feature' is likely to
hamstring any development through no fault of your own (and only the
fault of a mostly absent maintainer, imho).

> When put that way it does seem like I put that goal over a correct
> implementation but that's not the case.
> I'll be honest - I didn't know TileCache all that well when I started
> but I know it much better now and these emails are improving my
> understanding even more. This is why the implementation is not exactly
> how you would like it to be. I'm aware that most users and
> contributors to TileCache won't be interested in WMTS and didn't want
> to be the cause of added complexity for them but I wouldn't sacrifice
> the quality of the implementation for it. Looking back, I should have
> badgered you earlier and more often than I did to get your feedback on
> my implementation plans.

That's okay. I should have badgered you more on what you were working
on; working in a vaccum was definitely not the result of your lack of
trying to get me interested. (My fundamental dislike of the spec,
especially as it seems to have gotten so much more ridiculous in more
recent revisions, plays a significant role, obviously.)

> 4 Describe tile_matrix more, and I can probably comment on it. What is a
>   tile_matrix? What does it map to most closely in existing code?
> 
> A tile matrix set controls what tiles are available for a layer. In
> TileCache terms a Tile Matrix Set replaces the bounding box,
> resolutions list, image size and CRS. A tile matrix set is made up of
> multiple "TileMatrix", one for each zoom level. This means a Tile
> Matrix maps to one element of the resolution array, but because WMTS
> supports different bounds and image sizes per zoom level, it also maps
> to bounding box and image size. In TileCache configuration I've made
> it so that you can set things like top left point and image width /
> height at the Tile Matrix Set level and all tile matrices for that
> level will inherit the same settings. The WMTSGuide-excerpt.txt
> attachment has the documentation I wrote on WMTS configuration of Tile
> Matrix Sets in it. It's from docs/WMTSGuide.txt This is why
> Layers.WMTS handles asking for tiles from the backend - why WMTS is
> its own backend.

> 
> 
> Based on your feedback, let's think of an alternative implementation:
> 
> First let's get rid of Layers.WMTS and move the config handling part
> somewhere else (say the Service class, that's where other layers are
> created from config). Now when config sees a layer of any type with
> WMTS properties (styles, dimensions, tilematrixsets) it will instead
> create many layer instances to handle each
> style/dimension/tilematrixset combination. When a WMTS request comes
> in, the WMTS service will find and use the right layer for the unique
> style/dimension/tilematrixset combination of that request.

Yes, I think this is how I would have it done. The only question in my
mind, given that, is where the first part lives. My suggestion earlier
of a config file 'preprocessor' which takes these config options and
writes out a 'regular' tilecache config was designed to address this.

> Now we are left with the problem that the way TileCache addresses
> tiles is different to how WMTS does it. I'd say they are different,
> but not incompatible. The way TileCache does it with resolutions,
> bbox, image size and crs can be represented in a TileMatrixSet, we
> could support the existing config files and generate a TileMatrixSet
> instead. Then each backend would use the tile matrix set instead of
> resolution, image width / height, bbox, crs. This would work without
> the need for code duplication but it would impact every layer and
> service. Also, not all of the capabilities of WMTS are supported by
> every layer (such as dimensions and multiple styles) - Of course they
> don't have to be supported right away or even at all in some cases.

It seems to me that 'dimensions' and 'multiple styles' are likely to be
a WMS-only option. If we went with the 'preprocessing' method I
suggested above, and someone said "Okay, here is my WMTS config, and I
want you to write out a bunch of Mapnik layers, with these
'dimensions'", the script might throw an error saying "only WMS supports
dimensions."

> Is this headed in the right direction? Is it better than what we have
> at the moment where WMTS is only supported for the WMS backend? While
> much of the code needed (at least for WMS service and back end, and
> the WMTS service) is already done I am daunted by the amount of work
> involved, any suggestions?

So, first of all, let's put the work you have in a branch, and get you
commit access to TileCache SVN. http://svn.tilecache.org/branches/wmts/
, and I'll email you a password.

Next, let's think about what the right implementation is. *My* desire is
to continue to use bbox/resolution/etc. for all layers, and if we need
to give the set of parameters that is a name, we can add additional data
to do that. So for example, we might extend the TileCache config to
support any sections which start with [tilematrix_blah] to describe a
tilematrix; then, you have two options when setting up a layer:

bbox=
resolution=
etc.=

*or*

tilematrix=blah

The Service init can then take those and 'look up' the tile matrix and
match it to the bbox/resolution.

The difficulties I see with this are that I'm not sure that a layer in
TileCache actually has all the config info you need: you mentioned that
each 'zoom level' can basically use a different tileset, and TileCache
doesn't really have a good way of doing that, and I"m not sure that I
want to extend it to do so. So I guess then the question is: "how
important is that?" Is it something that the spec absolutely requires?
is it something that we can avoid? (And who the hell is ever going to
use it?) Honestly, when there's a 'feature' in a spec that even ESRI
hasn't seen fit to implement, I Get a little bit concerned at the idea that
it is the Most Important Feature Ever; it seems an unlikely-to-be-used
feature to me.

If it really is important, then I guess we'll have to think about the
right way to do it.

So, here's what I think I would like, and you can tell me if this is
going to be useful:

 1. We keep the WMTS service. It looks good.
 2. We move the properties we need for WMTS "layers" into the base
    layer. 
 3. We figure out whether we should be: 
    * Creating Layers on the fly when the config is loaded
  or
    * Creating layers via a seperate tool that uses a WMTS-like config
      using the library functions you've already built. (The latter
      allows users to tweak the resulting configs more easily; the
      former allows the 'write and forget', but at the cost of the
      config now being 'magical' in that it creates additional layers.)

 4. We extend TileCache to support named descriptions of layer extents
    and resolutions (and whatever else). (These are 'tilematrixes', I believe.) 

 5. For now, we punt on the "WMTSTile" and "WMTSCache" until we get the
     basics working. At that point, we can establish how to get the
     tiles written on disk in a wya that you would access them through
     the WMTS 'REST' service.

Does that sound reasonable?      


Regards,
-- 
Christopher Schmidt
MetaCarta