[mapguide-internals] RFC required for tile caching changes?
Paul Spencer
pspencer at dmsolutions.ca
Wed Jan 24 07:57:15 EST 2007
I support the modifications proposed without requiring an RFC. I do
agree with Jason about a larger RFC covering enhancements to the tile
cache.
Cheers
Paul
On 24-Jan-07, at 12:01 AM, Jason Birch wrote:
>
> Being able to specify tile size is very important for setting up
> the global profiles of TMS, but I would much rather see it
> implemented at the "TileGroup" (Andy's idea) level than at the
> server level. Same goes for the block size.
>
> I agree that TMS would be easy to implement with a script but was
> hoping for raw http sharing of the tile directory, with a custom
> handler for 404s that generates missing tiles (or sends a 404 if it
> really doesn't exist). This would give the benefits of the
> webserver without additional load on each request.
>
> At this point, I would support the alteration of the directory
> structure to fix a defect (as long as there are no changes to the
> API because of it), but the enhancements are running pretty close
> to the line. As well, there is a fair amount of interest in the
> caching mechanism and it would likely make sense to revisit it as
> part of a larger RFC to make sure that all of the desired
> modifications fit properly with our future direction.
>
> However I'm willing to go with the group on this if there's
> majority support for the small enhancements without an RFC.
>
> Jason
>
> ________________________________
>
> From: mapguide-internals-bounces at lists.osgeo.org on behalf of Chris
> Claydon
> Sent: Tue 2007-01-23 2:18 PM
> To: MapGuide Internals Mail List
> Subject: RE: [mapguide-internals] RFC required for tile caching
> changes?
>
>
>
> The impact on the AJAX Viewer complicates the issue of dynamic tile
> sizing, and Trevor will be addressing that issue in an RFC.
>
>
>
> I am going to implement the TileBlockSize config parameter (which
> has no impact on the viewer) and Traian's tile-organizing scheme.
>
>
>
> Chris.
>
>
>
> ________________________________
>
> From: mapguide-internals-bounces at lists.osgeo.org [mailto:mapguide-
> internals-bounces at lists.osgeo.org] On Behalf Of Walt Welton-Lair
> Sent: January 23, 2007 2:26 PM
> To: MapGuide Internals Mail List
> Subject: RE: [mapguide-internals] RFC required for tile caching
> changes?
>
>
>
> If you make the cached tile size a parameter then this will also
> require some AJAX Viewer work. IIRC the AJAX code has the current
> tile size hard-coded. If you make it a parameter then you'll need
> to update ajaxviewer.templ (I think) to let it be set dynamically.
>
>
>
> There may also be other things you need to fix in AJAX. For
> example, it currently requests tiles for rows / columns which are
> slightly outside your visible extents. I don't know offhand how
> many extra rows / columns it requests, but it seems this also
> depends on the tile size. Maybe you'd want to request enough rows/
> columns to give you an additional 25% screen coverage. So the
> number of extra rows/columns you'll need to request on each side is
> 0.25 * screen width in pixels / tile size. I don't know if AJAX
> viewer currently has such logic.
>
>
>
> Walt
>
> -----Original Message-----
> From: mapguide-internals-bounces at lists.osgeo.org on behalf of
> Chris Claydon
> Sent: Tue 1/23/2007 9:42 PM
> To: MapGuide Internals Mail List
> Cc:
> Subject: RE: [mapguide-internals] RFC required for tile caching
> changes?
>
> That should be relatively straightforward to achieve. I would add
> a comment into serverconfig.ini indicating that any modifications
> to these settings would:
>
>
>
> a) Only take effect after a server restart.
>
> b) Invalidate ALL existing cached tiles.
>
>
>
> Chris.
>
>
>
>
> ________________________________
>
>
> From: mapguide-internals-bounces at lists.osgeo.org [mailto:mapguide-
> internals-bounces at lists.osgeo.org] On Behalf Of Trevor Wekel
> Sent: January 23, 2007 1:34 PM
> To: MapGuide Internals Mail List
> Subject: RE: [mapguide-internals] RFC required for tile caching
> changes?
>
>
>
> Hi Chris,
>
>
>
> As an additional item, can we add serverconfig.ini parameters to
> control the following:
>
>
>
> - Size of the cache tile where the tiles assumed to be square
> "TileSize = 300" for 300x300 tiles
>
> - Tile block size, ie. the mod value in the discussion below
> "TileBlockSize = 30"
>
>
>
> The first item is good for tuning and is useful if you need to
> match MapGuide tiles against an externally tiled raster source.
> The second item could be used for tuning and eventually to support
> "block rendering" of tiles.
>
>
>
> Thanks,
>
> Trevor
>
>
>
>
> ________________________________
>
>
> From: mapguide-internals-bounces at lists.osgeo.org [mailto:mapguide-
> internals-bounces at lists.osgeo.org] On Behalf Of Chris Claydon
> Sent: Tuesday, January 23, 2007 1:26 PM
> To: MapGuide Internals Mail List
> Subject: RE: [mapguide-internals] RFC required for tile caching
> changes?
>
> We seem to have digressed a little from my original question! I
> have summarized the issues discussed in this thread below. I would
> like to implement the changes as proposed in item 2), and to do so
> without the need for an RFC. Please let me know if this is
> unacceptable for any reason.
>
>
>
> Thanks,
>
>
>
> Chris.
>
>
>
> 1) We cannot conform directly with the Tile Map Service
> Specification, because it does not currently support the concept of
> grouping sets of tiles into individual folders. However, it would
> be very simple to write a simple script or servlet to map from the
> specification's path format directly to the MapGuide format.
>
> 2) Traian suggested a slightly modified folder naming
> convention that provides an easier mapping to the original tile
> index. I like this approach and would be happy to implement it this
> way. His proposal is as follows:
>
>
>
> "Another suggestion I have -- name the folder part of the path (30
> * tile index div 30) and then the file part would be (tile index
> mod 30).
>
>
>
> So for tile 33,64 the folder would be:
>
>
>
> Base Layer Group/30/60/3_4.png
>
>
>
> And for tile -33,-64 the folder would be:
>
>
>
> Base Layer Group/-30/-60/-3_-4.png
>
>
>
> For tile 1,1, the folder would be:
>
>
>
> Base Layer Group/0/0/1_1.png
>
>
>
> For tile -1,-1 the folder would be:
>
>
>
> Base Layer Group/-0/-0/-1_-1.png"
>
>
>
> 3) Various issues were brought up regarding how tiles are
> generated, and how label placement can be optimized. These are
> beyond the scope of my proposal, and can be addressed separately
> without being affected by the changes I've proposed.
>
>
>
>
>
>
> ________________________________
>
>
> From: mapguide-internals-bounces at lists.osgeo.org [mailto:mapguide-
> internals-bounces at lists.osgeo.org] On Behalf Of Robert Bray
> Sent: January 17, 2007 9:08 AM
> To: MapGuide Internals Mail List
> Subject: Re: [mapguide-internals] RFC required for tile caching
> changes?
>
>
>
> Hey Paul, I think that is because you explicitly added him, for
> some weird reason this list is already set to reply to list. The
> change in question would not have made a difference in this case...
>
> Bob
>
> Paul Spencer wrote:
>
> Steve's response bounced from the list, but fortunately reply-to-
> all was still actually sending to everyone so I actually got the
> message ... hint, hint ...
>
> Paul
>
>
> On 16-Jan-07, at 11:18 PM, Steve Lime wrote:
>
> Paul: Er, I spent too much time at the Camp-to-Camp beer tap so my
> memory is a bit fuzzy. We talked about several things from pre-
> computing
> label positions to what you mention below. You are correct in how
> MapServer works. It doesn't have any special "tiled" mode and hence
> considers each tile just as any other map. In a single map context,
> computing label positions post-clip makes sense.
>
> Computing label positions pre-clip would result in a single,
> consistent
> label position for each feature. For points and polygons that's
> generally
> OK I believe. For linear features the results might not always be
> optimal
> since the clipping often provides a nice means of segmenting long
> features
> so that get labeled multiple times. One could work around that at the
> data level by segmenting long features ahead of time.
>
> I don't believe it would hard to hack MapServer drawing routine
> to do this and someone could test it out. Let me know if interested.
>
> Steve
>
> Paul Spencer <pspencer at dmsolutions.ca>
> <mailto:pspencer at dmsolutions.ca> 01/15/07 7:03 PM >>>
>
> Walt,
>
> I don't believe it got documented. I spoke with Steve Lime (cc'd)
> directly about this during OSGeo.
>
> At the time, I believe Steve suggested label placement would be done
> before features are clipped for rendering. I believe the way
> mapserver works, it grabs all features that intersect a given bbox
> then clips the features to the bbox then renders them. Labelling is
> currently done post-clip. If it were done pre-clip, then labels
> would be in a fixed position and could overlap tile boundaries
> without problems.
>
> I do believe that there were some potential problems with this
> approach, but I don't know enough of mapserver internals to remember
> what they were.
>
> Steve, perhaps you could jump in and fix my explanation? Also, do
> you think that label rendering changes would be a potential for
> version 5?
>
> Cheers
>
> Paul
>
> On 15-Jan-07, at 2:54 PM, Walt Welton-Lair wrote:
>
> "In lieu of fixing this in MapServer (there is actually a proposal
> to fix this)..."
>
> Paul - is there per chance a public link to this proposal? I'd be
> interested in looking at it.
>
> Thanks,
> Walt
>
> -----Original Message-----
> From: mapguide-internals-bounces at lists.osgeo.org on behalf of
>
> Paul
>
> Spencer (External)
> Sent: Mon 1/15/2007 7:40 PM
> To: MapGuide Internals Mail List
> Cc:
> Subject: Re: [mapguide-internals] RFC required for tile caching
>
>
>
> changes?
>
>
>
> Trevor,
>
> I don't think I got my idea across clearly :) Let me explain a
> little more clearly.
>
> In ka-Map, the size of a tile is configurable and by default is
> 200x200 px. There is a second concept that I've called a 'meta
> tile'. A meta tile is some number of tiles wide x high,
>
> typically 10
>
> x 10. When a request for a tile comes to the tile server, it
>
> checks
>
> the cache.
>
> If the tile exists, the tile (200x200px) is returned. This
>
> happens
>
> without invoking any map rendering ... the tile exists as an
>
> image on
>
> disk.
>
> If the tile does not exist, then a map render happens. A map
>
> the
>
> size of the meta tile is rendered and sliced into tile-sized
>
> chunks
>
> that are stored on disk in the cache directory structure. While
>
> the
>
> tiles are being created, a lock file is used to 'pause' requests
>
> for
>
> tiles in the same meta-tile directory so that they can all be
>
> served
>
> from the same map draw when it is finished.
>
> We chose this approach for several reasons. First, MapServer
>
> suffers
>
> from edge effects when rendering map images. This is normally
>
> not
>
> noticeable, but when you put several map draws together in a
>
> tiled
>
> client, it becomes horribly obvious. The solution was to render
>
> a
>
> slightly larger map image (about 10 pixels in fact) and clip out
>
> the
>
> center as the tile. Second, MapServer labelling only considers
>
> the
>
> current map draw. Depending on the size of the rendered label,
>
> it
>
> may not fit into a single tile. Depending on the size of a
>
> feature,
>
> the label may appear on several adjacent tiles. In lieu of
>
> fixing
>
> this in MapServer (there is actually a proposal to fix this),
> rendering a much larger map and slicing it into smaller tiles
> produces a reasonable effect. Finally, it is relatively
>
> expensive to
>
> parse the map file and find/read the data compared to rendering
>
> so
>
> rendering a single large image and slicing it is cheaper than
> rendering each smaller tile (with diminishing returns).
>
> I don't know how you actually render the tiles in MapGuide, but
>
> I
>
> don't think you suffer from the same edge and label issues.
>
> The ka-Map directory structure looks something like this:
>
>
> <map>/<layer-group>/<scale>/<row-of-metatile>/<col-of-
> metatile>/<row-
>
> of-tile><col-of-tile>.<ext>
>
> ... which is similar to Chris' suggestion :) I don't get the
> impression that you render the entire contents of the metatile
> directory in one go though. Because I do, I only have to check
>
> if
>
> the meta-tile directory exists to know if any tile in the
>
> directory
>
> exists, which is a small optimizatin.
>
> One difference between Chris' proposal and the way ka-Map works
>
> is
>
> that the scale and layer group are reversed. This makes it
> theoretically possible to share the <layer-group> cache
>
> directory
>
> between different maps simply by creating symlinks or by
>
> allowing a
>
> per-layer-group setting that indicates the root directory to
>
> keep the
>
> tile cache in.
>
> Cheers
>
> Paul
>
> On 15-Jan-07, at 11:49 AM, Trevor Wekel wrote:
>
> > Hi everyone,
> >
> > I did some analysis on file systems a little while ago in
>
> support
>
> > of the
> > directory structure change. Here are some approximate numbers
>
> for
>
> > directory entry and file system entries that I pulled from an
>
>
>
> internet
> > search:
> >
> > Linux ext3 - each file or directory is approx 8 bytes +
>
> length of
>
> > file
> > or directory name
> > Linux Reiserfs - each file or directory is 18 bytes + length
>
> of
>
> > file or
> > directory name
> >
> > If we assume each file name has a format of R99999_C99999.PNG
>
> then
>
> > each
> > file entry will require 26 bytes under ext3 and 34 bytes under
> > ReiserFS.
> >
> > To facilitate fast directory access, tiles which are close to
>
> each
>
> > other
> > should like in the same directory. This reduces the directory
>
>
>
> entry
> > reads for each tile request. If we also assume that the read
>
> block
>
> > size
> > for the file system is 32kbytes (32/64k is often used in RAID
>
>
>
> arrays)
> > then a directory with 900 entries (30x30 block) will get read
>
> in a
>
> > single IOP (900*34 = 30600). And since the tiles are blocked
> > together,
> > it is very likely that adjacent tiles will fall within the
>
> same
>
> > directory. Since the directory was just read, it will be
>
> cached
>
> and
> > there will be no disk access required.
> >
> > If we increase the size of the directories to more than 900
>
> entries
>
> > then
> > we will incur more than IOP to read the bottom-most directory
> > structure.
> >
> > I also think Traian's suggestion of Base Layer Group/
> 30/60/3_4.png is
> > reasonable. It is more readable than the original mod scheme.
> >
> >> From a rendering perspective, dealing with 2000x2000 pixel
>
> blocks
>
> >> may be
> > more efficient but cutting the tiles up from these larger
>
> images
>
> will
> > require some computational effort. Rendering the tiles in the
>
> same
>
> > size
> > as requested from the client reduces server load because the
>
> HTTP
>
> > request ends up being simply a file serving operation.
> >
> > With some of the latest optimizations I have been working on,
>
>
>
> MapGuide
> > tile serving speeds using "client-sized" tiles are quite
> respectable.
> > On a machine with a single 3GHz CPU, MapGuide can service more
>
> than
>
> > 100
> > tiles per second assuming the tiles are being served from
>
> memory,
>
> > ie. no
> > disk access. The additional computational effort required to
>
>
>
> generate
> > "client" tiles from larger pixel blocks will significantly
>
> impact
>
> > performance.
> >
> >
> > Thanks,
> > Trevor
> >
> > -----Original Message-----
> > From: mapguide-internals-bounces at lists.osgeo.org
> > [mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf
>
> Of
>
> Paul
> > Spencer (External)
> > Sent: Sunday, January 14, 2007 11:59 AM
> > To: MapGuide Internals Mail List
> > Subject: Re: [mapguide-internals] RFC required for tile
>
> caching
>
> > changes?
> >
> > FWIW, Traian's suggestion for naming folders based on a mod of
>
> r/
>
> c is
> > essentially the same approach I took with ka-Map (for the same
>
>
>
> reason
> > that Chris brought this up) and it seems to work very well.
>
> In ka-
>
> > Map,
> > tiles are rendered in blocks of about 2000x2000 pixels (tile
> size is
> > configurable but you might get 10 x 10 tiles in a typical
> > configuration) and they would go in one of these sub-
> directories. I
> > think I figured out that this would be enough directories to
> decently
> > cache a pretty large area (say, all of the US) without running
>
> into
>
> > file
> > system limits for most scales.
> >
> > It would be interesting to know the limits on:
> >
> > * files per directory
> > * directories per directory
> > * total number of directories
> > * total number of files
> > * minimum block(?) size in the file system compared to typical
>
> size
>
> > of a
> > tile (wasted space per tile).
> >
> > for various operating systems/file system combinations. I
>
> know
>
> we use
> > ReiserFS because it handles lots of small files better than
>
> ext3
>
> for
> > instance. Knowing some of this might help make the caching
>
> system
>
> > more
> > tunable in different environments.
> >
> > Paul
> >
> > On 13-Jan-07, at 11:09 AM, Traian Stanev wrote:
> >
> >> Can you also give an example for tiles with negative indices?
> >> For example if in the current scheme the tile is -33, -64,
>
> what
>
> would
> >> be the resulting directory path?
> >>
> >> R-2/C-3/-3_-4
> >>
> >> What about tile -1,-1
> >>
> >> Another suggestion I have -- name the folder part of the path
>
>
>
> (30 *
> >> tile index div 30) and then the file part would be (tile
>
> index mod
>
> >> 30).
> >>
> >> So for tile 33,64 the folder would be:
> >>
> >> Base Layer Group/30/60/3_4.png
> >>
> >> And for tile -33,-64 the folder would be:
> >>
> >> Base Layer Group/-30/-60/-3_-4.png
> >>
> >> For tile 1,1, the folder would be:
> >>
> >> Base Layer Group/0/0/1_1.png
> >>
> >> For tile -1,-1 the folder would be:
> >>
> >> Base Layer Group/-0/-0/-1_-1.png
> >>
> >>
> >> It's a little weird around 0, but it allows for arbitrary
> >> groupings of
> >
> >> tiles in folders (they don't have to be 30x30) since the tile
>
>
>
> index
> >> can be computed directly from the file path, by adding the
>
> folder
>
> >> term
> >
> >> to the file path term.
> >>
> >> Also, can you tell us what the problem is with having too
>
> many
>
> images
> >> in one directory? Is it a file system issue? And how many
>
> tiles is
>
> >> the
> >
> >> limit, i.e. why is 30x30 the preferred grouping and not
> 1000x1000 for
> >> example?
> >>
> >> Traian
> >> -----Original Message-----
> >> From: mapguide-internals-bounces at lists.osgeo.org on behalf of
>
>
>
> Jason
> >> Birch
> >> Sent: Fri 1/12/2007 8:57 PM
> >> To: MapGuide Internals Mail List
> >> Cc:
> >> Subject: RE: [mapguide-internals] RFC required for tile
>
> caching
>
> >> changes?
> >>
> >> Can I forward this to the "tiling" list for comment?
> >>
> >>
> >> From: mapguide-internals-bounces at lists.osgeo.org on behalf of
>
>
>
> Chris
> >> Claydon
> >> Sent: Fri 2007-01-12 12:24 PM
> >> To: MapGuide Internals Mail List
> >> Subject: RE: [mapguide-internals] RFC required for tile
>
> caching
>
> >> changes?
> >>
> >> A sample tile location would be as follows:
> >>
> >>
> >> ......\
>
> Repositories\TileCache\Samples_Sheboygan_Maps_Sheboygan\7
>
> >> \Base
> >> Layer Group\R2\C3\3_4.png
> >>
> >>
> >> The R and C indices correspond to rows and columns where each
>
> row
>
> >> contains 30 tiles vertically and each column contains 30
>
> tiles
>
> >> horizontally.
> >>
> >>
> >> This indicates that:
> >>
> >>
> >> 1) We're at zoom level 7
> >>
> >> 2) We're in grouped row 2
> >>
> >> 3) We're in grouped column 3
> >>
> >> 4) The tile location within this grouped row/column
> >> combination is (3,4).
> >>
> >>
> >> The R and C indices never take a zero value (to avoid issues
>
> with
>
> >> negative values around the origin), so this tile corresponds
>
> to a
>
> >> location of (33, 64) in the old scheme.
> >>
> >>
> >> I took a quick look at the Tile Map Service Specification,
>
> but it
>
> >> doesn't appear to support the concept of grouping blocks of
>
> tiles
>
> >> into subfolders in its current form. Can you provide more
> >> information on whether this is possible?
> >>
> >>
> >> Chris.
> >>
> >>
> >> _______________________________________________
> >> mapguide-internals mailing list
> >> mapguide-internals at lists.osgeo.org
> >> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
> >
> >
>
> +-----------------------------------------------------------------+
>
> > |Paul Spencer pspencer at dmsolutions.ca
>
> |
>
> >
>
> +-----------------------------------------------------------------+
>
> > |Chief Technology Officer
>
> |
>
> > |DM Solutions Group Inc
>
> http://www.dmsolutions.ca/ |
>
> >
>
> +-----------------------------------------------------------------+
>
> >
> >
> >
> >
> > _______________________________________________
> > mapguide-internals mailing list
> > mapguide-internals at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapguide-internals
> >
> > _______________________________________________
> > mapguide-internals mailing list
> > mapguide-internals at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>
>
>
> +-----------------------------------------------------------------+
>
> |Paul Spencer pspencer at dmsolutions.ca
>
> |
>
>
>
>
> +-----------------------------------------------------------------+
>
> |Chief Technology Officer
>
> |
>
> |DM Solutions Group Inc
>
> http://www.dmsolutions.ca/ |
>
>
>
>
> +-----------------------------------------------------------------+
>
>
>
>
>
> _______________________________________________
> mapguide-internals mailing list
> mapguide-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>
>
> <winmail.dat>
> _______________________________________________
> mapguide-internals mailing list
> mapguide-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>
>
> +-----------------------------------------------------------------+
> |Paul Spencer pspencer at dmsolutions.ca |
> +-----------------------------------------------------------------+
> |Chief Technology Officer |
> |DM Solutions Group Inc http://www.dmsolutions.ca/ |
> +-----------------------------------------------------------------+
>
>
>
>
>
> +-----------------------------------------------------------------+
> |Paul Spencer pspencer at dmsolutions.ca |
> +-----------------------------------------------------------------+
> |Chief Technology Officer |
> |DM Solutions Group Inc http://www.dmsolutions.ca/ |
> +-----------------------------------------------------------------+
>
>
>
>
> _______________________________________________
> mapguide-internals mailing list
> mapguide-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>
>
>
> <winmail.dat>
> _______________________________________________
> mapguide-internals mailing list
> mapguide-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
+-----------------------------------------------------------------+
|Paul Spencer pspencer at dmsolutions.ca |
+-----------------------------------------------------------------+
|Chief Technology Officer |
|DM Solutions Group Inc http://www.dmsolutions.ca/ |
+-----------------------------------------------------------------+
More information about the mapguide-internals
mailing list