[mapguide-internals] MapGuide RFC 112 - sqlite based tile cache

[Traian Stanev]

Hi Tom,

It would depend on what exactly the problem is -- sure if one is using Windows Explorer drag and drop to backup the files it would be faster to have one file. But if one is using rsync or robocopy (or similar), it still makes sense to use files, since those programs know how to copy only the changed files (or even changed parts of files).

Traian


-----Original Message-----
From: mapguide-internals-bounces at lists.osgeo.org [mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf Of Tom Fukushima
Sent: Thursday, May 12, 2011 1:00 PM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 112 - sqlite based tile cache

Hi Trevor,

What about the main problem: "The file base tile cache can be problematic for backup or copying due to the massive number of files and directories"?  I'm assuming that's the main problem because it was listed first.  How would that be resolved?

Thanks
Tom

-----Original Message-----
From: mapguide-internals-bounces at lists.osgeo.org [mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf Of Trevor Wekel
Sent: Thursday, May 12, 2011 9:57 AM
To: MapGuide Internals Mail List
Subject: RE: [mapguide-internals] MapGuide RFC 112 - sqlite based tile cache

Hi Zac,

From the RFC, the only significant advantage I see to adding SQLite as back end storage is to reduce the number of files being managed.  It seems like a lot of effort for only moderate gain.  Implementing additional "tile management" APIs for MgTileService will make tile cache management easier and does not preclude using SQLite as a back end storage mechanism in a future release.  Using FDO to manage tiles seems like an additional complication for our user base.  

Assuming we stayed with the existing file-based storage mechanism, generating a world file for every created tile and a "config.xml" document for every scale would solve the extents issue.  Adding APIs to MgTileService could resolve the other issues.  For example, MgTileService already knows how to map between extent and tile on disk and an fstat call can determine whether the tile exists.

We should also add "management" APIs to clear and generate the tile set on a background thread within the server and a server side "Tile.log" file for seeing the results of the generation.  The management APIs could be coded to be smart enough to only generate missing tiles or to regenerate the entire set.

ClearTileSet(MgResourceIdentifier* mapDef, CREFSTRING baseMapLayerGroupName, INT32 scaleIndex, boolean regenAll)
GenerateTileSet(MgResourceIdentifier* mapDef, CREFSTRING baseMapLayerGroupName, INT32 scaleIndex, boolean regenAll)
 
Extent-based APIs could also be exposed

GenerateTileSet(MgResourceIdentifier* mapDef, CREFSTRING baseMapLayerGroupName, MgEnvelope* extent, boolean regenAll)

Regards,
Trevor

-----Original Message-----
From: mapguide-internals-bounces at lists.osgeo.org [mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf Of Zac Spitzer
Sent: May 11, 2011 10:13 PM
To: MapGuide Internals Mail List
Subject: Re: [mapguide-internals] MapGuide RFC 112 - sqlite based tile cache

answers inline

On Thu, May 12, 2011 at 12:56 AM, Traian Stanev <traian.stanev at autodesk.com> wrote:
>
> Hi Zac,
>
>
> "Generating tiles is currently done using a rather brute force approach, by storing tiles in a database, it can be queried for existing tiles and only missing tiles can be requested."
>
> What do you mean by brute force? Doesn't tile existence checking come down to a file existence check?

By brute force, I mean that if mapguide goes down in the middle of seeding the tile cache, every tile needs to be requested again, which involves a lot of http overhead.

>
> "Using a sqlite database will use less disk space than a disk based tile cache."
>
> Do you have some numbers about the expected improvement? Inside its file, SQLite is basically a file system which will exhibit incomplete page filling and overflow, similar to a regular file system.

Individual files waste a lot of disk space, depending on the tile size and file system block size, I'm not a expert on this, but I'd expect a sqlitedb to be much more efficent. Similiar to what would be achieved by zipping up a tilecache with zero compression.

>
>
> Traian
>
>
>
> -----Original Message-----
> From: mapguide-internals-bounces at lists.osgeo.org
> [mailto:mapguide-internals-bounces at lists.osgeo.org] On Behalf Of Zac 
> Spitzer
> Sent: Saturday, May 07, 2011 3:12 AM
> To: MapGuide Internals Mail List
> Subject: [mapguide-internals] MapGuide RFC 112 - sqlite based tile 
> cache
>
> http://trac.osgeo.org/mapguide/wiki/MapGuideRfc112
>
> --
> Zac Spitzer
> Solution Architect / Director
> Ennoble Consultancy Australia
> http://www.ennoble.com.au
> http://zacster.blogspot.com
> +61 405 847 168
> _______________________________________________
> mapguide-internals mailing list
> mapguide-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>
> _______________________________________________
> mapguide-internals mailing list
> mapguide-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapguide-internals
>
>



--
Zac Spitzer
Solution Architect / Director
Ennoble Consultancy Australia
http://www.ennoble.com.au
http://zacster.blogspot.com
+61 405 847 168
_______________________________________________
mapguide-internals mailing list
mapguide-internals at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapguide-internals