[fdo-internals] Raster Catalogs (aka Tileindexes)

[Frank Warmerdam]

Traian Stanev wrote:
> Hi Frank,
>  
> Here my thoughts on this. A raster catalog does not need to be exposed 
> at the API level. For example, one way to do this would be to "connect" 
> the raster provider to a directory containing a large number of rasters. 
> Internally, the provider could index these in whatever way it wants.

Traian,

Sometimes the rasters may be spread in a variety of locations, so one
advantage of a catalog is allowing this as opposed to the current
requirement (I guess unless a config.xml file is used) to have them all
in one directory.

But yes, I can see that "auto-indexing" could be built quite easily, and
would not complicate the user experience at all.

BTW, I hadn't intended to expose this all as new API calls - but rather
as some sort of extra config variables or something like that for the
provider.

> I would personally use an SDF file for the spatial index -- like you 
> say, insert a polygon corresponding to the bounds of the raster and use 
> the raster name (or index) as ID property. An SHP file will probably 
> work fine too, I've seen lots of raster catalogs stored in shp files before.

Well, I suggest shapefiles because I have lots of experience using
them as catalogs, and because gdaltindex can already create them.
The problem with SDF at this point is I don't know how to do anything
with them short of creating a custom FDO application.  Clearly it would
be a step forward for me to build an OGR-FDO driver, so I can use my
normal OGR tools to read, create and manipulate FDO feature stores
like SDF.

I've also had this idea that SDF support needs to be extracted from
FDO and made into an easy to use standalone library.  But I haven't
had time to even contemplate this which would be a substantial project.
And, as has been pointed out to me, forking a copy of the SDF code
from FDO would make it harder for the SDF format to evolve.

> One thing I always wondered is whether it's worth it to persist the 
> index into a file or is it easier to initialize it the first time an FDO 
> connection is opened and then keep it in memory until the connection is 
> closed. Given that FDO connections remain open for a while, it seems 
> like keeping it in memory is not such a bad idea, and avoids the trouble 
> of dealing with an index file.

This of course is something I have trouble wrapping my mind around
coming from the very stateless (or perhaps better to say all-state-in-files)
world of mapserver.  You are correct that indexing in memory would go a
long ways towards resolving performance issues with large numbers of files
in a directory.

I guess someone would need to restart the mapguide service if they wanted
to ensure things were re-indexed?

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org