[Gdal-dev] GeoQuadTree - an open format for storinggeoreferencedimages

Tue Nov 14 21:06:04 EST 2006

Simon -

TILEINDEX is a MapServer/quasi-GDAL mechanism for handling a large
number of rasters as a single unit.  The rasters don't need to be the
same size, and can overlap - I suspect they can even be different file
formats, although I've never tried, and they can be stored anywhere in
the file system.

The TILEINDEX scheme uses an ESRI shapefile (with an optional quadtree
index on it) as a spatial index for the rasters.  One polygon is
generated for each raster, with that polygon representing the bounding
box of that file.  The path to the raster file itself is stored as a
string attribute of the polygon.  Applications use the spatial/geometry
features of the polygon to determine which input image(s) if any
intersect the area of interest for drawing.

It's not the same sort of thing, exactly, and there are no tools I'm
aware of for managing the individual image organization on disk.  But it
works well for me.  I use TILEINDEX structures to manage about 500,000
individual raster files that are about 40 terabytes altogether (haven't
counted lately <g>).  It could certainly use enhancement, but it might
be a good starting point for such a project.

    - Ed
Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396
ed at topozone.com 

________________________________

From: Simon Perkins [mailto:sy at perkins.net] 
Sent: Tuesday, November 14, 2006 8:52 PM
To: Ed McNierney
Cc: Brent Fraser; Jordi Gilabert Vall; gdal-dev at lists.maptools.org
Subject: Re: [Gdal-dev] GeoQuadTree - an open format for
storinggeoreferencedimages

TIFF has some pretty serious file size limitations so wouldn't be good
for these kinds of large rasters. I'm not familiar with TILEINDEX, but
is this a MapServer related thing? Is that applicable to local image
storage on the desktop?

I think the GeoQuadTree idea is interesting - it doesn't rely on large
file support on the OS (not that that's a serious problem for any modern
OSes), and dealing with lots of small files could be more efficient than
one large file in some cases I would think.

As for PNG vs JPEG, it depends on whether you're happy to accept lossy
compression or not. For many applications, customers are averse to
losing any resolution. It might make sense to make the underlying file
format in GeoQuadTree flexible acording to application. Actually, I'd
prefer to see a format that can handle multispectral files - PNG and
JPEG are both limited to 1 or 3 bands, aren't they? Or are multiple
bands stored as separate grayscale images?

I would guess that with very large rasters, some sort of pyramid scheme
becomes important in addition to the tiling, so that if you just want to
get an overview of the whole image, you don't have to read every single
file. Does GeoQuadTree handle that?

But, Ed is right that storing very large rasters is a not a new problem.
So, what do people do? I guess that at the high end, outfits like Google
Earth use a spatial database to organize multiple individual raster
files and then stich them together. Could somebody outline the solution
used by the Virtual Terrain project?

Cheers,

Sy

Ed McNierney wrote: 

	Jordi -

	You said you wanted to "avoid the use of tile indexes or
catalogues of images", but isn't that exactly what your XML
catalogue/index does?  It seems that the GeoQuadTree format is a
different form of that same sort of structure.  There are several
different ways of doing this now, including the TILEINDEX mechanism,
tiled TIFF files, etc.  I'm sure there are limitations to each, but I'm
not sure that yet another tiling scheme will help.

	In particular, PNG is not the best format for all images, and
it's important to support other encoding mechanisms, especially JPEG.
Photographic images are huge when stored in PNG format, and JPEG is
usually a much better choice.  Conversely, scanned line art and
synthetic images generally compress and store better as PNG images.

	    - Ed
	Ed McNierney
	President and Chief Mapmaker
	TopoZone.com / Maps a la carte, Inc.
	73 Princeton Street, Suite 305
	North Chelmsford, MA  01863
	ed at topozone.com
	(978) 251-4242 

________________________________

	From: gdal-dev-bounces at lists.maptools.org
[mailto:gdal-dev-bounces at lists.maptools.org] On Behalf Of Brent Fraser
	Sent: Tuesday, November 14, 2006 3:05 PM
	To: Jordi Gilabert Vall; gdal-dev at lists.maptools.org
	Subject: Re: [Gdal-dev] GeoQuadTree - an open format for
storinggeoreferencedimages

	For those interested tiling,

	  There is similar tiling related discussion/work going on at:

	        http://lists.eogeo.org/mailman/listinfo/tiling
<http://lists.eogeo.org/mailman/listinfo/tiling>  (do they have a web
page?)

	    and 
	            http://www.stereofx.org/terrain.html, implemented in
http://vterrain.org/, particularly VTBuilder
(http://vterrain.org/Doc/VTBuilder/overview.html)

	Brent Fraser

		----- Original Message ----- 
		From: Jordi Gilabert Vall <mailto:jordi at geoquadtree.org>

		To: gdal-dev at lists.maptools.org 
		Sent: Tuesday, November 14, 2006 3:25 AM
		Subject: [Gdal-dev] GeoQuadTree - an open format for
storing georeferencedimages

		Hi,

		Some time ago I needed the retrieval from very large
georeferenced raster images in a OGC WMS server, and I wanted to avoid
the use of tile indexes or catalogues of images, neither a database. I
started thinking of an open format for storing arbitrarily large
georeferenced images. I named this format "GeoQuadTree", as it would be
based on a quadtree of rectangular tiles, each in PNG format on the
filesystem, in a simple hierarchical structure of folders. I wrote a
command line utility for creating it, importing from PNG/JPEG/TIFF and
exporting to PNG/JPEG/TIFF/GDAL. I also wrote a GDAL driver for this
format. I tested successfully with my own WMS server software, and on
MapServer compiled with GDAL support. It worked very well on Blue Marble
Next Generation at a resolution of 15 arc-seconds, i.e. 86400x43200
pixels. 

		You can find more information on the project web site:

		http://geoquadtree.org/ 

		I'm testing the release 1.0.0, you can test it on the
subversion repository if you want (I haven't packaged it yet).

		I think it's a useful format, open, very easy to use,
and very efficient (in terms of response time).

		Do you think it could be useful for you ? Would you like
to include it on GDAL's next release ?

		jordi at geoquadtree org

________________________________

		_______________________________________________
		Gdal-dev mailing list
		Gdal-dev at lists.maptools.org
		http://lists.maptools.org/mailman/listinfo/gdal-dev

________________________________

	_______________________________________________
	Gdal-dev mailing list
	Gdal-dev at lists.maptools.org
	http://lists.maptools.org/mailman/listinfo/gdal-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20061114/5a3102e9/attachment.html