GeoTIFF overviews / TILEINDEX / Large dataset performance

[Ed McNierney]

Dejan -

That's close, but not quite correct.  It is true that as a larger number
of files need to be read to display the map, the TILEINDEX becomes less
useful.  The TILEINDEX helps MapServer efficiently select which input
images are needed - if you need all of them, there's no value in
efficiently selecting the right ones.

There is a separate, independent effect due to the resampling of the
source data.  It is more expensive to resample an image ("zoom out" on
it) than to display it at "native" resolution.  So to create the most
efficient source data for any given map request, you would (ideally)
read as few files as possible AND resample each file as little as
possible.

Before doing lots of performance tests, first understand your
user/application model.  You don't need to optimize performance for
scenarios that won't happen.  If, for example, your application allows
the user to select from a short list of pre-specified view scales, you
should make sure your data is stored at exactly those scales, with no
resampling needed.  If the map image size is constrained, then you can
also organize your source data so a large fraction of map image requests
can be served by only reading one image.

If you have external constraints on your source data (other requirements
than MapServer) then you might not be able to do all these things.  But
define and understand the usage model first, then test and optimize for
that model's requirements.

	- Ed

Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396
ed at topozone.com

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Dejan Gambin
Sent: Monday, June 13, 2005 5:22 PM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] GeoTIFF overviews / TILEINDEX / Large
dataset performance

Thanks Frank,

Does it mean the following (as discussed in "Tileindex questions"
thread):

When viewing the map at a larger scale (zoomed in) the TILEINDEX is very
helpfull because only a small number of tiles have to be retrieved. As
the map scale becomes smaller (zoomed out), more files need to be
touched so TILEINDEX becomes less useful. The performance degrades and
when it becomes unacceptable the it is the time to create a new layer
that is built on lower resolution image(s). This new layer can be built
as one image or as a set of tile images but with the smaller number of
tiles so that Mapserver needs to open a smaller number of files.

If this is so then I suppose it is best to do a lot of performance tests
(as Mark said) to see what is the best combination. Just tell me if this
is this is right.

regards, dg

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Frank Warmerdam
Sent: Monday, June 13, 2005 3:56 PM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] GeoTIFF overviews / TILEINDEX / Large
dataset performance


On 6/13/05, Gambin Dejan <Dejan.Gambin at pula.hr> wrote:
> 1. Frank talks about different types of tilings and overviews:
>         - is the tiling type related to overview type and how? To be 
> more concrete, if I have a set of tiff files (and i am using tileindex

> shapefile to associate thme into one layer) can I use internal 
> overviews? Which are the possible combinations? What are the 
> advantages and disadvantages? Is it better to use internal 
> overviews/tiles or what?

Gambin, 

My suggestion is to use internal overviews on your base files, but still
to provide *some* additional overview map layers.  The built-in
overviews work fine for extracting overview images efficiently, but if
the user zooms out a long ways such that they are looking at hundreds of
files, the very action of opening and parsing the headers of the many
files will end up slowing things down alot. 

So the "map overview layers" are essentially intended to prevent
MapServer from having to touch too many files for any one render
request. 

> 3. This is not so much related to overviews/tiles but I would like to 
> know if there is any difference in using tiffs with their associating 
> world files or real Geotiffs that have the geocoding informations 
> embedded within the file? Is there any benefits to convert to "real"
> geotiff files using for example geotifcp utility?