GeoTIFF overviews / TILEINDEX / Large dataset performance

[Gambin Dejan]

Ed, Frank,

Thanks very much for the explanations. I realise that the first and the
main thing is to understand my application/usage model before doing any
performance tests/optimizations. My GIS site is now in experimental
phase and I just want to learn as much as I can so that I really KNOW
what to do when the site becomes productional, with many users and maybe
(I hope not) performance problems. I just want to be ready if and when
this comes and that will be very soon. I am working in a local
municipality where many GIS data exists and I wanted to show to my
"bosses" that amazingly good things can be done using open source
software like MapServer. It is not easy to explain to the city
government people the open source philosophy and its benefits. Because
if something bad happens when the production period comes, then all my
efforts will fail and they will be the winners (they - those who are
prefering commercial solutions). As there is none here that can support
such kind of software as the MapServer, then I have to be ready and
armed with knowledge as much as I can.

Well, I have made a big digression from the theme, but I hope you can
understand me and my tiring and boring questions I occupy you with.

Ed, can you tell me how can I see if resampling is occuring to me? I
have set a LOG and DEBUG parameter in map file but I haven't got any
output. Then I saw a bug 709 that discusses this problem. I have set
CONFIG MS_ERRORFILE mapserver.log instead of LOG parameter and I have
seen it is being filled but no resampling messages appear. Must I have
some kind of debug build to enable logging? Is there a way to find out
if resampling occurs (it should occur because I have unconstrained
zooming enabled and built internal overviews with just a few different
resolutions)?

thanks very much

dejan gambin

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Ed McNierney
Sent: Tuesday, June 14, 2005 3:15 AM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] GeoTIFF overviews / TILEINDEX / Large
dataset performance


Dejan -

That's close, but not quite correct.  It is true that as a larger number
of files need to be read to display the map, the TILEINDEX becomes less
useful.  The TILEINDEX helps MapServer efficiently select which input
images are needed - if you need all of them, there's no value in
efficiently selecting the right ones.

There is a separate, independent effect due to the resampling of the
source data.  It is more expensive to resample an image ("zoom out" on
it) than to display it at "native" resolution.  So to create the most
efficient source data for any given map request, you would (ideally)
read as few files as possible AND resample each file as little as
possible.

Before doing lots of performance tests, first understand your
user/application model.  You don't need to optimize performance for
scenarios that won't happen.  If, for example, your application allows
the user to select from a short list of pre-specified view scales, you
should make sure your data is stored at exactly those scales, with no
resampling needed.  If the map image size is constrained, then you can
also organize your source data so a large fraction of map image requests
can be served by only reading one image.

If you have external constraints on your source data (other requirements
than MapServer) then you might not be able to do all these things.  But
define and understand the usage model first, then test and optimize for
that model's requirements.

	- Ed

Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396
ed at topozone.com

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Dejan Gambin
Sent: Monday, June 13, 2005 5:22 PM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] GeoTIFF overviews / TILEINDEX / Large
dataset performance

Thanks Frank,

Does it mean the following (as discussed in "Tileindex questions"
thread):

When viewing the map at a larger scale (zoomed in) the TILEINDEX is very
helpfull because only a small number of tiles have to be retrieved. As
the map scale becomes smaller (zoomed out), more files need to be
touched so TILEINDEX becomes less useful. The performance degrades and
when it becomes unacceptable the it is the time to create a new layer
that is built on lower resolution image(s). This new layer can be built
as one image or as a set of tile images but with the smaller number of
tiles so that Mapserver needs to open a smaller number of files.

If this is so then I suppose it is best to do a lot of performance tests
(as Mark said) to see what is the best combination. Just tell me if this
is this is right.

regards, dg

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Frank Warmerdam
Sent: Monday, June 13, 2005 3:56 PM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] GeoTIFF overviews / TILEINDEX / Large
dataset performance


On 6/13/05, Gambin Dejan <Dejan.Gambin at pula.hr> wrote:
> 1. Frank talks about different types of tilings and overviews:
>         - is the tiling type related to overview type and how? To be
> more concrete, if I have a set of tiff files (and i am using tileindex

> shapefile to associate thme into one layer) can I use internal
> overviews? Which are the possible combinations? What are the 
> advantages and disadvantages? Is it better to use internal 
> overviews/tiles or what?

Gambin, 

My suggestion is to use internal overviews on your base files, but still
to provide *some* additional overview map layers.  The built-in
overviews work fine for extracting overview images efficiently, but if
the user zooms out a long ways such that they are looking at hundreds of
files, the very action of opening and parsing the headers of the many
files will end up slowing things down alot. 

So the "map overview layers" are essentially intended to prevent
MapServer from having to touch too many files for any one render
request. 

> 3. This is not so much related to overviews/tiles but I would like to
> know if there is any difference in using tiffs with their associating 
> world files or real Geotiffs that have the geocoding informations 
> embedded within the file? Is there any benefits to convert to "real"
> geotiff files using for example geotifcp utility?