[mapserver-users] Image Speed Questions---Generic

[Paul Ramsey]

Could I get some enlightenment from the raster experts on the benefits
of various image formats?

Right now, I have 25metre relief for all of BC, in about 120 files, in
PNG format. I am guessing that I can radically improve my speed by
changing to a format which does not require decompression. Would TIFF be
a faster format? Assuming it is, Ed mentions this concept of
strips-and-lines with respect to TIFF files: is this some way of making
random access of the tiff files faster? What do the concepts refer to,
and what is the best balance of the variables?

Thanks,
Paul

Antonio Cosenza wrote:
> 
> Il 22:09, lunedì 30 luglio 2001, scrivesti:
> > Lowell -
> >
> > Since November, 1999, the TopoZone has been in the business of serving
> > "high-res imagery" "for a large spatial extent" by providing USGS DRG
> > (topographic) maps of the entire US online.  We're now migrating that
> > application to a MapServer platform so we can add some interesting stuff
> > to it.  I've been in and out of the PC graphics biz for about 20 years
> > now, and I've learned a bit about performance.  I wanted to take the
> > time to write a thorough commentary on raster tips, but that's going to
> > take me a while.  Here's a shorter version along with a promise for a
> > more extensive treatment someday as soon as I can.
> >
> > For computer graphics in general I like to live by two golden rules -
> > (1) precompute the hell out of everything, and (2) trivially reject as
> > much as you can as fast as you can.  Here are some notes on both rules.
> >
> > 1. Precompute everything - if you're trying to serve a lot of users on
> > the Web with relatively inexpensive hardware, you'll find that a fast
> > RAID-5 SCSI array is much cheaper than lots of RAM and CPU.  Wherever
> > it's possible, think hard about how you can precompute raster results
> > rather than doing them on the fly.  Look at a simple case - zooming out
> > from a raster image that's viewed at 100% (1:1) zoom.  When you zoom out
> > 4x, you're reading 16 times more input pixels than you are writing
> > output pixels.  At 16x it's 256 times more; you're basically reading a
> > lot of data and throwing 99.5% of it away.  For the TopoZone site we
> > chose to offer four fixed zoom levels - period.  They are all
> > precomputed and there is no on-the-fly scaling.  This is a more
> > reasonable approach for topo maps, because they're useful (i.e.
> > readable) only at a relatively small range of scales.  For aerial or
> > satellite imagery, or other continuous-tone imagery, you can zoom out
> > farther and still have things work.
> >
> > Using GDAL support with overviews (built with GDALADDO) is a very easy
> > way to do this.  Precomputing zoom scales offers a tradeoff between disk
> > space and speed - the more space you use, the faster things will go.
> > However, the highest-level overviews require the smallest amount of
> > space but actually save the most time.  If you created only one 4x
> > overview, it would (roughly) increase your storage space by only about
> > 7% (since the 4x view is 1/16th the size of the original) but for all
> > views zoomed out beyond 4x you're going to deal with 1/16th as many
> > pixels.  Creating lots of overviews (2, 4, 8, 16, 32, etc.) gives you
> > the best performance, but think very hard about whether you really need
> > that 2x overview.  Your mileage may vary based on usage, CPU speed, RAM,
> > disk performance, etc. so you really should test things yourself in your
> > real environment.
> >
> > 2. Trivially reject as much as you can - "trivial rejection" is the
> > process of very quickly deciding which input data is not needed in the
> > output map, and ignoring it as quickly as possible.  For raster data
> > sets, the way to do this is with a TILEINDEX shapefile.  By reading ONE
> > input fileset (the TILEINDEX shapefile), you can very quickly determine
> > exactly which TIFF input files are needed to create the requested map.
> > This is enormously faster than reading lots of TIFF/GeoTIFF files,
> > extracting the bounding box from each, and then discovering that you
> > never needed to open most of them in the first place.  Disk seek time is
> > expensive compared to sequential reading, so reading 100,000 bytes from
> > one file is much, much faster than reading 100 bytes from each of 1,000
> > files.
> >
> > In addition to these general rules, make sure your images don't make
> > life difficult for MapServer.  For TIFF images, this particularly means
> > making sure they are created with only one (or a few) rows per strip.
> > Some image libraries create single-strip TIFF files, with all the rows
> > in one strip.  If you are using any sort of data compression (PackBits,
> > for example) MapServer will need to read and interpret (i.e. decompress)
> > all the data in that strip up to the point you need.  For example, if
> > you've got a 7,000-row file in one strip, and you need to display the
> > bottom few rows of that file, MapServer will need to read from disk AND
> > decompress essentially the entire file just to get a few lines from it.
> > If you have one row per strip, you will get less effective compression
> > (because each row is compressed independently) but you will only need to
> > read the rows you need.  A compromise with a small number of rows per
> > strip might give you better compression with acceptable performance.
> >
> > Finally, know your data and your users.  Some data types compress better
> > than others.  Some are virtually incompressible (at least within the
> > world of formats TIFF supports).  Some data types are used differently
> > than others - as I suggested above, a 32x zoom out on a 1-meter aerial
> > photo is a somewhat reasonable thing to ask for, while the same zoom on
> > a topographic map produces illegible mush.
> >
> > I'm still shuffling around some R&D equipment, but sometime this week
> > I'll post a link to a MapServer site that offers a 30-meter shaded
> > relief map of the entire US and 2.5-meter topographic maps of the entire
> > US.  Together they're over 60,000 files with about 300 GB of data, and
> > they illustrate some of the things I've talked about here.
> >
> >       - Ed
> >
> > Ed McNierney
> > Chief Mapmaker
> > TopoZone.com
> > ed at topozone.com
> > (978) 251-4242
> >
> 
> Thank you very much Ed.
> What you said has been very important for me, cause i'm working on my TESI,
> and my problem was to decide how serve big satellites image using mapserver.
> I like the 'trivial rejection' rule, it's fantastic. But i have one more
> question about tileindex shapefile. If i have a very big raster image should
> i have to manually create a set of little tiles images or can I use some
> program that can do it quicly for me?
> Thanks a lot to you and to all mapserver community
> Toni