[mapserver-users] How to improve raster imagery display ?
Rainer Kalliany
rainer at kalliany.at
Tue Sep 16 04:31:46 EDT 2008
Dear Ed (and all other newsgroup folks),
Many thanks for your comment and link on previous postings, which I tried
to explore. The follow-up posting on "Ed's rules for best raster performance"
by Brent indeed seems to contain the essence of your recommendations.
As it is often with good advice, all appear to be very reasonable to me.
Since I cannot install additional servers via WMS and things like that
(many thanks to those who suggested that, but right now that is not feasible
for our project; possibly we come back to that at a much later stage ...)
I am developing a strategy which I think is in accordance with most of
"Ed's rules".
Our basic datset is covering more than 12,000 square-kilometres (=one of 9
provinces of Austria) at a ground resolution of 0.25 metres. By uncompressed
24-bit-RGB-TIFF-Files that would be some 600 GByte of data. Indeed not
so much in our days. However, our customer does not want to provide more than
the 30 GB or so webspace, which are necessary at reasonable ECW compression.
The actual web-users will work on comparatively small project-areas of
1 to 30 square-kilometres size, demandig reasonable display-performance.
No my plan, which (with one exception) should follow Ed's rules, is like that:
1) Resample the Orthos to all required resolutions (0.25, 0.5, 1.25, 2.5, 12.5
metres) but store them as ECW (what Ed might not like, however ...)
2) Increase viewing-TileWidth/TileHeight to 512 by 512 or 1024 by 1024
3) When a new project-area is defined by a user: Convert all tiles necessary
for that area at all scales from ECW to TIFF or PNG (launching a GDAL-job).
These tile-files (with their a bit weird names and directory-structure)
then will stay on the server as long as the project is active (what
might be for a few weks or months)
4) When there is a display-request: Feed exactly the tiles needed right into
the display-process, bypassing all ortho-searching/resampling/reprojecting
usually done by MapServer at this stage
Since by now this is just a planned strategy and not yet implemented
(currently I in particular do not really know how to skip certain parts of
MapServer functionality and introduce my own stuff at step 4), I am most
gratefule for any comment and hint on my plans.
Good luck in particular to Ed for his future activities (but I still
hope he never will find the unsubscribe button here in that group ...)
and best regards to all other folks,
Rainer
2008-09-16, 8:32 UTC
---
Ed McNierney schrieb:
> Folks -
>
> I would suggest searching the mailing list archives for the many posts
> mentioning “McNierney”, or “TopoZone” or “raster” or all three, as I’ve
> posted at length and repeatedly on this sort of topic before. Others
> have done so as well – I just remember my own writings better <g>. I
> successfully served over 50 TB of raster imagery to relatively heavy
> user loads with MapServer, and commented frequently on the approach I used.
>
> The most important point is to preprocess as much as possible. In this
> case that means (a) avoiding resampling by storing the data at the
> permitted output resolutions, and (b) storing the data in an
> uncompressed simple raster format. PNG is an option, as is TIFF. If
> you need 24-bit data you may want to consider tiled TIFFs using JPEG
> compression as a compromise (again, search the archives for this topic,
> especially comments by Frank Warmerdam).
>
> - Ed
>
> Ed McNierney
> 205 Indian Hill Road
> Groton, MA 01450
> ed at mcnierney.com
> +1 (978) 761-0049
>
>
>
> On 9/13/08 8:13 PM, "pcreso at pcreso.com" <pcreso at pcreso.com> wrote:
>
> Hi Rainer,
>
> Firstly, are you using FastCGI? If not, implementing this may help.
> I assume you have indexes to all the raster images.
>
> I'd also consider an architectural approach, where you abstract the
> raster data to a custom built system & access this via WMS. We have
> improved performance of such systems by having mapfiles access data
> from PostGIS databases on disparate servers rather than one
> congested system.
>
> Similarly, for rasters, you may also be able to use hardware rather
> than software to address your performance issues by splitting your
> raster files up between servers, effectively a further level of
> tiling. With caching (as below), multiple users hitting the same
> region should generally hit the cache, and users hitting multiple
> regions will have their load spread across different servers.
>
> Look at http://geowebcache.org/
> While it possibly integrates better with GeoServer than mapserver,
> it will supposedly work with any WMS server.
>
> Given the cheap price of hardware & memory/cpu performance these
> days, adding a couple of multi-core boxes with a few GB of memory &
> a Tb drive or two each isn't a huge outlay, & Gb LAN is also cheap.
>
>
>
> I guess when your programming skills are as bad as mine, you look at
> other solutions first :-)
>
> I'm interested in hearing about your project, we are in the process
> of accumulating around 11Tb of aerial imagery & will be looking at
> ways to provide access to this.
>
>
> Cheers,
>
> Brent Wood
>
>
> --- On Sun, 9/14/08, Rainer Kalliany <rainer at kalliany.at> wrote:
>
> > From: Rainer Kalliany <rainer at kalliany.at>
> > Subject: [mapserver-users] How to improve raster imagery display ?
> > To: mapserver-users at lists.osgeo.org
> > Date: Sunday, September 14, 2008, 11:24 AM
> > Dear all,
> >
> > Yet for some months, my wife (who is the far more
> > experienced programmer,
> > and me (rather responsible for
> > collecting/re-formatting/providing data)
> > are developing a MapServer-based webGIS. Your community
> > already has
> > helped us several times a lot with your experience, and we
> > are most
> > grateful for that great support.
> >
> > Now, with our project almost finished, the so far pretty
> > poor performance
> > in displaying raster images (aerial orthophotos at a ground
> > resolution of
> > 0.25 metres) is becoming a critical issue. In particular
> > when several
> > users want to work at the same time, what happened last
> > week :-(
> >
> > So far we had our orthophotos organized in a seamless grid
> > of ECW-images,
> > 10000 by 10000 pixel each (corresponding to 2500 by 2500
> > metres on ground).
> >
> > From that dataset our MapServer produces the
> > "orthophoto" layer on demand,
> > at scales of 1 by 1000, 2000, 5000 or 10000. With the
> > standard settings
> > of 72 dpI, that renders pixels of 0.353, 0.706, 1.764 or
> > 3.528 metres.
> >
> > I am well aware that at least for the 10000 scale I should
> > introduce
> > a layer with reduced resolution. I already have prepared
> > such a dataset.
> >
> > But our main concern are the scales 2000 and 1000, where
> > the displaying
> > process is producing many dozens of tiles, each 256 x 256
> > pixel .png-files.
> > Along with the other layers (where the vector-data also are
> > displayed by
> > rasterized tiles which at least are much smaller), the
> > composition of one
> > screen sometimes takes up to 3 minutes !
> >
> > NOW MY IDEA is to produce all those .PNGs (with their weird
> > directory
> > structure and naming, like
> > \t-86016\l3072\t-84992l4352.png, which how-
> > ever I already have understood) in advance and myself. For
> > the conversion
> > from ECW to PNG I might use GDAL, but without any
> > resampling necessary.
> >
> > I also consider raising tilewidth/tileheight from the
> > standard value 256
> > to 512 or even 1024 pixels.
> >
> > Now my question is, if that is a good idea or not.
> >
> > If you disagree with me: Do you have some alternative
> > proposal ?
> >
> > If you think my planned approach basically is a good one,
> > do you have
> > any supporting advices, on issues I possibly did not
> > consider ?
> >
> > THis is becoming a dcisive isuue for the success of our
> > whole project,
> > and therefore we will be most grateful for any hint !
> >
> > With kind regards,
> >
> > Rainer & Susanne Kalliany
> >
> > 2008-09-13, 23:25 UTC
> > --
> > DI. Rainer Kalliany - rainer at kalliany.at
> >
> > TeFiS - www.tefis.at
> > Technische und forstliche Informationssysteme (Forstliche
> > Software, GIS)
> > KALLIANY - www.kalliany.at
> > Ingenieurbüro für Vermessungswesen (Photogrammetrie,
> > Fernerkundung, GPS)
> >
> > Mobiltelephon: +43 664 1811098
> > Telephon: +43 316 931746-0, Fax: -15
> >
> > A-8010 Graz, Schillerstrasse 58
> > A-1040 Wien, Schaumburgergasse 11/13
> > AUSTRIA
> > _______________________________________________
> > mapserver-users mailing list
> > mapserver-users at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapserver-users
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
--
DI. Rainer Kalliany - rainer at kalliany.at
TeFiS - www.tefis.at
Technische und forstliche Informationssysteme (Forstliche Software, GIS)
KALLIANY - www.kalliany.at
Ingenieurbüro für Vermessungswesen (Photogrammetrie, Fernerkundung, GPS)
Mobiltelephon: +43 664 1811098
Telephon: +43 316 931746-0, Fax: -15
A-8010 Graz, Schillerstrasse 58
A-1040 Wien, Schaumburgergasse 11/13
More information about the mapserver-users
mailing list