VS: [mapserver-users] How to improve raster imagery display ?
Rahkonen Jukka
Jukka.Rahkonen at mmmtike.fi
Tue Sep 16 02:00:26 PDT 2008
Hi,
I do not quite undertand this:
> > But our main concern are the scales 2000 and 1000, where
> > the displaying
> > process is producing many dozens of tiles, each 256 x 256
> > pixel .png-files.
> > Along with the other layers (where the vector-data also are
> > displayed by
> > rasterized tiles which at least are much smaller), the
> > composition of one
> > screen sometimes takes up to 3 minutes !
It seems to me that your server and client might want to work in different ways. Client is willing to have small tiles in fixed scales, and rendering them on demand is not very effective. Usually these clients are served with prerendered tiles from some caching system like tilecache, geowebcache of Kamap.
I am pretty sure that if you have a try with standard WMS client that is requesting just one output you will feel much better response times. My guess is something around one second at maximum.
ECW format has internal tiling system which works just fine. I fear you won't win anything by creating subsampled ecw images to be used as overviews. Instead, your ecw images might work faster if you mosaic and compress them all the one single ecw image. In that case there would be always just one image file opening per request.
My experience about ecw and Mapserver is that this combination gives about half of the speed achieved by uncompressed geotiff images, with 5 percent of the disk usage. But this is the case in WMS usage and not with a client requesting tiles. Tiles should be server from tilecache and that would be super fast, but then you would need a lot of disk space.
-Jukka Rahkonen-
> -----Alkuperäinen viesti-----
> Lähettäjä: mapserver-users-bounces at lists.osgeo.org
> [mailto:mapserver-users-bounces at lists.osgeo.org] Puolesta
> Rainer Kalliany
> Lähetetty: 16. syyskuuta 2008 11:32
> Vastaanottaja: Ed McNierney
> Kopio: mapserver-users at lists.osgeo.org
> Aihe: Re: [mapserver-users] How to improve raster imagery display ?
>
> Dear Ed (and all other newsgroup folks),
>
> Many thanks for your comment and link on previous postings,
> which I tried to explore. The follow-up posting on "Ed's
> rules for best raster performance"
> by Brent indeed seems to contain the essence of your recommendations.
> As it is often with good advice, all appear to be very
> reasonable to me.
>
> Since I cannot install additional servers via WMS and things
> like that (many thanks to those who suggested that, but right
> now that is not feasible for our project; possibly we come
> back to that at a much later stage ...) I am developing a
> strategy which I think is in accordance with most of "Ed's rules".
>
> Our basic datset is covering more than 12,000
> square-kilometres (=one of 9 provinces of Austria) at a
> ground resolution of 0.25 metres. By uncompressed
> 24-bit-RGB-TIFF-Files that would be some 600 GByte of data.
> Indeed not so much in our days. However, our customer does
> not want to provide more than the 30 GB or so webspace, which
> are necessary at reasonable ECW compression.
>
> The actual web-users will work on comparatively small project-areas of
> 1 to 30 square-kilometres size, demandig reasonable
> display-performance.
>
> No my plan, which (with one exception) should follow Ed's
> rules, is like that:
>
> 1) Resample the Orthos to all required resolutions (0.25,
> 0.5, 1.25, 2.5, 12.5
> metres) but store them as ECW (what Ed might not like,
> however ...)
>
> 2) Increase viewing-TileWidth/TileHeight to 512 by 512 or 1024 by 1024
>
> 3) When a new project-area is defined by a user: Convert all
> tiles necessary
> for that area at all scales from ECW to TIFF or PNG
> (launching a GDAL-job).
> These tile-files (with their a bit weird names and
> directory-structure)
> then will stay on the server as long as the project is
> active (what
> might be for a few weks or months)
>
> 4) When there is a display-request: Feed exactly the tiles
> needed right into
> the display-process, bypassing all
> ortho-searching/resampling/reprojecting
> usually done by MapServer at this stage
>
> Since by now this is just a planned strategy and not yet
> implemented (currently I in particular do not really know how
> to skip certain parts of MapServer functionality and
> introduce my own stuff at step 4), I am most gratefule for
> any comment and hint on my plans.
>
> Good luck in particular to Ed for his future activities (but
> I still hope he never will find the unsubscribe button here
> in that group ...) and best regards to all other folks,
>
> Rainer
>
> 2008-09-16, 8:32 UTC
> ---
> Ed McNierney schrieb:
> > Folks -
> >
> > I would suggest searching the mailing list archives for the
> many posts
> > mentioning "McNierney", or "TopoZone" or "raster" or all three, as
> > I've posted at length and repeatedly on this sort of topic before.
> > Others have done so as well - I just remember my own
> writings better
> > <g>. I successfully served over 50 TB of raster imagery to
> relatively
> > heavy user loads with MapServer, and commented frequently
> on the approach I used.
> >
> > The most important point is to preprocess as much as possible. In
> > this case that means (a) avoiding resampling by storing the data at
> > the permitted output resolutions, and (b) storing the data in an
> > uncompressed simple raster format. PNG is an option, as is
> TIFF. If
> > you need 24-bit data you may want to consider tiled TIFFs
> using JPEG
> > compression as a compromise (again, search the archives for this
> > topic, especially comments by Frank Warmerdam).
> >
> > - Ed
> >
> > Ed McNierney
> > 205 Indian Hill Road
> > Groton, MA 01450
> > ed at mcnierney.com
> > +1 (978) 761-0049
> >
> >
> >
> > On 9/13/08 8:13 PM, "pcreso at pcreso.com" <pcreso at pcreso.com> wrote:
> >
> > Hi Rainer,
> >
> > Firstly, are you using FastCGI? If not, implementing
> this may help.
> > I assume you have indexes to all the raster images.
> >
> > I'd also consider an architectural approach, where you
> abstract the
> > raster data to a custom built system & access this via
> WMS. We have
> > improved performance of such systems by having mapfiles
> access data
> > from PostGIS databases on disparate servers rather than one
> > congested system.
> >
> > Similarly, for rasters, you may also be able to use
> hardware rather
> > than software to address your performance issues by
> splitting your
> > raster files up between servers, effectively a further level of
> > tiling. With caching (as below), multiple users hitting the same
> > region should generally hit the cache, and users
> hitting multiple
> > regions will have their load spread across different servers.
> >
> > Look at http://geowebcache.org/
> > While it possibly integrates better with GeoServer than
> mapserver,
> > it will supposedly work with any WMS server.
> >
> > Given the cheap price of hardware & memory/cpu performance these
> > days, adding a couple of multi-core boxes with a few GB
> of memory &
> > a Tb drive or two each isn't a huge outlay, & Gb LAN is
> also cheap.
> >
> >
> >
> > I guess when your programming skills are as bad as
> mine, you look at
> > other solutions first :-)
> >
> > I'm interested in hearing about your project, we are in
> the process
> > of accumulating around 11Tb of aerial imagery & will be
> looking at
> > ways to provide access to this.
> >
> >
> > Cheers,
> >
> > Brent Wood
> >
> >
> > --- On Sun, 9/14/08, Rainer Kalliany <rainer at kalliany.at> wrote:
> >
> > > From: Rainer Kalliany <rainer at kalliany.at>
> > > Subject: [mapserver-users] How to improve raster
> imagery display ?
> > > To: mapserver-users at lists.osgeo.org
> > > Date: Sunday, September 14, 2008, 11:24 AM
> > > Dear all,
> > >
> > > Yet for some months, my wife (who is the far more
> > > experienced programmer,
> > > and me (rather responsible for
> > > collecting/re-formatting/providing data)
> > > are developing a MapServer-based webGIS. Your community
> > > already has
> > > helped us several times a lot with your experience, and we
> > > are most
> > > grateful for that great support.
> > >
> > > Now, with our project almost finished, the so far pretty
> > > poor performance
> > > in displaying raster images (aerial orthophotos at a ground
> > > resolution of
> > > 0.25 metres) is becoming a critical issue. In particular
> > > when several
> > > users want to work at the same time, what happened last
> > > week :-(
> > >
> > > So far we had our orthophotos organized in a seamless grid
> > > of ECW-images,
> > > 10000 by 10000 pixel each (corresponding to 2500 by 2500
> > > metres on ground).
> > >
> > > From that dataset our MapServer produces the
> > > "orthophoto" layer on demand,
> > > at scales of 1 by 1000, 2000, 5000 or 10000. With the
> > > standard settings
> > > of 72 dpI, that renders pixels of 0.353, 0.706, 1.764 or
> > > 3.528 metres.
> > >
> > > I am well aware that at least for the 10000 scale I should
> > > introduce
> > > a layer with reduced resolution. I already have prepared
> > > such a dataset.
> > >
> > > But our main concern are the scales 2000 and 1000, where
> > > the displaying
> > > process is producing many dozens of tiles, each 256 x 256
> > > pixel .png-files.
> > > Along with the other layers (where the vector-data also are
> > > displayed by
> > > rasterized tiles which at least are much smaller), the
> > > composition of one
> > > screen sometimes takes up to 3 minutes !
> > >
> > > NOW MY IDEA is to produce all those .PNGs (with their weird
> > > directory
> > > structure and naming, like
> > > \t-86016\l3072\t-84992l4352.png, which how-
> > > ever I already have understood) in advance and myself. For
> > > the conversion
> > > from ECW to PNG I might use GDAL, but without any
> > > resampling necessary.
> > >
> > > I also consider raising tilewidth/tileheight from the
> > > standard value 256
> > > to 512 or even 1024 pixels.
> > >
> > > Now my question is, if that is a good idea or not.
> > >
> > > If you disagree with me: Do you have some alternative
> > > proposal ?
> > >
> > > If you think my planned approach basically is a good one,
> > > do you have
> > > any supporting advices, on issues I possibly did not
> > > consider ?
> > >
> > > THis is becoming a dcisive isuue for the success of our
> > > whole project,
> > > and therefore we will be most grateful for any hint !
> > >
> > > With kind regards,
> > >
> > > Rainer & Susanne Kalliany
> > >
> > > 2008-09-13, 23:25 UTC
> > > --
> > > DI. Rainer Kalliany - rainer at kalliany.at
> > >
> > > TeFiS - www.tefis.at
> > > Technische und forstliche Informationssysteme (Forstliche
> > > Software, GIS)
> > > KALLIANY - www.kalliany.at
> > > Ingenieurbüro für Vermessungswesen (Photogrammetrie,
> > > Fernerkundung, GPS)
> > >
> > > Mobiltelephon: +43 664 1811098
> > > Telephon: +43 316 931746-0, Fax: -15
> > >
> > > A-8010 Graz, Schillerstrasse 58
> > > A-1040 Wien, Schaumburgergasse 11/13
> > > AUSTRIA
> > > _______________________________________________
> > > mapserver-users mailing list
> > > mapserver-users at lists.osgeo.org
> > > http://lists.osgeo.org/mailman/listinfo/mapserver-users
> > _______________________________________________
> > mapserver-users mailing list
> > mapserver-users at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapserver-users
> >
>
>
> --
> DI. Rainer Kalliany - rainer at kalliany.at
>
> TeFiS - www.tefis.at
> Technische und forstliche Informationssysteme (Forstliche
> Software, GIS) KALLIANY - www.kalliany.at Ingenieurbüro für
> Vermessungswesen (Photogrammetrie, Fernerkundung, GPS)
>
> Mobiltelephon: +43 664 1811098
> Telephon: +43 316 931746-0, Fax: -15
>
> A-8010 Graz, Schillerstrasse 58
> A-1040 Wien, Schaumburgergasse 11/13
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
More information about the MapServer-users
mailing list