Help with rasters

Fri Oct 20 15:45:02 PDT 2006

Jeff -

It sounds like you're viewing the entire extent of your TILEINDEX, so
you're asking MapServer to open and render the contents of "thousands"
of JPEG files.  I think 10 minutes sounds pretty good <g>.

The first question you should ask is whether you're actually going to
run into that scenario in practice, or whether you just did something
unusual.  If you actually intend to do this more than once, you have
essentially three major performance problems:

1. You're reading thousands of files.
2. You're zooming out so far you're only using a tiny bit of data from
each file.
3. Your files are JPEGs.

In order to access the data in a JPEG you need to decode the entire
file.  So for each JPEG you are opening that file, reading the entire
file from disk, decoding the entire image into memory, picking a few
dozen pixels out of that image (or even fewer) and then repeating that
process thousands of times.  That's painful.

I listed the three problems above in their order of importance.  In the
ideal scenario, an optimum MapServer raster request would:

1. Open exactly one TIFF image file.
2. Read and use the entire contents of that file.
3. Do so without any resampling.

In other words, you would have the extent and scale of one of your input
images exactly match the extent and scale of your requested output
image.  That's obviously not going to happen, but it's the goal you
should try to approach.

By far the biggest problem in this situation is that you're opening
thousands of files to generate one output image.  Disk seeks (for each
new file) are just about the slowest thing your server does.  Disk reads
are the next slowest.  The overviews suggested (using gdaladdo) will
allow resampled low-resolution overviews to be stored with your files.
This will help with disk reading, because MapServer will be able to read
only the (small) low-resolution overview rather than reading lots of
high-resolution data and then throwing most of it away.  But if you've
still got thousands of files, you'll still have a big problem.  In that
case you want to mosaic those files together to create a smaller number
of input files.

It all depends on what exactly your usage goals are.  If you intend to
actually use this data at full zoomed-in resolution, you may not have
much of a problem at all.  If you modify your application to let you
zoom in on the imagery, you'll see that as you zoom further in
performance will improve dramatically.

	- Ed

Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396
ed at topozone.com

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Jeff Dege
Sent: Friday, October 20, 2006 2:24 PM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] Help with rasters

I've been trying to do the same, but the shp2img that came with ms4w
won't output TIFF files.

> -----Original Message-----
> From: Brent Pedersen [mailto:bpederse at nature.Berkeley.EDU]
> Sent: Friday, October 20, 2006 12:38 PM
> To: Jeff Dege
> Cc: MAPSERVER-USERS at LISTS.UMN.EDU
> Subject: Re: [UMN_MAPSERVER-USERS] Help with rasters
> 
> hi, i just did something similar with a bunch of MrSID imagery, i got 
> a bunch of help from #gdal,
> 
> i generated a 15K * 15K pixel (.tiff) overview using shp2img, then 
> (after gdaladdo) used that for large scales in the .map file and only 
> resort to the MrSIDs when zoomed in pretty far.
> 
> by the way, i think it took > 1 hour to generate the 15K by 15K pixel 
> overview. but once it's done...
> 
> 
> -brent
> 
> 
> 
> On Fri, 20 Oct 2006, Jeff Dege wrote:
> 
> > Yep. That did it. I know have an image of the full extent
> of the map.
> >
> > Only problem is that it took ten minutes to generate.
> >
> > Next?  Read up on optimization and tuning of raster maps.
> >
> > Thanks for the help.
> >
> >
> >> -----Original Message-----
> >> From: Bart van den Eijnden (OSGIS) [mailto:bartvde at osgis.nl]
> >> Sent: Friday, October 20, 2006 11:49 AM
> >> To: Jeff Dege
> >> Cc: MAPSERVER-USERS at LISTS.UMN.EDU
> >> Subject: Re: [UMN_MAPSERVER-USERS] Help with rasters
> >>
> >> First of all, change:
> >>
> >> DATA "rasterindex"
> >>
> >> into:
> >>
> >> TILEINDEX "rasterindex"
> >>
> >> Best regards,
> >> Bart
> >>
> >> Jeff Dege schreef:
> >>> A customer has sent us a bunch of mapdata, to see if we
> can make any
> >>> sense from it.  I've been messing around with mapserver
> for a while,
> >>> now, and I've figured out how to work some of its
> >> capabilities, but this
> >>> is my first attempt to handle rasters.
> >>>
> >>> The data set has four sets of shape files, three of which are 
> >>> point/line/polygon data and the fourth which seems to be a
> >> tileindex for
> >>> the thousands of jpg and jgw files.
> >>>
> >>> I'm starting simple - writing a map file and trying to get
> >> a map image
> >>> using shp2img.  I have no problem doing this with the three 
> >>> point/line/polygon layers.  But I've not been able to do
> >> this with the
> >>> raster data.
> >>>
> >>> I can run gdalinfo on the jpg files:
> >>>
> >>>   Driver: JPEG/JPEG JFIF
> >>>   Size is 1610, 1609
> >>>   Coordinate System is `'
> >>>   Origin = (-93.112361,44.933805)
> >>>   Pixel Size = (0.00001279,-0.00000900)
> >>>   Corner Coordinates:
> >>>   Upper Left  ( -93.1123608,  44.9338046)
> >>>   Lower Left  ( -93.1123608,  44.9193245)
> >>>   Upper Right ( -93.0917678,  44.9338046)
> >>>   Lower Right ( -93.0917678,  44.9193245)
> >>>   Center      ( -93.1020643,  44.9265646)
> >>>   Band 1 Block=1610x1 Type=Byte, ColorInterp=Red
> >>>   Band 2 Block=1610x1 Type=Byte, ColorInterp=Green
> >>>   Band 3 Block=1610x1 Type=Byte, ColorInterp=Blue
> >>>
> >>> I can run gdaltindex to create my own tileindex shape file:
> >>>
> >>>   gdaltindex  \\server\share\test\data\rasterindex
> >>> \\server\share\test\data\data\*.jpg
> >>>
> >>> And I can run ogrinfo on the generated shapefile:
> >>>
> >>>   INFO: Open of `\\server\share\app\data'
> >>>   using driver `ESRI Shapefile' successful.
> >>>
> >>>   Layer name: rasterindex
> >>>   Geometry: Polygon
> >>>   Feature Count: 683
> >>>   Extent: (-93.338884, 44.455961) - (-92.721094, 44.933805)
> >>>   Layer SRS WKT:
> >>>   (unknown)
> >>>   location: String (255.0)
> >>>
> >>> But when I run shp2img, I get nada.  With DEBUG=ON:
> >>>
> >>>   Unable to open file rasterindex for layer A ... ignoring
> >> this missing
> >>> data.
> >>>   msDrawMap(): Layer 0 (A), 0.000s
> >>>   msDrawMap(): Drawing Label Cache, 0.000s
> >>>   msDrawMap() total time: 0.000s
> >>>
> >>> The rastedindex shape files (.dbf, .shp, and .shf) files
> >> are in the same
> >>> directories as the other point/line/polygon shape files
> >> that I've been
> >>> able to successfully draw layers with using this mapfile.
> >> Ogrinfo has
> >>> no trouble accessing the file.
> >>>
> >>> The underlying .jpg files are in the same directory, and
> >> gdalinfo and
> >>> gdaltindex had no trouble opening them.
> >>>
> >>> The layer in the mapfile itself is pretty simple:
> >>>
> >>>   LAYER
> >>>     NAME "A"
> >>>     TYPE RASTER
> >>>     STATUS ON
> >>>     DATA "rasterindex"
> >>>   END
> >>>
> >>> Any hints as to what I'm doing wrong?
> >>>
> >>>
> >>>
> >>
> >>
> >> --
> >> Bart van den Eijnden
> >> OSGIS, Open Source GIS
> >> bartvde at osgis.nl
> >> http://www.osgis.nl
> >>
> >>
> >
> 