Help with rasters
Jeff Dege
jdege at KORTERRA.COM
Mon Oct 23 10:31:03 EDT 2006
For this app, we'll need to allow the user to zoom from roughly 1 pixel
= 10 km to 1 pixel = 10 meters, with intermediate zoom levels at ratios
of 1/2.
Which is a bunch.
I never expected the data to work efficiently, as-is. What I'm in the
process of learning is what I need to do to speed things up.
To date, I've used gdalwarp to create a single 16000x16000 .tif overview
file (which took 54 hours to generate). The hints I've seen in the
online docs, browsing the archives of this forum, etc., said that it is
possible to create additional overlay layers within this .tif file,
though I haven't had time to start, yet.
Or time to see how fast the map generates using this .tif.
I think I'm heading in the right direction, but I have a good ways to
go, yet.
> -----Original Message-----
> From: Ed McNierney [mailto:ed at topozone.com]
> Sent: Friday, October 20, 2006 5:45 PM
> To: Jeff Dege; MAPSERVER-USERS at LISTS.UMN.EDU
> Subject: RE: [UMN_MAPSERVER-USERS] Help with rasters
>
> Jeff -
>
> It sounds like you're viewing the entire extent of your TILEINDEX, so
> you're asking MapServer to open and render the contents of "thousands"
> of JPEG files. I think 10 minutes sounds pretty good <g>.
>
> The first question you should ask is whether you're actually going to
> run into that scenario in practice, or whether you just did something
> unusual. If you actually intend to do this more than once, you have
> essentially three major performance problems:
>
> 1. You're reading thousands of files.
> 2. You're zooming out so far you're only using a tiny bit of data from
> each file.
> 3. Your files are JPEGs.
>
> In order to access the data in a JPEG you need to decode the entire
> file. So for each JPEG you are opening that file, reading the entire
> file from disk, decoding the entire image into memory, picking a few
> dozen pixels out of that image (or even fewer) and then repeating that
> process thousands of times. That's painful.
>
> I listed the three problems above in their order of
> importance. In the
> ideal scenario, an optimum MapServer raster request would:
>
> 1. Open exactly one TIFF image file.
> 2. Read and use the entire contents of that file.
> 3. Do so without any resampling.
>
> In other words, you would have the extent and scale of one of
> your input
> images exactly match the extent and scale of your requested output
> image. That's obviously not going to happen, but it's the goal you
> should try to approach.
>
> By far the biggest problem in this situation is that you're opening
> thousands of files to generate one output image. Disk seeks (for each
> new file) are just about the slowest thing your server does.
> Disk reads
> are the next slowest. The overviews suggested (using gdaladdo) will
> allow resampled low-resolution overviews to be stored with your files.
> This will help with disk reading, because MapServer will be
> able to read
> only the (small) low-resolution overview rather than reading lots of
> high-resolution data and then throwing most of it away. But if you've
> still got thousands of files, you'll still have a big
> problem. In that
> case you want to mosaic those files together to create a
> smaller number
> of input files.
>
> It all depends on what exactly your usage goals are. If you intend to
> actually use this data at full zoomed-in resolution, you may not have
> much of a problem at all. If you modify your application to let you
> zoom in on the imagery, you'll see that as you zoom further in
> performance will improve dramatically.
>
> - Ed
>
> Ed McNierney
> President and Chief Mapmaker
> TopoZone.com / Maps a la carte, Inc.
> 73 Princeton Street, Suite 305
> North Chelmsford, MA 01863
> Phone: +1 (978) 251-4242
> Fax: +1 (978) 251-1396
> ed at topozone.com
>
> -----Original Message-----
> From: UMN MapServer Users List
> [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
> Behalf Of Jeff Dege
> Sent: Friday, October 20, 2006 2:24 PM
> To: MAPSERVER-USERS at LISTS.UMN.EDU
> Subject: Re: [UMN_MAPSERVER-USERS] Help with rasters
>
> I've been trying to do the same, but the shp2img that came with ms4w
> won't output TIFF files.
>
> > -----Original Message-----
> > From: Brent Pedersen [mailto:bpederse at nature.Berkeley.EDU]
> > Sent: Friday, October 20, 2006 12:38 PM
> > To: Jeff Dege
> > Cc: MAPSERVER-USERS at LISTS.UMN.EDU
> > Subject: Re: [UMN_MAPSERVER-USERS] Help with rasters
> >
> > hi, i just did something similar with a bunch of MrSID
> imagery, i got
> > a bunch of help from #gdal,
> >
> > i generated a 15K * 15K pixel (.tiff) overview using shp2img, then
> > (after gdaladdo) used that for large scales in the .map
> file and only
> > resort to the MrSIDs when zoomed in pretty far.
> >
> > by the way, i think it took > 1 hour to generate the 15K by
> 15K pixel
> > overview. but once it's done...
> >
> >
> > -brent
> >
> >
> >
> > On Fri, 20 Oct 2006, Jeff Dege wrote:
> >
> > > Yep. That did it. I know have an image of the full extent
> > of the map.
> > >
> > > Only problem is that it took ten minutes to generate.
> > >
> > > Next? Read up on optimization and tuning of raster maps.
> > >
> > > Thanks for the help.
> > >
> > >
> > >> -----Original Message-----
> > >> From: Bart van den Eijnden (OSGIS) [mailto:bartvde at osgis.nl]
> > >> Sent: Friday, October 20, 2006 11:49 AM
> > >> To: Jeff Dege
> > >> Cc: MAPSERVER-USERS at LISTS.UMN.EDU
> > >> Subject: Re: [UMN_MAPSERVER-USERS] Help with rasters
> > >>
> > >> First of all, change:
> > >>
> > >> DATA "rasterindex"
> > >>
> > >> into:
> > >>
> > >> TILEINDEX "rasterindex"
> > >>
> > >> Best regards,
> > >> Bart
> > >>
> > >> Jeff Dege schreef:
> > >>> A customer has sent us a bunch of mapdata, to see if we
> > can make any
> > >>> sense from it. I've been messing around with mapserver
> > for a while,
> > >>> now, and I've figured out how to work some of its
> > >> capabilities, but this
> > >>> is my first attempt to handle rasters.
> > >>>
> > >>> The data set has four sets of shape files, three of which are
> > >>> point/line/polygon data and the fourth which seems to be a
> > >> tileindex for
> > >>> the thousands of jpg and jgw files.
> > >>>
> > >>> I'm starting simple - writing a map file and trying to get
> > >> a map image
> > >>> using shp2img. I have no problem doing this with the three
> > >>> point/line/polygon layers. But I've not been able to do
> > >> this with the
> > >>> raster data.
> > >>>
> > >>> I can run gdalinfo on the jpg files:
> > >>>
> > >>> Driver: JPEG/JPEG JFIF
> > >>> Size is 1610, 1609
> > >>> Coordinate System is `'
> > >>> Origin = (-93.112361,44.933805)
> > >>> Pixel Size = (0.00001279,-0.00000900)
> > >>> Corner Coordinates:
> > >>> Upper Left ( -93.1123608, 44.9338046)
> > >>> Lower Left ( -93.1123608, 44.9193245)
> > >>> Upper Right ( -93.0917678, 44.9338046)
> > >>> Lower Right ( -93.0917678, 44.9193245)
> > >>> Center ( -93.1020643, 44.9265646)
> > >>> Band 1 Block=1610x1 Type=Byte, ColorInterp=Red
> > >>> Band 2 Block=1610x1 Type=Byte, ColorInterp=Green
> > >>> Band 3 Block=1610x1 Type=Byte, ColorInterp=Blue
> > >>>
> > >>> I can run gdaltindex to create my own tileindex shape file:
> > >>>
> > >>> gdaltindex \\server\share\test\data\rasterindex
> > >>> \\server\share\test\data\data\*.jpg
> > >>>
> > >>> And I can run ogrinfo on the generated shapefile:
> > >>>
> > >>> INFO: Open of `\\server\share\app\data'
> > >>> using driver `ESRI Shapefile' successful.
> > >>>
> > >>> Layer name: rasterindex
> > >>> Geometry: Polygon
> > >>> Feature Count: 683
> > >>> Extent: (-93.338884, 44.455961) - (-92.721094, 44.933805)
> > >>> Layer SRS WKT:
> > >>> (unknown)
> > >>> location: String (255.0)
> > >>>
> > >>> But when I run shp2img, I get nada. With DEBUG=ON:
> > >>>
> > >>> Unable to open file rasterindex for layer A ... ignoring
> > >> this missing
> > >>> data.
> > >>> msDrawMap(): Layer 0 (A), 0.000s
> > >>> msDrawMap(): Drawing Label Cache, 0.000s
> > >>> msDrawMap() total time: 0.000s
> > >>>
> > >>> The rastedindex shape files (.dbf, .shp, and .shf) files
> > >> are in the same
> > >>> directories as the other point/line/polygon shape files
> > >> that I've been
> > >>> able to successfully draw layers with using this mapfile.
> > >> Ogrinfo has
> > >>> no trouble accessing the file.
> > >>>
> > >>> The underlying .jpg files are in the same directory, and
> > >> gdalinfo and
> > >>> gdaltindex had no trouble opening them.
> > >>>
> > >>> The layer in the mapfile itself is pretty simple:
> > >>>
> > >>> LAYER
> > >>> NAME "A"
> > >>> TYPE RASTER
> > >>> STATUS ON
> > >>> DATA "rasterindex"
> > >>> END
> > >>>
> > >>> Any hints as to what I'm doing wrong?
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> > >> --
> > >> Bart van den Eijnden
> > >> OSGIS, Open Source GIS
> > >> bartvde at osgis.nl
> > >> http://www.osgis.nl
> > >>
> > >>
> > >
> >
>
More information about the mapserver-users
mailing list