Serving raster images via MapServer?

[Ed McNierney]

Paul -

It's good to hear someone's really doing that, and I agree that Armin's
approach is excellent.

However, I would not want readers of this thread to think the PostGIS is
the "best" way to manage TILEINDEXes, or that it is somehow necessary to
get good performance.  For standard TILEINDEX usage, the use of a
shapefile index is perfectly fine.  Armin's approach is intended for
users who need an additional level of selection, by providing a way to
create a TILEINDEX with *additional* attribute columns, giving the user
control over which subset of images in a single TILEINDEX are displayed
in response to a particular query.

	- Ed

Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396
ed at topozone.com 

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Paul Ramsey
Sent: Thursday, March 30, 2006 8:27 PM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] Serving raster images via MapServer?

Wow, Armin's solution (using PostGIS as the tileindex) is well nigh
perfect.  Have an image metadata table with nice information like
capture date, sensor, whatnot, and the path reference to the raw data...
then using the arbitrary SQL capabilities of Mapserver/ PostGIS set up
to do the "latest-and-greatest" query.  For a PoC you could just do a
latest-limit-1 query.  For a real one, you'll probably have to do some
pl/pgsql in order to reduce the return set to just the minimal latest
overlapping set.  Could be v. v. slick.

P.

On Mar 30, 2006, at 7:47 AM, Thom DeCarlo wrote:

> Hey Ed,
> (So, you and Norm are over here, too. Small world!)
>
> This sounds good. I've been reading up on the gdaltindex and I think I

> understand how to get it set up. But, what is this shptree thing you 
> mentioned?
> Also, two other issues. First, the client is going to be continually 
> requesting map updates and the server will be getting frequent 
> additions to its repository. If the tile index is held in a shapefile 
> will the MapServer re-read that file when a new client request comes 
> in or is it only read when Mapserver starts?
> Second, since I've already got a PostgreSQL server managing the vector

> data, can the tile index be held in it, too? I'm going to have a lot 
> of overlapping imagery and it seems (to the novice that I am) that SQL

> queries could find the best/newest data to serve out to the client. 
> Or, is this just unnecessary overhead?
>
> Thanks!
> Thom
>
> --
> Thom DeCarlo
> --------------------------------------------------------
>       Any sufficiently advanced technology
>     is indistinguishable from a rigged demo.
>
>
>> -----Original Message-----
>> From: Ed McNierney
>> Sent: Wednesday, March 29, 2006 3:36 PM
>> To: MAPSERVER-USERS at LISTS.UMN.EDU
>> Subject: Re: [UMN_MAPSERVER-USERS] Serving raster images via 
>> MapServer?
>>
>> Thom -
>>
>> There are a few general principles for working with large raster data

>> sets in MapServer.  Some of these are generalizations that apply to 
>> many similar applications and aren't necessarily MapServer-specific.
>>
>> If your imagery is updated with any frequency, you'll want to 
>> simplify things and avoid lots of complex fiddling with the images.  
>> For most users, it's good to keep the data on disks organized in a 
>> manner that's consistent with the way you receive the data and 
>> updates.
>>
>> For performance reasons, GeoTIFF is almost always the preferred 
>> format.
>> Compression generally costs you in performance, and particularly 
>> costs you in scalability; some decompressors run quickly but require 
>> lots of RAM.  That's fine for a small number of users, but if you 
>> scale up you may suffer as you can run out of physical RAM quickly.  
>> Disk space is cheap - I never compress imagery.
>>
>> If you can structure your GeoTIFFs to be internally tiled (using
>> gdal_translate) with overviews (via gdaladdo) you can get GeoTIFFs 
>> that can be read quickly at a variety of scale levels.
>>
>> Organize your GeoTIFFs using TILEINDEX shapefiles (via gdaltindex) 
>> and index those shapefiles with shptree.
>>
>> Reprojection of raster data is faster than many folks think, but it's

>> much slower than reprojecting vector data.  Where possible, store 
>> your imagery in the same projection as you serve it.
>>
>> Keeping things simple is good - it's really not that complicated.   
>> I'm
>> currently providing WMS service (via MapServer) to just over 40 
>> terabytes of imagery in around 500,000 image files, using roughly the

>> sort of system described above.
>>
>> 	- Ed
>>
>> Ed McNierney
>> President and Chief Mapmaker
>> TopoZone.com / Maps a la carte, Inc.
>> 73 Princeton Street, Suite 305
>> North Chelmsford, MA  01863
>> ed at topozone.com
>> (978) 251-4242
>>
>> -----Original Message-----
>> From: Thom DeCarlo
>> Sent: Wednesday, March 29, 2006 3:06 PM
>> To: MAPSERVER-USERS at LISTS.UMN.EDU
>> Subject: [UMN_MAPSERVER-USERS] Serving raster images via MapServer?
>>
>> Hi,
>> I've got a fairly large archive of raster images that we would like 
>> to deliver to remote clients through WMS. I've still got much to 
>> learn about the technology. I thought I should ask the first obvious 
>> question here.
>>
>> Is there an accepted "best way" to organize and deliver imagery using

>> MapServer? I have a PostgreSQL/PostGIS server set up to handle our 
>> vector data, but there doesn't seem to be support for raster data in 
>> that database.
>>
>> Thanks,
>> Thom
>> --
>> Thom DeCarlo
>> --------------------------------------
>>      That Which Does Not Kill Me
>>      Can Still Hurt Really Bad.