Cluster/Supercomputer/HPC variants of Mapserver

[Ed McNierney]

Biz -

We were all talking about optimization of the data sets, not the code.

It is not clear that your "parallelism" approach described below will
help very much.  It is quite possible that it will make things worse.
There are quite a few high-performance MapServer implementations in
existence already; you seem to be under the impression that your
application is quite unusual, and that may not be the case.

But if you really can't provide any more information about your
application, it's very difficult for us to provide any help.  Please
remember, however, that whether you are working on one server or
multiple servers, well-organized data will help you considerably.

	- Ed

Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396
ed at topozone.com

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Biz King
Sent: Friday, January 20, 2006 4:53 AM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] Cluster/Supercomputer/HPC variants of
Mapserver

Whilst wholeheartedly agreeing that the code requires substantial
optimisation, I think I have arrived at a reasonably logical process
that allows me to create a limited parallelization of the serial process
of drawing maps.
I intend to segment (tile?) the extents of the parameters being rendered
by different machines into a series of proportional grids, send these
parallel requests to cluster/Xgrid members, get the results back
(probably as a stream) and then join the stream images back together
into one coherent image and save that to the central file system which
the web servers access.

We're actually having to produce live feedback of item locations (they
can move, but can also remain static!) AND item activity (We're subject
to NDA, and cannot give enough detail to explain clearly!  
Which is irritating.) which we then have to place into a graphical
format (map with colour-coded dots indicating location, activity and
status). Just to make things interesting, the extents of the data being
displayed changes over time, so a series of pre-defined layer images
just won't work as there are no 'standard' coordinate extents that we
can work to.

The client wants as near real-time feedback as is possible, which
compounds things even more!

I'll report back with more information as soon as I can.

cheers
Biz
On 19 Jan 2006, at 16:47, David Bitner wrote:

> Without getting into any clustering, there is probably a lot of 
> optimization that you could do to your datasets.  There are a number 
> of posts in the archives for this list and documentation on 
> mapserver.gis.umn.edu on doing things like creating overviews at 
> different resolutions and tiling for rasters that could likely help 
> speed up the process.
>
>
> On 1/19/06, Biz King <biz.king at mac.com> wrote:
>> Hi All.
>>
>> Is anyone aware of anywhere (or better still, has experience of) 
>> running Mapserver via an MPI/Grid interface or as a cluster?
>>
>> We're trying to develop a high-performance mapserver that can cope 
>> with the load we're going to be throwing at it!  Currently it takes
>> 298 seconds (on a Mac OSX Server, 3.5 Gb Ram, dual 2Ghz processors) 
>> to do what we need done on under 60 seconds!  There's not much we can

>> do to cut down the load as we're creating a whole series of nodes on 
>> a layer via a database and we're then creating the imagery based on 
>> these items and outputting them to graphics formats in varying sizes.
>>
>> The results get fed to users on demand without the delays associated 
>> with 'on the fly' image creation.
>>
>> Any help will be welcomed!
>>
>> cheers
>>
>> Biz
>>