Mapserver caching and map generation

[tim]

On Wed, 2006-03-01 at 16:19 -0500, Ed McNierney wrote:
> Emmanuel -
> 
> The client ALWAYS requests an image that's "not in the cache".  It's not
> a cache.  It's a temporary directory where MapServer writes files.  If
> five clients request the same image from MapServer five times in one
> second, MapServer will ALWAYS create five different temporary files, and
> will return five unique URLs to five different files - even though
> they're all the same image.
> 
> 	- Ed

I have seen this appear to happen when 'chaining' wfs requests.  The
setup is as follows:



Central wfs/wms server   ----->   agency wms/wfs server ---> sde server

The central server 'chains' to multiple agency servers that point to the
sde server of the respective data custodian.  When a request on a layer
is made against the central server it then makes a request on the
relevant agency server which returns the required gml which is stored in
a temp file on the central server.  So far, so good.  We can turn off
the agency box and make the same request of the central server and get
the same response back, despite the agency box being turned off.

This behaviour was picked up by one of our testers helping evaluate
various ogc servers for a large whole of govt implementation.

His report to me:
Tim,
    MapServer appears to use the tmp directory as a cache (intentionally
or not) during cascaded WFS requests. MapServer stores the GML returned
from the remote service in the tmp directory. Subsequent WFS requests
for the same layer are not forwarded to the remote server (nothing in
Apache logs on the remote); however, deleting the contents of the tmp
directory causes the next WFS request to be forwarded to the remote
service.

Interestingly, the file names in the tmp directory appear to be related
to the request: similar requests (same layers, different bbox) cause the
same file name to be created in the tmp.

Regards, Geoff

This was with 4.6.2.  Haven't tried it yet with 4.8.1.  Is it possible a
bug has crept in?

Regards,
Tim Bowden
> 
> Ed McNierney
> President and Chief Mapmaker
> TopoZone.com / Maps a la carte, Inc.
> 73 Princeton Street, Suite 305
> North Chelmsford, MA  01863
> ed at topozone.com
> (978) 251-4242  
> 
> -----Original Message-----
> From: Emmanuel BELO [mailto:emmanuel.belo at microgis.ch] 
> Sent: Wednesday, March 01, 2006 4:15 PM
> To: Ed McNierney
> Cc: MAPSERVER-USERS at LISTS.UMN.EDU
> Subject: Re: [UMN_MAPSERVER-USERS] Mapserver caching and map generation
> 
> To clarify my first question:
> 
> If a client requests an image that doesn't yet exists in the cache, the
> application will launch the process to create the file (I suppose with a
> pseudo-random filename) and give back the link. That's ok.
> Now imagine you have 5 more requests for the same image (same query)
> while the first one is still being processed.
> Will the application detect that and queue these requests for a while to
> give them the link created by the first once it has finished, or will it
> just launch five other processes, just as if it was other queries,
> resulting of the creation of 6 images that will be exactly the same
> (result of the same query) ?
> I tend to imagine that actually the second case will happen, but maybe
> is there some kind of a "query caching mechanism" that may take care of
> that and avoid duplicated work...
> 
> regards,
> 
> eb
> 
> Ed McNierney wrote:
> > Emmanuel -
> > 
> > Each map request generates a unique image, and the URL for that image
> is
> > returned to the client browser requesting that image.  Since your Web
> > server can process several HTTP requests at one time, there can be
> > several MapServer instances all running at the same time.
> > 
> > There will be some interaction with the disk subsystem on which the
> > temporary images are written, and from which they are read by the
> client
> > browsers.  Some disk systems can handle multiple simultaneous
> read/write
> > requests better than others.  Nothing will break - but some disk
> > subsystems will take longer than others to respond when multiple reads
> > and writes are happening.
> > 
> > In case you haven't discovered it yet, using MapServer as a WMS server
> > or in the CGI "mode=map" mode, the output image is written directly to
> > the client and no temporary files are written to disk.
> > 
> >      - Ed
> > 
> > Ed McNierney
> > President and Chief Mapmaker
> > TopoZone.com / Maps a la carte, Inc.
> > 73 Princeton Street, Suite 305
> > North Chelmsford, MA  01863
> > ed at topozone.com
> > (978) 251-4242  
> > 
> > -----Original Message-----
> > From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU]
> On
> > Behalf Of Emmanuel BELO
> > Sent: Wednesday, March 01, 2006 12:26 PM
> > To: MAPSERVER-USERS at LISTS.UMN.EDU
> > Subject: [UMN_MAPSERVER-USERS] Mapserver caching and map generation
> > 
> > Hello,
> > 
> > after cleaning the cache, mapserver generates the new images on
> request.
> > 
> > What happens if 10 users ask for the same map during the generation of
> > the first map? Are their request delayed or will 10 processes in
> > parallel be initiated?
> > 
> > Best regards,
> > 
> > Emmanuel BELO
> > emmanuel.belo at microgis.ch
-- 
Mapforge Geospatial
Level 3/ 267 St Georges Tce
Perth 6000
Western Australia