Mapserver WFS Request Caching

[Ed McNierney]

Tim -

First, let's not confuse Emmanuel - the scenario you describe is a
completely different situation than what he was asking about.  The
discussion here isn't relevant to his problem, and my earlier replies
are not relevant to this problem!

You're reporting that WFS requests made by MapServer to a remote WFS
server have their responses cached locally, and subsequent requests use
the cached replies.  Is that correct?  I'm a little confused by your
last point - if the request is for a similar set of layers but a
different BBOX, are you saying the (wrong) cached data is used, or does
the correct data get used, overwriting the cached file?  Thanks.

	- Ed

Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
ed at topozone.com
(978) 251-4242 

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of tim
Sent: Friday, March 03, 2006 9:40 AM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] Mapserver caching and map generation

On Wed, 2006-03-01 at 16:19 -0500, Ed McNierney wrote:
> Emmanuel -
> 
> The client ALWAYS requests an image that's "not in the cache".  It's 
> not a cache.  It's a temporary directory where MapServer writes files.

> If five clients request the same image from MapServer five times in 
> one second, MapServer will ALWAYS create five different temporary 
> files, and will return five unique URLs to five different files - even

> though they're all the same image.
> 
> 	- Ed

I have seen this appear to happen when 'chaining' wfs requests.  The
setup is as follows:



Central wfs/wms server   ----->   agency wms/wfs server ---> sde server

The central server 'chains' to multiple agency servers that point to the
sde server of the respective data custodian.  When a request on a layer
is made against the central server it then makes a request on the
relevant agency server which returns the required gml which is stored in
a temp file on the central server.  So far, so good.  We can turn off
the agency box and make the same request of the central server and get
the same response back, despite the agency box being turned off.

This behaviour was picked up by one of our testers helping evaluate
various ogc servers for a large whole of govt implementation.

His report to me:
Tim,
    MapServer appears to use the tmp directory as a cache (intentionally
or not) during cascaded WFS requests. MapServer stores the GML returned
from the remote service in the tmp directory. Subsequent WFS requests
for the same layer are not forwarded to the remote server (nothing in
Apache logs on the remote); however, deleting the contents of the tmp
directory causes the next WFS request to be forwarded to the remote
service.

Interestingly, the file names in the tmp directory appear to be related
to the request: similar requests (same layers, different bbox) cause the
same file name to be created in the tmp.

Regards, Geoff

This was with 4.6.2.  Haven't tried it yet with 4.8.1.  Is it possible a
bug has crept in?

Regards,
Tim Bowden
> 
> Ed McNierney
> President and Chief Mapmaker
> TopoZone.com / Maps a la carte, Inc.
> 73 Princeton Street, Suite 305
> North Chelmsford, MA  01863
> ed at topozone.com
> (978) 251-4242
> 
> -----Original Message-----
> From: Emmanuel BELO [mailto:emmanuel.belo at microgis.ch]
> Sent: Wednesday, March 01, 2006 4:15 PM
> To: Ed McNierney
> Cc: MAPSERVER-USERS at LISTS.UMN.EDU
> Subject: Re: [UMN_MAPSERVER-USERS] Mapserver caching and map 
> generation
> 
> To clarify my first question:
> 
> If a client requests an image that doesn't yet exists in the cache, 
> the application will launch the process to create the file (I suppose 
> with a pseudo-random filename) and give back the link. That's ok.
> Now imagine you have 5 more requests for the same image (same query) 
> while the first one is still being processed.
> Will the application detect that and queue these requests for a while 
> to give them the link created by the first once it has finished, or 
> will it just launch five other processes, just as if it was other 
> queries, resulting of the creation of 6 images that will be exactly 
> the same (result of the same query) ?
> I tend to imagine that actually the second case will happen, but maybe

> is there some kind of a "query caching mechanism" that may take care 
> of that and avoid duplicated work...
> 
> regards,
> 
> eb
> 
> Ed McNierney wrote:
> > Emmanuel -
> > 
> > Each map request generates a unique image, and the URL for that 
> > image
> is
> > returned to the client browser requesting that image.  Since your 
> > Web server can process several HTTP requests at one time, there can 
> > be several MapServer instances all running at the same time.
> > 
> > There will be some interaction with the disk subsystem on which the 
> > temporary images are written, and from which they are read by the
> client
> > browsers.  Some disk systems can handle multiple simultaneous
> read/write
> > requests better than others.  Nothing will break - but some disk 
> > subsystems will take longer than others to respond when multiple 
> > reads and writes are happening.
> > 
> > In case you haven't discovered it yet, using MapServer as a WMS 
> > server or in the CGI "mode=map" mode, the output image is written 
> > directly to the client and no temporary files are written to disk.
> > 
> >      - Ed
> > 
> > Ed McNierney
> > President and Chief Mapmaker
> > TopoZone.com / Maps a la carte, Inc.
> > 73 Princeton Street, Suite 305
> > North Chelmsford, MA  01863
> > ed at topozone.com
> > (978) 251-4242
> > 
> > -----Original Message-----
> > From: UMN MapServer Users List 
> > [mailto:MAPSERVER-USERS at LISTS.UMN.EDU]
> On
> > Behalf Of Emmanuel BELO
> > Sent: Wednesday, March 01, 2006 12:26 PM
> > To: MAPSERVER-USERS at LISTS.UMN.EDU
> > Subject: [UMN_MAPSERVER-USERS] Mapserver caching and map generation
> > 
> > Hello,
> > 
> > after cleaning the cache, mapserver generates the new images on
> request.
> > 
> > What happens if 10 users ask for the same map during the generation 
> > of the first map? Are their request delayed or will 10 processes in 
> > parallel be initiated?
> > 
> > Best regards,
> > 
> > Emmanuel BELO
> > emmanuel.belo at microgis.ch
--
Mapforge Geospatial
Level 3/ 267 St Georges Tce
Perth 6000
Western Australia