[MapServer-users] Can MapServer cache directly to AWS S3?

[Rob Dennett]

Currently, an example of a URL used by one of our front-end apps is

https://myserver.com?map=example.map&mode=tile&tilemode=gmap&tile={x}+{y}+{z}&layers=all&map.imagetype=png

For the XYZ Tile service.

For WMS, it looks like
https://<https://mapserver.tnris.org/?map=/tnris_mapfiles/area_type.map>myserver.com/?map=<https://mapserver.tnris.org/?map=/tnris_mapfiles/area_type.map>example.map<https://mapserver.tnris.org/?map=/tnris_mapfiles/area_type.map>

I am not sure how that would map back to a URL like /cgi-bin/mapserv.fcgi

Would it be just
<url>https://myserver.com/</url>?

Thanks,
Rob


________________________________
From: Richard Greenwood <richard.greenwood at gmail.com>
Sent: Friday, March 7, 2025 8:44 AM
To: Rob Dennett <Rob.Dennett at twdb.texas.gov>
Cc: mapserver <mapserver-users at lists.osgeo.org>
Subject: Re: [MapServer-users] Can MapServer cache directly to AWS S3?


External: Beware of links/attachments.


On Fri, Mar 7, 2025 at 7:29 AM Rob Dennett <Rob.Dennett at twdb.texas.gov<mailto:Rob.Dennett at twdb.texas.gov>> wrote:
So, can you have both the MapCache module and the MapServer CGI script running on the same Apache server?

Sure. There's no conflict.

Is there an example showing how to configure this?

Look at the MapCache documentation. MapCache is configured with an xml file. If MapServer is on the same server then you'd have a statement in the the "source" section like:
     <source name="something" type="wms">
          . . .
          <url>http://localhost/cgi-bin/mapserv.fcgi</url>
          . . .
     </source>

We have a bunch of front ends which use our MapServer (it's customer-facing, too) and I am not clear if all those URLs would have to change.

Your app makes requests to https://youserver/mapcache for tiles or https://yourserver/cgi-bin/mapserv.fcgi for non-tiles requests. MapCache is reading and writing to and from S3, your app is not directly access S3.

Rich


Thanks,
Rob
________________________________
From: Richard Greenwood <richard.greenwood at gmail.com<mailto:richard.greenwood at gmail.com>>
Sent: Friday, March 7, 2025 7:44 AM
To: Rob Dennett <Rob.Dennett at twdb.texas.gov<mailto:Rob.Dennett at twdb.texas.gov>>
Cc: mapserver <mapserver-users at lists.osgeo.org<mailto:mapserver-users at lists.osgeo.org>>
Subject: Re: [MapServer-users] Can MapServer cache directly to AWS S3?


External: Beware of links/attachments.



On Thu, Mar 6, 2025, 3:27 PM Rob Dennett via MapServer-users <mapserver-users at lists.osgeo.org<mailto:mapserver-users at lists.osgeo.org>> wrote:
Does MapCache have to be built from source?

Probably not. If you're on Linux it will be in your distribution's repo. Not sure about Windows or Mac.

Also, does it have to be it's own server?

MapCache is a module that the Apache web server runs.

  It's not clear to me how it connects with MapServer.  Is it the case that MapServer renders tiles as necessary based on requests and then MapCache serves those files if requested again?

No. Apache passes tile requests to MapCache. If MapCache has the tile it gets returned. If MapCache doesn't have the tile it makes a WMS request to MapServer, saves the tile in its cache and returns the tile to the web server. MapCache can save the tile on the local file system, S3 or elsewhere. You configure where it saves the cached tiles.

As mentioned, MapServer is currently storing tiles in a directory mapped to S3 which is specified by IMAGEPATH.

Using MapCache you don't need IMAGEPATH.

These tiles are not pre-seeded, so I can't specify a particular raster or vector file to serve from S3.  I don't think I understand how caching is supposed to work with S3 in the way we're doing it.  Can you simply specify a folder in S3 where cached data lives and have MapServer render anything that doesn't exist yet?

If you are trying to serve tiles directly from S3 by making tile requests directly to the S3 bucket that is a whole different thing and not supported by MapCache.

HTH,
Rich


Thanks,
Rob
________________________________
From: MapServer-users <mapserver-users-bounces at lists.osgeo.org<mailto:mapserver-users-bounces at lists.osgeo.org>> on behalf of Scott via MapServer-users <mapserver-users at lists.osgeo.org<mailto:mapserver-users at lists.osgeo.org>>
Sent: Wednesday, March 5, 2025 11:50 AM
To: mapserver-users at lists.osgeo.org<mailto:mapserver-users at lists.osgeo.org> <mapserver-users at lists.osgeo.org<mailto:mapserver-users at lists.osgeo.org>>
Subject: Re: [MapServer-users] Can MapServer cache directly to AWS S3?

External: Beware of links/attachments.



You can store any raster/vector file format supported by GDAL on S3 and
use it via MapServer. Note the /vsis3 virtual connection:

Raster:
DATA "/vsis3/myS3Bucket/raster.tif"

Vector:
CONNECTION "/vsis3/myS3Bucket/vector.fgb"

MapServer is definitely a WMS server. Any OGC compliant WMS url can be
parsed and processed. Using MapCache along MapServer is highly desirable
as all subsequent WMS requests are cached. Further, you can seed any
cache directly with MapCache.

Scott


On 3/5/25 07:57, Rob Dennett via MapServer-users wrote:
> Is it possible to cache tiles  to and serve them from S3 buckets?  We
> currently mount S3 folders in EC2 instances one of which is the
> directory we specify in MAP/IMAGEPATH, but it requires us to use s3fs-
> fuse and that means we have to use EC2 as opposed to Fargate, which we
> would prefer.  Can MapServer act as a client to S3?
>
> I have been looking at MapCache, and as far as I can tell, it's a
> separate server that works alongside a WMS server, although I am not
> sure that MapServer counts as a WMS server.  We use it to serve geoJSON,
> UTF grid, etc.  Is it what we want to use for this?
>
> Thanks,
> Rob
>
> _______________________________________________
> MapServer-users mailing list
> MapServer-users at lists.osgeo.org<mailto:MapServer-users at lists.osgeo.org>
> https://lists.osgeo.org/mailman/listinfo/mapserver-users

_______________________________________________
MapServer-users mailing list
MapServer-users at lists.osgeo.org<mailto:MapServer-users at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/mapserver-users
_______________________________________________
MapServer-users mailing list
MapServer-users at lists.osgeo.org<mailto:MapServer-users at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/mapserver-users


--
Richard W. Greenwood
www.greenwoodmap.com<http://www.greenwoodmap.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20250307/c23947e0/attachment-0001.htm>