<div dir="ltr">Hey Rob,<div><br></div><div>Couple things. It's best to keep the discussion on the list so that all subscribers can participate. And this discussion has drifted pretty far from the original subject about using S3 for tile caching.</div><div><br></div><div>MapServer and MapCache can certainly be run on the same server and MapCache can certainly serve WMS, XYZ Tiles, UTF Grids, etc. but the HTML requests have to identify which service they want. I think your server must be set to default to WMS in the Apache config so you need to add additional options. These are often called "end points" or "routes". Apache has a lot of different ways that you can do this, which is great, but can also be challenging when you're getting started. So check out the MapCache documentation and then come back to the list with specific questions that we can help with.</div><div><br></div><div>Rich</div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Mon, Mar 10, 2025 at 7:29 AM Rob Dennett <<a href="mailto:Rob.Dennett@twdb.texas.gov">Rob.Dennett@twdb.texas.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg8010766712922884830">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
We use an Apache conf file very similar to what's found in the CampToCamp repo: <a href="https://github.com/camptocamp/docker-mapserver/blob/master/runtime/etc/apache2/conf-enabled/mapserver.conf" id="m_8010766712922884830LPlnk604805" target="_blank">
https://github.com/camptocamp/docker-mapserver/blob/master/runtime/etc/apache2/conf-enabled/mapserver.conf</a><br>
<br>
It uses ScriptAliasMatch to call a wrapper script which invokes mapserv in turn, followed by a LocationMatch which sets up mod_fcgid. We changed ours to ignore some temporary directories, including one where we write cached tiles using IMAGEPATH.<br>
<br>
Our app describes links for WMS, XYZ Tile Service, UTF Grid Service, and MVT service. Are the last three not something you can use with MapCache, or am I misunderstanding what a "source" is?</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Rob<br>
<br>
</div>
<div id="m_8010766712922884830appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_8010766712922884830divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Richard Greenwood <<a href="mailto:richard.greenwood@gmail.com" target="_blank">richard.greenwood@gmail.com</a>><br>
<b>Sent:</b> Friday, March 7, 2025 9:30 AM<br>
<b>To:</b> Rob Dennett <<a href="mailto:Rob.Dennett@twdb.texas.gov" target="_blank">Rob.Dennett@twdb.texas.gov</a>><br>
<b>Subject:</b> Re: [MapServer-users] Can MapServer cache directly to AWS S3?</font>
<div> </div>
</div>
<div>
<div style="background-color:rgb(255,255,255);border-style:solid;border-color:rgb(244,58,19);border-width:2pt;font-size:10pt;line-height:12pt;font-family:Calibri;color:red">
<p align="center" style="text-align:center;background:rgb(255,255,255)">
<span style="font-size:12pt;font-weight:bold;color:red">External: Beware of links/attachments.
</span></p>
</div>
<br>
<br>
<div>
<div dir="ltr">
<div dir="ltr">On Fri, Mar 7, 2025 at 8:03 AM Rob Dennett <<a href="mailto:Rob.Dennett@twdb.texas.gov" target="_blank">Rob.Dennett@twdb.texas.gov</a>> wrote:</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Currently, an example of a URL used by one of our front-end apps is <br>
<br>
</div>
<div style="line-height:19px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<a href="https://myserver.com?map=example.map&mode=tile&tilemode=gmap&tile=%7Bx%7D+%7By%7D+%7Bz%7D&layers=all&map.imagetype=png" target="_blank">https://myserver.com?map=example.map&mode=tile&tilemode=gmap&tile={x}+{y}+{z}&layers=all&map.imagetype=png</a><br>
<br>
For the XYZ Tile service.</div>
<div style="line-height:19px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="line-height:19px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
For WMS, it looks like</div>
<div style="line-height:19px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<a href="https://mapserver.tnris.org/?map=/tnris_mapfiles/area_type.map" id="m_8010766712922884830x_m_8238117067157559096LPlnk528012" target="_blank">https://</a><a href="http://myserver.com" target="_blank">myserver.com</a><a href="https://mapserver.tnris.org/?map=/tnris_mapfiles/area_type.map" id="m_8010766712922884830x_m_8238117067157559096LPlnk528012" target="_blank">/?map=</a>example<a href="https://mapserver.tnris.org/?map=/tnris_mapfiles/area_type.map" id="m_8010766712922884830x_m_8238117067157559096LPlnk528012" target="_blank">.map</a><br>
<br>
I am not sure how that would map back to a URL like /cgi-bin/mapserv.fcgi<br>
<br>
Would it be just <br>
<url><a href="https://myserver.com/" target="_blank">https://myserver.com/</a></url>?<br>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Can't say for sure. Your web server obviously has some configuration that is directing requests to "/" (the root of the server) to the mapserv binary. In other words, your server has some additional configuration setup that is routing requests in a non-standard
way. By "non-standard" I don't mean it's wrong, just that without seeing your web server configuration we can only guess at the routing.</div>
<div><br>
</div>
<div>Rich</div>
<div><br>
</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="line-height:19px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
</div>
<div id="m_8010766712922884830x_m_8238117067157559096appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_8010766712922884830x_m_8238117067157559096divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Richard Greenwood <<a href="mailto:richard.greenwood@gmail.com" target="_blank">richard.greenwood@gmail.com</a>><br>
<b>Sent:</b> Friday, March 7, 2025 8:44 AM<br>
<b>To:</b> Rob Dennett <<a href="mailto:Rob.Dennett@twdb.texas.gov" target="_blank">Rob.Dennett@twdb.texas.gov</a>><br>
<b>Cc:</b> mapserver <<a href="mailto:mapserver-users@lists.osgeo.org" target="_blank">mapserver-users@lists.osgeo.org</a>><br>
<b>Subject:</b> Re: [MapServer-users] Can MapServer cache directly to AWS S3?</font>
<div> </div>
</div>
<div>
<div style="background-color:rgb(255,255,255);border-style:solid;border-color:rgb(244,58,19);border-width:2pt;font-size:10pt;line-height:12pt;font-family:Calibri;color:red">
<p align="center" style="text-align:center;background:rgb(255,255,255)"><span style="font-size:12pt;font-weight:bold;color:red">External: Beware of links/attachments.
</span></p>
</div>
<br>
<br>
<div>
<div dir="ltr">
<div dir="ltr">On Fri, Mar 7, 2025 at 7:29 AM Rob Dennett <<a href="mailto:Rob.Dennett@twdb.texas.gov" target="_blank">Rob.Dennett@twdb.texas.gov</a>> wrote:</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
So, can you have both the MapCache module and the MapServer CGI script running on the same Apache server?
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Sure. There's no conflict. </div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Is there an example showing how to configure this? </div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Look at the MapCache documentation. MapCache is configured with an xml file. If MapServer is on the same server then you'd have a statement in the the "source" section like:</div>
<div> <source name="something" type="wms"></div>
<div> . . .</div>
<div> <url><a href="http://localhost/cgi-bin/mapserv.fcgi" target="_blank">http://localhost/cgi-bin/mapserv.fcgi</a></url></div>
<div> . . .</div>
<div> </source></div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
We have a bunch of front ends which use our MapServer (it's customer-facing, too) and I am not clear if all those URLs would have to change.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Your app makes requests to <a href="https://youserver/mapcache" target="_blank">
https://youserver/mapcache</a> for tiles or <a href="https://yourserver/cgi-bin/mapserv.fcgi" target="_blank">
https://yourserver/cgi-bin/mapserv.fcgi</a> for non-tiles requests. MapCache is reading and writing to and from S3, your app is not directly access S3.</div>
<div><br>
</div>
<div>Rich</div>
<div><br>
</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:12pt">Thanks,</span></div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Rob</div>
<div id="m_8010766712922884830x_m_8238117067157559096x_m_4154713686534273295appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_8010766712922884830x_m_8238117067157559096x_m_4154713686534273295divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Richard Greenwood <<a href="mailto:richard.greenwood@gmail.com" target="_blank">richard.greenwood@gmail.com</a>><br>
<b>Sent:</b> Friday, March 7, 2025 7:44 AM<br>
<b>To:</b> Rob Dennett <<a href="mailto:Rob.Dennett@twdb.texas.gov" target="_blank">Rob.Dennett@twdb.texas.gov</a>><br>
<b>Cc:</b> mapserver <<a href="mailto:mapserver-users@lists.osgeo.org" target="_blank">mapserver-users@lists.osgeo.org</a>><br>
<b>Subject:</b> Re: [MapServer-users] Can MapServer cache directly to AWS S3?</font>
<div> </div>
</div>
<div>
<div style="background-color:rgb(255,255,255);border-style:solid;border-color:rgb(244,58,19);border-width:2pt;font-size:10pt;line-height:12pt;font-family:Calibri;color:red">
<p align="center" style="text-align:center;background:rgb(255,255,255)"><span style="font-size:12pt;font-weight:bold;color:red">External: Beware of links/attachments.
</span></p>
</div>
<br>
<br>
<div>
<div dir="ltr">
<div dir="auto">
<div dir="auto"><br>
</div>
<div dir="auto">
<div dir="ltr">On Thu, Mar 6, 2025, 3:27 PM Rob Dennett via MapServer-users <<a href="mailto:mapserver-users@lists.osgeo.org" target="_blank">mapserver-users@lists.osgeo.org</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Does MapCache have to be built from source? </div>
</div>
</blockquote>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Probably not. If you're on Linux it will be in your distribution's repo. Not sure about Windows or Mac. </div>
<div dir="auto"><br>
</div>
<div dir="auto">
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Also, does it have to be it's own server?</div>
</div>
</blockquote>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">MapCache is a module that the Apache web server runs. </div>
<div dir="auto"><br>
</div>
<div dir="auto">
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
It's not clear to me how it connects with MapServer. Is it the case that MapServer renders tiles as necessary based on requests and then MapCache serves those files if requested again?<br>
</div>
</div>
</blockquote>
</div>
<div dir="auto"><br>
</div>
No. Apache passes tile requests to MapCache. If MapCache has the tile it gets returned. If MapCache doesn't have the tile it makes a WMS request to MapServer, saves the tile in its cache and returns the tile to the web server. MapCache can save the tile on
the local file system, S3 or elsewhere. You configure where it saves the cached tiles.<br>
<div dir="auto"><br>
</div>
<div dir="auto">
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
As mentioned, MapServer is currently storing tiles in a directory mapped to S3 which is specified by IMAGEPATH.
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Using MapCache you don't need IMAGEPATH.</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
These tiles are not pre-seeded, so I can't specify a particular raster or vector file to serve from S3. I don't think I understand how caching is supposed to work with S3 in the way we're doing it. Can you simply specify a folder in S3 where cached data lives
and have MapServer render anything that doesn't exist yet?</div>
</div>
</blockquote>
<div><br>
</div>
<div>If you are trying to serve tiles directly from S3 by making tile requests directly to the S3 bucket that is a whole different thing and not supported by MapCache.</div>
<div><br>
</div>
<div>HTH,</div>
<div>Rich</div>
<div><br>
</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-size:12pt">Thanks,</span></div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Rob</div>
<div id="m_8010766712922884830x_m_8238117067157559096x_m_4154713686534273295x_m_4892799894223543394m_-4034951559898070911appendonsend">
</div>
<hr style="display:inline-block;width:98%">
<div id="m_8010766712922884830x_m_8238117067157559096x_m_4154713686534273295x_m_4892799894223543394m_-4034951559898070911divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> MapServer-users <<a href="mailto:mapserver-users-bounces@lists.osgeo.org" rel="noreferrer" target="_blank">mapserver-users-bounces@lists.osgeo.org</a>> on behalf of Scott via
MapServer-users <<a href="mailto:mapserver-users@lists.osgeo.org" rel="noreferrer" target="_blank">mapserver-users@lists.osgeo.org</a>><br>
<b>Sent:</b> Wednesday, March 5, 2025 11:50 AM<br>
<b>To:</b> <a href="mailto:mapserver-users@lists.osgeo.org" rel="noreferrer" target="_blank">
mapserver-users@lists.osgeo.org</a> <<a href="mailto:mapserver-users@lists.osgeo.org" rel="noreferrer" target="_blank">mapserver-users@lists.osgeo.org</a>><br>
<b>Subject:</b> Re: [MapServer-users] Can MapServer cache directly to AWS S3?</font>
<div> </div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div>External: Beware of links/attachments.<br>
<br>
<br>
<br>
You can store any raster/vector file format supported by GDAL on S3 and<br>
use it via MapServer. Note the /vsis3 virtual connection:<br>
<br>
Raster:<br>
DATA "/vsis3/myS3Bucket/raster.tif"<br>
<br>
Vector:<br>
CONNECTION "/vsis3/myS3Bucket/vector.fgb"<br>
<br>
MapServer is definitely a WMS server. Any OGC compliant WMS url can be<br>
parsed and processed. Using MapCache along MapServer is highly desirable<br>
as all subsequent WMS requests are cached. Further, you can seed any<br>
cache directly with MapCache.<br>
<br>
Scott<br>
<br>
<br>
On 3/5/25 07:57, Rob Dennett via MapServer-users wrote:<br>
> Is it possible to cache tiles to and serve them from S3 buckets? We<br>
> currently mount S3 folders in EC2 instances one of which is the<br>
> directory we specify in MAP/IMAGEPATH, but it requires us to use s3fs-<br>
> fuse and that means we have to use EC2 as opposed to Fargate, which we<br>
> would prefer. Can MapServer act as a client to S3?<br>
><br>
> I have been looking at MapCache, and as far as I can tell, it's a<br>
> separate server that works alongside a WMS server, although I am not<br>
> sure that MapServer counts as a WMS server. We use it to serve geoJSON,<br>
> UTF grid, etc. Is it what we want to use for this?<br>
><br>
> Thanks,<br>
> Rob<br>
><br>
> _______________________________________________<br>
> MapServer-users mailing list<br>
> <a href="mailto:MapServer-users@lists.osgeo.org" rel="noreferrer" target="_blank">
MapServer-users@lists.osgeo.org</a><br>
> <a href="https://lists.osgeo.org/mailman/listinfo/mapserver-users" rel="noreferrer" target="_blank">
https://lists.osgeo.org/mailman/listinfo/mapserver-users</a><br>
<br>
_______________________________________________<br>
MapServer-users mailing list<br>
<a href="mailto:MapServer-users@lists.osgeo.org" rel="noreferrer" target="_blank">MapServer-users@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/mapserver-users" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/mapserver-users</a><br>
</div>
</span></font></div>
</div>
_______________________________________________<br>
MapServer-users mailing list<br>
<a href="mailto:MapServer-users@lists.osgeo.org" rel="noreferrer" target="_blank">MapServer-users@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/mapserver-users" rel="noreferrer noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/mapserver-users</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div><br clear="all">
</div>
<div><br>
</div>
<span>-- </span><br>
<div dir="ltr">
<div dir="ltr">Richard W. Greenwood<br>
<a href="http://www.greenwoodmap.com" target="_blank">www.greenwoodmap.com</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div><br clear="all">
</div>
<div><br>
</div>
<span>-- </span><br>
<div dir="ltr">
<div dir="ltr">Richard W. Greenwood<br>
<a href="http://www.greenwoodmap.com" target="_blank">www.greenwoodmap.com</a></div>
</div>
</div>
</div>
</div>
</div>
</div></blockquote></div><div><br clear="all"></div><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr">Richard W. Greenwood<br><a href="http://www.greenwoodmap.com" target="_blank">www.greenwoodmap.com</a></div></div>