[MapServer-users] Redirecting to s3 via apache2 redirect/alias/proxy

[Timothy Kempisty - NOAA Federal]

Your original message described an attempt to use /vsis3/ , which is a GDAL
feature that lets it read datasets from S3.  Mapserver uses those GDAL
hooks to read datasets and rasters, but it does not use GDAL to read
mapfiles.  To my knowledge, Mapserver itself doesn't speak s3 natively.
Mapserver has no way of getting its mapfile from an s3 URL... or any URL,
for that matter.

The only way I can think of to get mapfiles from s3 is to use something
like s3fs-fuse.  That would mount the s3 bucket as a filesystem on the VM.
Then mapserver would read it like a normal local file.

But you said you didn't want to do it that way.  I think you're chasing
something that isn't possible.

Even if it were possible, I wouldn't necessarily advise it.  It seems like
it would be a security nightmare.  Also, mapfiles aren't usually that big.
When you spin up a new VM, why not just download a copy of the mapfiles
locally?  Huge rasters can live on S3, and that's fine.  The DATA line can
use /vsis3/ to get the raster. But mapfiles?  I'd question how much would
be gained at the cost of latency and security risks.

-Tim

On Mon, Oct 24, 2022 at 6:33 AM Marcin Niemyjski via MapServer-users <
mapserver-users at lists.osgeo.org> wrote:

> Jukka, Lars
>
> thanks for quick response.
>
> Sorry for making a mistake and replying to you directly.
>
> Lars
> Yes, my apache has all required modules active and the bucket I'm using is
> a public bucket. To be honest I'm not fully understand the mechanism of
> Apache, I'll go back and try using structure from docs. I do not understand
> the reasoning behind using Environment variable in this case. From my
> experience environmental variable for example, can be used to point file in
> which erros would be written. Could you please tell me more precisely how
> to use it in my case?
>
> Using env. variable for me wouldn't work because I'm using more than one
> mapfile (I'm displaying everyday S-2 scenes for Poland and Ukraine). The
> goal is to use mapserver hosted from docker, to which would be connected
> (apache redirect?) s3 bucket with mapfiles, so serving wms would be fully
> independent process without need to mount bucket.
>
> I think I'll try this way:
>
> https://stackoverflow.com/questions/9929487/using-apache-mod-alias-in-vhost-to-link-assets-from-external-server
>
> Thank you both,
> Marcin
> ------------------------------
> *From:* MapServer-users <mapserver-users-bounces at lists.osgeo.org> on
> behalf of Lars Fricke <lars.fricke at skendata.de>
> *Sent:* Monday, October 24, 2022 11:37 AM
> *To:* mapserver-users at lists.osgeo.org <mapserver-users at lists.osgeo.org>
> *Subject:* Re: [MapServer-users] Redirecting to s3 via apache2
> redirect/alias/proxy
>
> Three things come to my mind:
> 1) Did you check if your Apache or NGINX respectively has the module
> installed and active?
> 2) You do not have credentials in your second S3 URL example. This may be
> an issue.
> 3) The structure in the doc's is different:
> ```
> RewriteEngine on
> RewriteRule wmsmap?(.*)
> /cgi-bin/mapserv?map=/home/www/mapserverstuff/mymapfile.map&$1
> ```
> I don't use this method myself but you may want to try using exactly the
> same replacement structure?
> As said before, you could als use an Environment variable.
> Cheers Lars
>
> Am 24.10.22 um 11:12 schrieb Marcin Niemyjski via MapServer-users:
>
> Hello,
>
> coming back to you with one more question about s3, namely:
>
> As I wrote before - I want to keep my mapfiles on buckets but I also want
> to access them without need to mount buckets to VMs, so I've came up with
> this:
>
> https://httpd.apache.org/docs/2.4/rewrite/flags.html#flag_p
> https://mapserver.org/ogc/wms_server.html#changing-the-online-resource-url
>
> generaly I want to do this:
>
> ProxyPass        /martest/ https://s3.amazon.com/Bucket/Key
> ProxyPassReverse /martest/ https://s3.amazon.com/Bucket/Key
>
> or
>
> RewriteEngine on
> RewriteRule /martest/ https://s3.amazon.com/Bucket/
>
> so, while using WMS online resource url https://www.lpis.pl/cgi-bin/mapserv?map=/martest/jrc.map i would connect to mapfile on s3.
>
> But it doesn't seem to be working. Is this even possible to connect mapserver and s3 this way?
>
> As always thank you for your help and time,
> Marcin
>
>
>
> _______________________________________________
> MapServer-users mailing listMapServer-users at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/mapserver-users
>
>
> _______________________________________________
> MapServer-users mailing list
> MapServer-users at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20221027/454a7407/attachment.htm>