[mapserver-users] Storing GeoTIFFs in Amazon S3 for use with Mapserver?

[Frank Warmerdam]

Robert,

I was actually using the vsicurl support from a prep-script similar to
gdaltindex, not from in mapserver directly, so I don't have a handy
example.  I'd also be concerned that accessing the data via /vsicurl/ would
have performance problems that would make it unwise for direct access from
MapServer.  One benefit of the S3 file system handlers is that they cache
the files locally so multiple tile renders in mapserver from the file would
be fast after the first fetch.  If you use /vsicurl/ then I don't think
that local caching would occur between mapserver render requests.

Best regards,
Frank

On Wed, Nov 5, 2014 at 12:07 PM, Robert Sanson <
Robert.Sanson at asurequality.com> wrote:

> Thanks Frank.
>
> Do you have an example of a mapfile layer section using GDAL with vsicurl
> to retrieve GeoTIFFs?
>
> In my case I use a tileindex to access a directory of GeoTIFFs but we want
> to move them into S3. Here is a layer from my mapfile:
>
> LAYER
>   NAME sthl-075m-rural-2005-11
>   GROUP 'highres'
>   TYPE RASTER
>   STATUS ON
>   TILEINDEX ../../data/nz/sthl-075m-rural-2005-11_ind
>   TILEITEM Location
>   MAXSCALE 100000
>   METADATA
>    'ows_title' 'sthl-075m-rural-2005-11'
>    'wms_opaque' '1'
>    'wms_group_title' 'highres'
>    'wms_resx' '0.75'
>    'wms_resy' '0.75'
>    'ows_extent' '1088800.0 4743600.0 1314400.0 5089200.0'
>   END
>  END
>
> Thanks,
>
> Robert
>
> >>> Frank Warmerdam <warmerdam at pobox.com> 5/11/2014 1:45 p.m. >>>
> Robert,
>
> I've done some variations on this.
>
> I'd encourage you to watch a video by Mark Korver on this topic from
> FOSS4G: http://vimeo.com/106833852
>
> I used his NAIP bucket in "requestor-pays" mode successfully.  One thing
> that I ended up doing when reading the metadata was to use the vsicurl
> driver to access the tiffs, so only the required portions were fetched.
> His example actually mounts the whole bucket as a file system in which case
> things are much easier.
>
> I will note that when using vsicurl to access GeoTIFFs in S3 I found I had
> to set the following configuration variables to get it working.
>
>     # Various /vsicurl hackery to get it working with signed aws urls well
>     gdal.SetConfigOption('CPL_VSIL_CURL_ALLOWED_EXTENSIONS', 'ter')
>     gdal.SetConfigOption('GDAL_DISABLE_READDIR_ON_OPEN', 'YES')
>     gdal.SetConfigOption('CPL_VSIL_CURL_USE_HEAD', 'NO')
>
> I do quite a bit of stuff here with MapServer from S3, but I usually fetch
> from S3 and store in /tmp for serving so by the time MapServer sees the
> file it is completely normal.
>
> Best regards,
> Frank
>
>
> On Tue, Nov 4, 2014 at 2:56 PM, Robert Sanson <
> Robert.Sanson at asurequality.com> wrote:
>
> > Does anyone have experience with using Amazon S3 to store GeoTIffs for
> use
> > as a back-end datastore accessible by Mapserver in order to create a WMS
> > service?
> >
> > Thanks,
> >
> > Robert Sanson
> >
> >
> > Robert Sanson Veterinary Epidemiologist
> > Batchelar Centre | Tennent Drive | Palmerston North
> > PO Box 585 | Palmerston North 4440 | New Zealand
> > p. +64 6 351 7990 | m. +64 21 448 472 | f. +64 6 351 7919
> > e. robert.sanson at asurequality.com
> >
> > Global experts in food safety and quality | www.asurequality.com
> >
> >
> >
> >
> > This email and any attachments are confidential and intended solely for
> > the addressee(s). If you are not the intended recipient, please notify us
> > immediately and then delete this email from your system.
> >
> > This message has been scanned for Malware and Viruses by Websense Hosted
> > Security.
> > www.websense.com
> > _______________________________________________
> > mapserver-users mailing list
> > mapserver-users at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapserver-users
> >
>
>
>
> --
>
> ---------------------------------------+--------------------------------------
> I set the clouds in motion - turn up   | Frank Warmerdam,
> warmerdam at pobox.com
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush    | Geospatial Software Developer
>
>


-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam,
warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Software Developer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20141105/a6cc5527/attachment.htm>