<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>There are multiple advantages to using s3 over ebs. First is availability. S3 is highly available (and replicated). EBS is limited to the single volume and is much more fragile. EBS is also more expensive for storage than S3. And S3 is available to multiple instances, EBS is not, it can only be mounted to one EC2 instance.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Mike<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:10.5pt;color:black'>--<o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:10.5pt;color:black'>Michael Smith<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.5pt;color:black'>Remote Sensing/GIS Center<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.5pt;color:black'>US Army Corps of Engineers</span><o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>gdal-dev <gdal-dev-bounces@lists.osgeo.org> on behalf of Richard Greenwood <richard.greenwood@gmail.com><br><b>Date: </b>Monday, March 4, 2019 at 9:21 AM<br><b>To: </b>Jeremy Palmer <palmerjnz@gmail.com><br><b>Cc: </b>gdal dev <gdal-dev@lists.osgeo.org><br><b>Subject: </b>Re: [gdal-dev] GDAL VRT with Cloud Optimized GeoTIFFs and AWS<o:p></o:p></span></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>What are the advantages of storing the imagery in S3 as opposed to EBS? I'm using the throughput optimized magnetic EBS which I is a little less expensive than S3. I did some testing a couple years ago and didn't see enough performance gain to justify SSD. Have you tested S3 against any of the EBS options?<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Rich<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On Sat, Mar 2, 2019 at 7:58 AM Jeremy Palmer <<a href="mailto:palmerjnz@gmail.com">palmerjnz@gmail.com</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p class=MsoNormal>I've now found this useful Mapserver wiki page <a href="https://github.com/mapserver/mapserver/wiki/Render-images-straight-out-of-S3-with-the-vsicurl-driver" target="_blank">https://github.com/mapserver/mapserver/wiki/Render-images-straight-out-of-S3-with-the-vsicurl-driver</a>. It seems to imply that it's better to merge a collection of dataset tiles into a single big Geotiff rather than create a VRT due to the repeated HTTP calls. I'm still interested in people's experiences using imagery direct from S3 and it's performance. <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Note some of my largest datasets have a total size of 1.5TBs uncompressed Geotiff with about 8000 tiles @ 175mb per tile. I'll likely compress the uncompressed Geotiffs and use JPEG compression, which I estimate can bring the size down to about 200GB total.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Cheers<o:p></o:p></p></div><div><p class=MsoNormal>Jeremy<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On Sat, Mar 2, 2019 at 9:15 AM Jeremy Palmer <<a href="mailto:palmerjnz@gmail.com" target="_blank">palmerjnz@gmail.com</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p class=MsoNormal>Hi All,<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Does anyone have any tips or experience in trying to serve RGB large imagery multi file datasets hosted on S3 to application servers/containers for bulk tile rendering? Is this possible using VRTs and is the performance manageable when compared to other mounted storage options? <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Thanks,<o:p></o:p></p></div><div><p class=MsoNormal>Jeremy<o:p></o:p></p></div></div></div></blockquote></div><p class=MsoNormal>_______________________________________________<br>gdal-dev mailing list<br><a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br><a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><o:p></o:p></p></blockquote></div><p class=MsoNormal><br clear=all><o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal>-- <o:p></o:p></p><div><div><p class=MsoNormal>Richard W. Greenwood, PLS<br><a href="http://www.greenwoodmap.com" target="_blank">www.greenwoodmap.com</a><o:p></o:p></p></div></div><p class=MsoNormal>_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev<o:p></o:p></p></div></body></html>