<div dir="ltr">Hi Jeremy,<div><br></div><div>We have come up with one technique to read/write directly to/from s3 using /vsis3/ and a simple file writer class a colleague wrote: <a href="https://gist.github.com/pedros007/55c6e33224596fb4d8e9e6b68b24ed9b">https://gist.github.com/pedros007/55c6e33224596fb4d8e9e6b68b24ed9b</a> In fact, I used this last week to add internal overviews to some of our images in S3. Here's a high-level overview of my Python implementation using gdal-2.2.0:</div><div><br></div><div># Open the image as a /vsimem/ file:<br></div><div>vsi_file = '/vsimem/image.tif'</div><div><div><font color="#24292e" face="SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace"><span style="font-size:12px;white-space:pre">ds = gdal.Translate('/vsis3/bucket/prefix.tif', vsi_file)</span></font></div><div><font color="#24292e" face="SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace"><span style="font-size:12px;white-space:pre"><br></span></font></div><div><font color="#24292e" face="SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace"><span style="font-size:12px;white-space:pre"># Add overviews</span></font></div><div><font color="#24292e" face="SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace"><span style="font-size:12px;white-space:pre">err = ds.BuildOverviews('AVERAGE', [2,4,8,16,32,64])</span></font></div><div><font color="#24292e" face="SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace"><span style="font-size:12px;white-space:pre">if err != 0:</span></font></div><div><font color="#24292e" face="SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace"><span style="font-size:12px;white-space:pre"> raise BuildOverviewsError('Failed to build overviews for %s' % </span></font><span style="color:rgb(36,41,46);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;white-space:pre">vsi_file</span><font color="#24292e" face="SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace"><span style="font-size:12px;white-space:pre">)</span></font></div><div><font color="#24292e" face="SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace"><span style="font-size:12px;white-space:pre"># Some overview levels wouldn't get written unless we flush & close the data set.</span></font></div><div><font color="#24292e" face="SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace"><span style="font-size:12px;white-space:pre">ds.FlushCache()</span></font></div></div><div>ds = None</div><div># even though the dataset is gone, the vsi_file still exists in memory.</div><div><br></div><div># Write the data back to s3 using a class my colleague wrote.</div><div><span style="color:rgb(36,41,46);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;white-space:pre">try:</span></div><div><font color="#24292e" face="SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace"><span style="font-size:12px;white-space:pre"> vsimem_file = gdalutil.SimpleVSIMEMFile(</span></font><span style="color:rgb(36,41,46);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;white-space:pre">vsi_file</span><font color="#24292e" face="SFMono-Regular, Consolas, Liberation Mono, Menlo, Courier, monospace"><span style="font-size:12px;white-space:pre">)</span></font></div><div> s3 = boto3.resource('s3')<br></div><div><div><div> obj = s3.Object(bucket_name='bucket', key='prefix.tif')</div></div></div><div> # the Object#put API needs a file-like object: <a href="http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Object.put">http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Object.put</a><br> # the SimpleVSIMemFile implements a file-like object that uses the VSI API to read a /vsimem/ file.</div><div> obj.put(Body=vsimem_file)<br></div><div>finally:</div><div> vsimem_file = None</div><div><br></div><div># Remove the /vsimem/ file. (you probably want this in a try/finally block to ensure it gets deleted)</div><div><div>gdal.Unlink(<span style="color:rgb(36,41,46);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;white-space:pre">vsi_file</span>)</div><div><br></div></div><div>The above process will be memory constrained. Make sure your instance is appropriately sized! You probably want to run this on an EC2 instance in the same region as the data sitting in s3.</div><div><br></div><div><br></div><div>Even - I found methods to check for VSI errors:</div><div><a href="https://gist.github.com/pedros007/55c6e33224596fb4d8e9e6b68b24ed9b#file-simplevsimemfile-py-L73-L74">https://gist.github.com/pedros007/55c6e33224596fb4d8e9e6b68b24ed9b#file-simplevsimemfile-py-L73-L74</a></div><div><br></div><div>Are these intended for public consumption? When reading from a /vsimem/foo.shp with gdal-2.2.0, gdal.VSIGetLastErrorMsg() reported an error "No such file or directory". The VSI reader actually works regardless if it's a /vsimem/ file or local path. A colleague used a local path and the error reported was a little more verbose: It printed something like "/mnt/data/foo.sbn: No such file or directory". I did not see that error gdal-2.1.3. We don't need the sbn file (we're building a qix file instead). I was surprised that the VSI system flagged this as an error. Seems like it should be more of a warning. My motivation for using adding error checking: sometimes the command-line gdal_translate with /vsis3/ paths would yield IFD read errors (I don't have the exact error message handy). Repeating the command would result in successful translates. I always assumed it was some transient packet loss/network error.</div><div><br></div><div>Cheers,</div><div>Pete</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 18, 2017 at 3:14 AM, Jeremy Palmer <span dir="ltr"><<a href="mailto:JPalmer@linz.govt.nz" target="_blank">JPalmer@linz.govt.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
Hi Even,
<div><br>
<div><span class="">
<blockquote type="cite">
<div>On 18/05/2017, at 9:12 PM, Even Rouault <<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>> wrote:</div>
<div>
<div style="white-space:pre-wrap;font-family:'Sans Serif';font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;margin:0px">
> </div>
<div style="white-space:pre-wrap;font-family:'Sans Serif';font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;margin:0px">
> Is is possible to directly write external overview to a S3 bucket? With GDAL</div>
<div style="white-space:pre-wrap;font-family:'Sans Serif';font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;margin:0px">
> 2.1.2 I get an error reporting that seek is not supported when writing to</div>
<div style="white-space:pre-wrap;font-family:'Sans Serif';font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;margin:0px">
> vsis3:</div>
<p style="white-space:pre-wrap;font-family:'Sans Serif';font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;margin:0px">
</p>
<div style="white-space:pre-wrap;font-family:'Sans Serif';font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;margin:0px">
No, /vsis3/ only supports sequential writing in files (the original use case was to generate and upload a huge CSV file on the fly). I don't have all the details in mind but random writing might not be possible given the S3 API constraints, at least with the
multipart upload API which is used currently.</div>
</div>
</blockquote>
<div><br>
</div>
</span><div>OK thanks for clarifying the situation.</div><span class="">
<br>
<blockquote type="cite">
<div>
<p style="white-space:pre-wrap;font-family:'Sans Serif';font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;margin:0px">
</p>
<div style="white-space:pre-wrap;font-family:'Sans Serif';font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;margin:0px">
And another constraint of the current implementation is that a /vsis3/ file is either read-only or write-only, but not a mix of both, which would be needed for gdaladdo internal overviews. Perhaps external overview would work, but I'm not completely sure as
creating a TIFF file might require seeking.</div>
<p style="white-space:pre-wrap;font-family:'Sans Serif';font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;margin:0px">
</p>
<div style="white-space:pre-wrap;font-family:'Sans Serif';font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;margin:0px">
Perhaps a fully fledged read-write-update file system would be possible, but that wasn't in my initial design constraints.</div>
</div>
</blockquote>
<div><br>
</div>
</span><div>For now we will work around the issue.</div>
<div><br>
</div>
<div>Thank for your help.</div>
<div><br>
</div>
<div>Cheers,</div>
<div>Jeremy</div>
</div>
<br>
</div>
<br>
<hr><span class="">
<font face="Verdana" color="Black" size="2">This message contains information, which may be in confidence and may be subject to legal privilege. If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message. If
you have received this message in error, please notify us immediately (Phone 0800 665 463 or <a href="mailto:info@linz.govt.nz" target="_blank">info@linz.govt.nz</a>) and destroy the original message. LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from
LINZ. Thank You.<br>
</font>
</span></div>
<br>______________________________<wbr>_________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/gdal-dev</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Pete</div>
</div>