[mapserver-users] Refreshing AWS credentials for the /vsis3/ driver

[Even Rouault]

Le jeudi 27 octobre 2016 18:47:38, Peter Schmitt a écrit :
> On Thu, Oct 27, 2016 at 10:42 AM, Even Rouault <even.rouault at spatialys.com>
> 
> wrote:
> > Le jeudi 27 octobre 2016 18:33:01, Peter Schmitt a écrit :
> > > On Wed, Oct 26, 2016 at 1:39 PM, Even Rouault <
> > 
> > even.rouault at spatialys.com>
> > 
> > > wrote:
> > > > I've written recently a best practice for generating cloud optimized
> > > > geotiffs:
> > > > https://trac.osgeo.org/gdal/wiki/CloudOptimizedGeoTIFF
> > > > (it mentions deflate compression, but jpeg-compressed can work of
> > 
> > course)
> > 
> > > Thanks again, Even! I did some experimentation with a 19584x19584 file
> > 
> > with
> > 
> > > 6 overview levels.  When the image is deflate-compressed:
> > > > gdal_translate in.tif deflate.tif -co TILED=YES -co COMPRESS=DEFLATE
> > 
> > -co
> > 
> > > COPY_SRC_OVERVIEWS=YES
> > > 
> > > then this gdalinfo command does two range requests:
> > > > env AWS_ACCESS_KEY_ID=foo
> > > 
> > > AWS_SECRET_ACCESS_KEY=bar CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif
> > > CPL_CURL_VERBOSE=YES
> > > VSI_CACHE=TRUE gdalinfo /vsis3/pschmitt-test/deflate.tif 2>&1 | grep
> > > "Content-Range:"
> > > 
> > > >  < Content-Range: bytes 0-49151/1038305419
> > > >  < Content-Range: bytes 49152-147455/1038305419
> > > 
> > > When I jpeg-compress
> > > 
> > > > gdal_translate deflate.tif jpeg.tif -co TILED=YES -co COMPRESS=JPEG
> > > > -co
> > > 
> > > PHOTOMETRIC=YCBCR -co COPY_SRC_OVERVIEWS=YES
> > > 
> > > then gdalinfo on the jpeg-compressed image does 13 range requests:
> > > > env AWS_ACCESS_KEY_ID=foo
> > > 
> > > AWS_SECRET_ACCESS_KEY=bar CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif
> > > CPL_CURL_VERBOSE=YES
> > > VSI_CACHE=TRUE gdalinfo /vsis3/pschmitt-test/jpeg.tif 2>&1 | grep
> > > "Content-Range:"
> > > 
> > > > < Content-Range: bytes 0-49151/83731710
> > > > < Content-Range: bytes 83656704-83705855/83731710
> > > > < Content-Range: bytes 83722240-83731709/83731710
> > > > < Content-Range: bytes 83705856-83722239/83731710
> > > > < Content-Range: bytes 25231360-25280511/83731710
> > > > < Content-Range: bytes 25165824-25214975/83731710
> > > > < Content-Range: bytes 25214976-25231359/83731710
> > > > < Content-Range: bytes 7602176-7651327/83731710
> > > > < Content-Range: bytes 2228224-2310143/83731710
> > > > < Content-Range: bytes 655360-704511/83731710
> > > > < Content-Range: bytes 229376-278527/83731710
> > > > < Content-Range: bytes 131072-180223/83731710
> > > > < Content-Range: bytes 98304-131071/83731710
> > 
> > Could you post the outpout of "tiffdump jpeg.tif" ? There are some hacks
> > in libtiff regarding JPEG compression and in particular JPEG tables that
> > might perhaps cause spurious IFD rewriting.
> 
> Here's the ouput of "tiffdump jpeg.tif": http://pastebin.com/raw/pX6XMYVT

That's what I suspected. The IFD are not placed at the beginning of the file, 
but rather rewritten after each "data section". 
Which libtiff version is used in your GDAL build: internal libtiff or system 
libtiff (if so, which version)? But I wouldn't surprise that the issue exists 
with any libtiff version because of what I mentionned with jpeg compression. 
That might probably be fixable (likely with some headaches)

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com