<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Let’s discuss this in a week when we have our check-in call. I’m not the expert here, but it sounds like adding tiling is simple enough that it’s worth considering.
<div class=""><br class="">
</div>
<div class="">However, I’m still +0 on pyramids at this point. We’re open to providing enhancements to the raw data in the future, but would like to get more feedback from users before we make any decisions.</div>
<div class=""><br class="">
</div>
<div class="">Until Wednesday…</div>
<div class=""><br class="">
<div class="">
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Feb 10, 2015, at 4:31 PM, Peter Becker <<a href="mailto:pbecker@esri.com" class="">pbecker@esri.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Jed<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Providing overviews for the imagery will enable user to use the imagery in a lot more applications without the need and time to generate pyramids themselves.
Creating pyramids does require the whole image to be read. Hence if we do not provide then users often be forced to download the complete image vs just using it from S3. The lack of overviews will restrict user from access the imagery efficiently and so I
recommend that we do added as external .OVR files with deflate and factor 3. Doing this at the same time as the base tif files are created will be minimal additional processing in comparison to having to create them later. Having these as external .OVR files
will mean that users that do not need them, need not download them. Applications working with GDAL will be able to directly use them. This will significantly increase usability of the data. If users want to generate pyramids their own way they can always still
download and run a similar process.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">Frank could you look to add this to the existing scripts and possibly apply to some samples.<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> gdaladdo.exe -r average -ro --config COMPRESS_OVERVIEW DEFALTE --config USE_RRD NO --config PREDICTOR_OVERVIEW 2 --config TILED YES<span class="Apple-converted-space"> </span><i class="">scene</i>.tif
3 9 27 81<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class="">_Peter<o:p class=""></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" class=""> </span></div>
<div class="">
<div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0in 0in;" class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<b class=""><span style="font-size: 10pt; font-family: Tahoma, sans-serif;" class="">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif;" class=""><span class="Apple-converted-space"> </span>Sundwall, Jed [<a href="mailto:jsundwal@amazon.com" class="">mailto:jsundwal@amazon.com</a>]<span class="Apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Friday, February 06, 2015 3:30 PM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>Peter Becker<br class="">
<b class="">Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:landsat-pds@lists.osgeo.org" class="">landsat-pds@lists.osgeo.org</a><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [Landsat-pds] Including Overviews<o:p class=""></o:p></span></div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
I’ve done the math, and we can certainly afford to take on an additional 13% of data volume over the next two years, but I’d like to take some more time to do research on demand for this. The approach we’ve taken so far has been to touch the data as little
as possible, and beyond the jpg thumbnails we produce, we aren’t creating any derivative products from the raw scenes.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
In the meantime, any customer could produce these overlays themselves if they wanted to, but if we hear from multiple customers that they’re doing redundant work to do this, we'll consider adding overviews to this data set. <o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Open to others’ thoughts on this.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
Jed.<o:p class=""></o:p></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
</div>
<div class="">
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
On Feb 5, 2015, at 4:41 PM, Peter Becker <<a href="mailto:pbecker@esri.com" style="color: purple; text-decoration: underline;" class="">pbecker@esri.com</a>> wrote:<o:p class=""></o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<o:p class=""> </o:p></div>
<div class="">
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Currently the scenes do not include any overviews. This forces applications to read the imagery at full resolution even if accessing at a small scale.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">It’s common to include overviews/pyramids/reduced resolution datasets with such imagery.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">I would like to recommend that such overviews are included with the imagery<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Typically this would increase the file size by approx 33%, which is quite considerable.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">The size of the overviews can be reduced by skipping a level (eg use level 4 8 16 32) vs 2 4 8 16 32, alternatively using a factor of 3 ie 3 9 27 81<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">I tested this out on some images using Deflate with predictor 2<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">2 4 8 16 32 adds about 35%<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">4 8 16 32 adds about 9%<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">3 9 27 81 adds about 13%<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">My recommendation is to go with the 3 9 27 81 factors. This keeps the factor constant between all levels.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Average sampling would be used, except for the BQA band for which Nearest (else Majority) should be used.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Implementation would only require adding GDALADDO to the existing script<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">We would also need to determine if we use internal or external overviews.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Internal overviews would increase the size of the files, but not change the directory structure.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Using external overviews (.tif.ovr) would add additional files to the directory, is more specific to GDAL.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">As I expect many users to be using GDAL to access these files, I would recommend that we use the external overviews. I don’t see the additional files as a burden.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">One other potential concern (and I’m hoping Frank can answer this) is the tile size of the overviews. GDAL appear to always use a tile size (BLOCKSIZE) of 128. It would be more optimum
if the file size could be set to 512.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">_Peter<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></span></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class="">
<span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">_______________________________________________<br class="">
Landsat-pds mailing list<br class="">
</span><a href="mailto:Landsat-pds@lists.osgeo.org" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;" class="">Landsat-pds@lists.osgeo.org</span></a><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><br class="">
</span><a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif; color: purple;" class="">http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds</span></a></div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</body>
</html>