<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
FWIW, the “Frank’s words” on compression and tiling that Mark references are here: <a href="https://github.com/landsat-pds/landsat_ingestor/blob/master/Compression.md" class="">https://github.com/landsat-pds/landsat_ingestor/blob/master/Compression.md</a>
<div class=""><br class="">
</div>
<div class="">
<div apple-content-edited="true" class=""><br class="">
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Feb 6, 2015, at 3:48 PM, Korver, Mark <<a href="mailto:mkorver@amazon.com" class="">mkorver@amazon.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">Normally when I am working with large TIFF files I will check to see their structure and if not tiled I will internally tile them to the same 512x512 that Peter mentions. That assumes clients are typically looking at parts of the TIFF file, which
is true if we are just trying to speed up a WMS server.<br class="">
In that sense the 512 rings true to me, but having gone back and read Frank's words on the landsat ingestor process, I can see that the 512 is my typical use-case and maybe not someone else's best practice for sat scenes.
<br class="">
In fact, now that I think about it, I just saw a great demo yesterday? (seems like yesterday) where the whole point was interactive coding that was processing multiple L8 bands, and not some part of scene, but the whole scene, or scenes. Not sure about this
but that code is probably quite happy with no tiles.<br class="">
<br class="">
Also, I would disagree on tiling for latency reasons. I don't think we will be doing range gets to S3 for this data. With what I saw yesterday and what I do with aerial imagery, the first request for the L8 band or aerial image might have some latency due to
having to get the file off of S3, but subsequent requests will be to a cached copy on SSD 'local' to the EC2 instance.<br class="">
Also, latency is an interesting question with S3. From the individual compute node perspective the first request will be slow vs a local disk. But if you have a large cluster of nodes requesting data at the same time off of the same shared storage system,
that storage starts looking very low latency. <br class="">
- Mark<br class="">
<br class="">
----------------------------------------------------------------------<br class="">
<br class="">
Message: 1will<br class="">
Date: Fri, 6 Feb 2015 00:41:25 +0000<br class="">
From: Peter Becker <<a href="mailto:pbecker@esri.com" class="">pbecker@esri.com</a>><br class="">
To: "<a href="mailto:landsat-pds@lists.osgeo.org" class="">landsat-pds@lists.osgeo.org</a>" <<a href="mailto:landsat-pds@lists.osgeo.org" class="">landsat-pds@lists.osgeo.org</a>><br class="">
Subject: [Landsat-pds] Tiling Scenes<br class="">
Message-ID: <br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><<a href="mailto:A6B8217F0F87DD47AD6ACF0E268DB249B94CCEC6@RED-INF-EXMB-P1.esri.com" class="">A6B8217F0F87DD47AD6ACF0E268DB249B94CCEC6@RED-INF-EXMB-P1.esri.com</a>><br class="">
Content-Type: text/plain; charset="us-ascii"<br class="">
<br class="">
I noticed that the scenes are currently compressed using Deflate and a predictor of 2 (horizontal difference).<br class="">
The deflate brings down the file size, but does not allow sections of the imagery to be read without reading all previous pixels. This will significantly affect performance of any applications that only need to read parts of the image.<br class="">
I would like to recommend that the following is added to the GDAL_translate command<br class="">
-co TILED=YES -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 Tiling will improve access and is a standard part of TIF.<br class="">
Using a tiling size of 512 is more optimum for storage that has higher latency.<br class="">
In my tests I noticed a slight (1-2%) decrease in the resulting filesize as well.<br class="">
<br class="">
_Peter<br class="">
<br class="">
-------------- next part --------------<br class="">
An HTML attachment was scrubbed...<br class="">
URL: <<a href="http://lists.osgeo.org/pipermail/landsat-pds/attachments/20150206/baa6bead/attachment-0001.html" class="">http://lists.osgeo.org/pipermail/landsat-pds/attachments/20150206/baa6bead/attachment-0001.html</a>><br class="">
<br class="">
------------------------------<br class="">
<br class="">
Message: 2<br class="">
Date: Fri, 6 Feb 2015 00:41:25 +0000<br class="">
From: Peter Becker <<a href="mailto:pbecker@esri.com" class="">pbecker@esri.com</a>><br class="">
To: "<a href="mailto:landsat-pds@lists.osgeo.org" class="">landsat-pds@lists.osgeo.org</a>" <<a href="mailto:landsat-pds@lists.osgeo.org" class="">landsat-pds@lists.osgeo.org</a>><br class="">
Subject: [Landsat-pds] Including Overviews<br class="">
Message-ID:<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span><<a href="mailto:A6B8217F0F87DD47AD6ACF0E268DB249B94CCECC@RED-INF-EXMB-P1.esri.com" class="">A6B8217F0F87DD47AD6ACF0E268DB249B94CCECC@RED-INF-EXMB-P1.esri.com</a>><br class="">
Content-Type: text/plain; charset="us-ascii"<br class="">
<br class="">
Currently the scenes do not include any overviews. This forces applications to read the imagery at full resolution even if accessing at a small scale.<br class="">
It's common to include overviews/pyramids/reduced resolution datasets with such imagery.<br class="">
I would like to recommend that such overviews are included with the imagery<br class="">
<br class="">
Typically this would increase the file size by approx 33%, which is quite considerable.<br class="">
The size of the overviews can be reduced by skipping a level (eg use level 4 8 16 32) vs 2 4 8 16 32, alternatively using a factor of 3 ie 3 9 27 81 I tested this out on some images using Deflate with predictor 2<br class="">
2 4 8 16 32 adds about 35%<br class="">
4 8 16 32 adds about 9%<br class="">
3 9 27 81 adds about 13%<br class="">
My recommendation is to go with the 3 9 27 81 factors. This keeps the factor constant between all levels.<br class="">
Average sampling would be used, except for the BQA band for which Nearest (else Majority) should be used.<br class="">
<br class="">
Implementation would only require adding GDALADDO to the existing script<br class="">
<br class="">
We would also need to determine if we use internal or external overviews.<br class="">
Internal overviews would increase the size of the files, but not change the directory structure.<br class="">
Using external overviews (.tif.ovr) would add additional files to the directory, is more specific to GDAL.<br class="">
<br class="">
As I expect many users to be using GDAL to access these files, I would recommend that we use the external overviews. I don't see the additional files as a burden.<br class="">
<br class="">
One other potential concern (and I'm hoping Frank can answer this) is the tile size of the overviews. GDAL appear to always use a tile size (BLOCKSIZE) of 128. It would be more optimum if the file size could be set to 512.<br class="">
<br class="">
_Peter<br class="">
<br class="">
-------------- next part --------------<br class="">
An HTML attachment was scrubbed...<br class="">
URL: <<a href="http://lists.osgeo.org/pipermail/landsat-pds/attachments/20150206/5d1223ae/attachment-0001.html" class="">http://lists.osgeo.org/pipermail/landsat-pds/attachments/20150206/5d1223ae/attachment-0001.html</a>><br class="">
<br class="">
------------------------------<br class="">
<br class="">
_______________________________________________<br class="">
Landsat-pds mailing list<br class="">
<a href="mailto:Landsat-pds@lists.osgeo.org" class="">Landsat-pds@lists.osgeo.org</a><br class="">
http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds<br class="">
<br class="">
<br class="">
End of Landsat-pds Digest, Vol 2, Issue 4<br class="">
*****************************************<br class="">
_______________________________________________<br class="">
Landsat-pds mailing list<br class="">
Landsat-pds@lists.osgeo.org<br class="">
http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>