[Landsat-pds] Including Overviews

[Sundwall, Jed]

Let’s discuss this in a week when we have our check-in call. I’m not the expert here, but it sounds like adding tiling is simple enough that it’s worth considering.

However, I’m still +0 on pyramids at this point. We’re open to providing enhancements to the raw data in the future, but would like to get more feedback from users before we make any decisions.

Until Wednesday…


On Feb 10, 2015, at 4:31 PM, Peter Becker <pbecker at esri.com<mailto:pbecker at esri.com>> wrote:

Jed
Providing overviews for the imagery will enable user to use the imagery in a lot more applications without the need and time to generate pyramids themselves. Creating pyramids does require the whole image to be read. Hence if we do not provide then users often be forced to download the complete image vs just using it from S3. The lack of overviews will restrict user from access the imagery efficiently and so I recommend that we  do added as external .OVR files with deflate and factor 3. Doing this at the same time as the base tif files are created will be minimal additional processing in comparison to having to create them later. Having these as external .OVR files will mean that users that do not need them, need not download them. Applications working with GDAL will be able to directly use them. This will significantly increase usability of the data. If users want to generate pyramids their own way they can always still download and run a similar process.

Frank could you look to add this to the existing scripts and possibly apply to some samples.
                gdaladdo.exe -r average -ro --config COMPRESS_OVERVIEW DEFALTE  --config USE_RRD NO --config PREDICTOR_OVERVIEW 2 --config TILED YES scene.tif 3 9 27 81

_Peter

From: Sundwall, Jed [mailto:jsundwal at amazon.com]
Sent: Friday, February 06, 2015 3:30 PM
To: Peter Becker
Cc: landsat-pds at lists.osgeo.org<mailto:landsat-pds at lists.osgeo.org>
Subject: Re: [Landsat-pds] Including Overviews

I’ve done the math, and we can certainly afford to take on an additional 13% of data volume over the next two years, but I’d like to take some more time to do research on demand for this. The approach we’ve taken so far has been to touch the data as little as possible, and beyond the jpg thumbnails we produce, we aren’t creating any derivative products from the raw scenes.

In the meantime, any customer could produce these overlays themselves if they wanted to, but if we hear from multiple customers that they’re doing redundant work to do this, we'll consider adding overviews to this data set.

Open to others’ thoughts on this.

Jed.

On Feb 5, 2015, at 4:41 PM, Peter Becker <pbecker at esri.com<mailto:pbecker at esri.com>> wrote:

Currently the scenes do not include any overviews. This forces applications to read the imagery at full resolution even if accessing at a small scale.
It’s common to include overviews/pyramids/reduced resolution datasets with such imagery.
I would like to recommend that such overviews are included with the imagery

Typically this would increase the file size by approx 33%, which is quite considerable.
The size of the overviews can be reduced by skipping a level (eg use level  4 8 16 32) vs 2 4 8 16 32, alternatively using a factor of 3 ie 3 9 27 81
I tested this out on some images using Deflate with predictor 2
2 4 8 16 32  adds about 35%
4 8 16 32  adds about 9%
3 9 27 81 adds about 13%
My recommendation is to go with the 3 9 27 81 factors. This keeps the factor constant between all levels.
Average sampling would be used, except for the BQA band for which Nearest (else Majority) should be used.

Implementation would only require adding GDALADDO to the existing script

We would also need to determine if we use internal or external overviews.
Internal overviews would increase the size of the files, but not change the directory structure.
Using external overviews (.tif.ovr) would add additional files to the directory, is more specific to GDAL.

As I expect many users to be using GDAL to access these files, I would recommend that we use the external overviews. I don’t see the additional files as a burden.

One other potential concern (and I’m hoping Frank can answer this) is the tile size of the overviews. GDAL appear to always use a tile size (BLOCKSIZE) of 128. It would be more optimum if the file size could be set to 512.

_Peter

_______________________________________________
Landsat-pds mailing list
Landsat-pds at lists.osgeo.org<mailto:Landsat-pds at lists.osgeo.org>
http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/landsat-pds/attachments/20150212/24efc0b3/attachment-0001.html>