[Landsat-pds] Bringing GLS Scenes into s3://landsat-pds

[Sundwall, Jed]

Thank you, Peter! Does anyone else want to chime in here?

I take the point about the GLS processing being a one-time thing, but I'd like to at least have the scripts documented on GitHub. It's certainly fine if they're part of a separate repo.

Jed.



On Apr 16, 2015, at 4:09 PM, Peter Becker <pbecker at esri.com<mailto:pbecker at esri.com>> wrote:

Jed
My suggestion:
# Directory Structure
landsat-pds/gls/epoch/path/row/
There needs to be a way differentiate the years easily.

# Compression, tiling, and overlays
Use same as Landsat 8. Only difference is that they data is 8bit vs 16bit/channel

# Creating the scripts
For the GLS data it is a 1 shot process. Should be mostly similar to L8. Would be great to have it generic to work with the different Landsat scenes so that we can later potentially add other scenes, but it may be more work than just getting these GLS scenes done.
Note that from our finding there is some disconnect about what the GLS scenes are. There are different lists and some include more than others. Best to take what is in the existing S3 NEX bucket and then identify those missing. I can look to provide some resources for this process, although would default first to see what Frank suggests.

_Peter


From: landsat-pds-bounces at lists.osgeo.org<mailto:landsat-pds-bounces at lists.osgeo.org> [mailto:landsat-pds-bounces at lists.osgeo.org] On Behalf Of Sundwall, Jed
Sent: Thursday, April 16, 2015 12:50 PM
To: landsat-pds at lists.osgeo.org<mailto:landsat-pds at lists.osgeo.org>
Subject: [Landsat-pds] Bringing GLS Scenes into s3://landsat-pds

We’re planning to add Landsat GLS data into s3://landsat-pds, and I’d like to validate our approach on organizing the files with the group, because organizing GLS data is slightly more complicated than L8 data.

# Directory Structure
The main complication of GLS data is that it’s divided into 5 “epochs” GLS1975, GLS1990, GLS2000, GLS2005, and GLS2010. There are variations in file names across epochs due to differences in sensors used, which are described thoroughly in the “GLS Collection” section of this page http://glcf.umd.edu/library/guide/namingconvention.shtml

My question is this: should we try to organize GLS data from all epochs according to the path/row structure that we’ve used for L8, or should we organize data from each epoch in its own directory, and then follow the path/row structure?

That is, do we list available scenes in a directory like this?

landsat-pds/gls/path/row/

or one like this?

landsat-pds/gls/epoch/path/row/

There’s something nice about making it easy to see all available data for a path/row within one directory, but I’m not sure if that will be the most useful organization for users of the data.

# Compression, tiling, and overlays

Does it make sense to apply the same in-file compression and internal tiling to GLS scenes? Should we also create overlays for the scenes?

# Creating the scripts

What scripts would we need for this and is there a best way for us to incorporate them into the ingestor?

Right now, we have l8_aws_config.py, l8_lib.py, l8_process_run.py, l8_process_scene.py, and l8_queue_tar.py scripts. Would we need the equivalent scripts for GLS data or is there a way we could rely on the existing scripts for most of the work, but apply slightly different rules for GLS scenes. However we go about it, it’d be nice to make this extensible so we could do the same thing for L7, L5, etc.

Any response to this would be very appreciated. As soon as we have a good plan, we can start bringing in the data!

Thanks, Jed.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/landsat-pds/attachments/20150423/02ef482f/attachment.html>