<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body dir="auto">
<div>Thank you, Peter! Does anyone else want to chime in here?</div>
<div><br>
</div>
<div>I take the point about the GLS processing being a one-time thing, but I'd like to at least have the scripts documented on GitHub. It's certainly fine if they're part of a separate repo. </div>
<div><br>
</div>
<div>Jed. <br>
<br>
<br>
</div>
<div><br>
On Apr 16, 2015, at 4:09 PM, Peter Becker <<a href="mailto:pbecker@esri.com">pbecker@esri.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Jed<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">My suggestion:<o:p></o:p></span></p>
<p class="MsoNormal"># Directory Structure<o:p></o:p></p>
<p class="MsoNormal" style="text-indent:.5in">landsat-pds/gls/epoch/path/row/<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">There needs to be a way differentiate the years easily.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"># Compression, tiling, and overlays<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Use same as Landsat 8. Only difference is that they data is 8bit vs 16bit/channel<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"># Creating the scripts<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">For the GLS data it is a 1 shot process. Should be mostly similar to L8. Would be great to have it generic to work with the different Landsat scenes so that
we can later potentially add other scenes, but it may be more work than just getting these GLS scenes done.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Note that from our finding there is some disconnect about what the GLS scenes are. There are different lists and some include more than others. Best to take
what is in the existing S3 NEX bucket and then identify those missing. I can look to provide some resources for this process, although would default first to see what Frank suggests.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">_Peter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:landsat-pds-bounces@lists.osgeo.org">landsat-pds-bounces@lists.osgeo.org</a> [<a href="mailto:landsat-pds-bounces@lists.osgeo.org">mailto:landsat-pds-bounces@lists.osgeo.org</a>]
<b>On Behalf Of </b>Sundwall, Jed<br>
<b>Sent:</b> Thursday, April 16, 2015 12:50 PM<br>
<b>To:</b> <a href="mailto:landsat-pds@lists.osgeo.org">landsat-pds@lists.osgeo.org</a><br>
<b>Subject:</b> [Landsat-pds] Bringing GLS Scenes into s3://landsat-pds<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We’re planning to add Landsat GLS data into <a href="s3://landsat-pds">
s3://landsat-pds</a>, and I’d like to validate our approach on organizing the files with the group, because organizing GLS data is slightly more complicated than L8 data.
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"># Directory Structure<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">The main complication of GLS data is that it’s divided into 5 “epochs” GLS1975, GLS1990, GLS2000, GLS2005, and GLS2010. There are variations in file names across epochs due to differences in sensors used, which are described thoroughly
in the “GLS Collection” section of this page <a href="http://glcf.umd.edu/library/guide/namingconvention.shtml">http://glcf.umd.edu/library/guide/namingconvention.shtml</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">My question is this: should we try to organize GLS data from all epochs according to the path/row structure that we’ve used for L8, or should we organize data from each epoch in its own directory, and then follow the path/row structure?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">That is, do we list available scenes in a directory like this?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">landsat-pds/gls/path/row/<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">or one like this?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">landsat-pds/gls/epoch/path/row/<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">There’s something nice about making it easy to see all available data for a path/row within one directory, but I’m not sure if that will be the most useful organization for users of the data. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"># Compression, tiling, and overlays<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Does it make sense to apply the same in-file compression and internal tiling to GLS scenes? Should we also create overlays for the scenes? <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"># Creating the scripts<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">What scripts would we need for this and is there a best way for us to incorporate them into the ingestor?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Right now, we have l8_aws_config.py, l8_lib.py, l8_process_run.py, l8_process_scene.py, and l8_queue_tar.py scripts. Would we need the equivalent scripts for GLS data or is there a way we could rely on the existing scripts for most of the
work, but apply slightly different rules for GLS scenes. However we go about it, it’d be nice to make this extensible so we could do the same thing for L7, L5, etc.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Any response to this would be very appreciated. As soon as we have a good plan, we can start bringing in the data!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks, Jed.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</blockquote>
</body>
</html>