<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 29, 2015 at 3:05 PM, Sundwall, Jed <span dir="ltr"><<a href="mailto:jsundwal@amazon.com" target="_blank">jsundwal@amazon.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<a href="http://38.media.tumblr.com/84eb17389e8be791d0c3de2d606dc838/tumblr_mxx0yrW3Rl1qa5znqo1_400.gif" target="_blank">Celebration!</a><br>
<div><br>
</div>
<div>We’ve updated the existing <a href="http://landsat-pds.s3.amazonaws.com/index.html" target="_blank">JavaScript S3 Explorer</a> to use the right endpoints to download files, but we need to resolve a few things before we’re happy with it.</div>
<div><br>
</div>
<div>1. It doesn’t update the URL in the browser as you navigate. I.e. If I click into row 100 and path 050 and try to copy and paste the URL from my browser, it will just send you back to where you started.</div>
<div>2. For the basic directory navigation, it’s way too crufty. We don’t need tables for that. We need lists!</div>
<div>3. I don’t want to use this approach for the individual scene index.html files. Scene index.html should be more or less like Frank’s and should be easy for search engines to index.</div>
<div><br>
</div>
<div>A few questions for Frank:</div>
<div><br>
</div>
<div>1. What’s expensive about the current approach? Is it the fact that all of the path/row index.html files need to be re-written every time a new scene is added?</div></div></blockquote><div><br></div><div>Yes, it is writing 12000 index files in S3 which is pretty fast, but still a couple minutes and a bunch of mostly zero value S3 churn. I could actually get smarter and only updates index files that will have changes if we wanted to keep doing it this way.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">
<div>2. What are the pain points for creating the scene specific index.html files? As far as search engine indexing goes, we wouldn’t need much info in plain text other than the scene name and some boiler plate language. </div></div></blockquote><div><br></div><div>There isn't really any pain here though I look forward to suggestions, or much better pull requests, to improve the formatting and useful information in these files. </div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">
<div>3. What do you think about creating a site map and updating it with URLs for individual scenes as they’re created? That way we don’t have to worry about crawlers not knowing how to navigate the JavaScript tree browser but can just get a list of
every scene’s URL?</div></div></blockquote><div><br></div><div>The urls are all in scene_list.gz so it would be wonderful if you could scan that periodically and write back a site map. I wasn't actually just thinking about search engines when I talked about walking the tree. It is also not uncommon for me to write a crawler for subareas of geodata download sites to find all the things I can download.</div><div><br></div><div>Best regards,</div><div>Frank</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">
<div><br>
</div>
<div>I’ll keep noodling on the JavaScript explorer.</div>
<div><br>
</div>
<div>Again, Celebration! Thanks to everyone who worked on this – but mostly Frank ;) We got a lot done in a very short amount of time. </div>
<div><br>
</div>
<div>Jed.</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>
<blockquote type="cite"><div><div class="h5">
<div>On Jan 29, 2015, at 2:35 PM, Frank Warmerdam <<a href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a>> wrote:</div>
<br>
</div></div><div><div><div class="h5">
<div dir="ltr">Folks,
<div><br>
</div>
<div>Good news, we are now caught up to the USGS feed, and modulo occasional wedges in the pipeline, and 503 related delays from USGS we should be up-to-date within a couple hours of USGS offering scenes.</div>
<div><br>
</div>
<div>There is an open ticket on semi-broken index.html files for scenes without the RGB bands I plan to work on soon, but I don't think that is too significant.</div>
<div><br>
</div>
<div>I mentioned the script that creates the index files at higher levels in the tree and this is running, but I'd like to drop that if Jed can offer a more dynamic (and less expensive to update) tree browser. I will say, the benefit of the plain
index.html files is that very ordinary web crawlers can walk them.</div>
<div><br>
</div>
<div>Best regards,</div>
<div>-- <br>
<div>---------------------------------------+--------------------------------------<br>
I set the clouds in motion - turn up | Frank Warmerdam, <a href="mailto:warmerdam@pobox.com" target="_blank">
warmerdam@pobox.com</a><br>
light and sound - activate the windows | <a href="http://pobox.com/~warmerdam" target="_blank">
http://pobox.com/~warmerdam</a><br>
and watch the world go round - Rush | Geospatial Software Developer<br>
</div>
</div>
</div></div></div>
_______________________________________________<br>
Landsat-pds mailing list<br>
<a href="mailto:Landsat-pds@lists.osgeo.org" target="_blank">Landsat-pds@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds" target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds</a><br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">---------------------------------------+--------------------------------------<br>I set the clouds in motion - turn up | Frank Warmerdam, <a href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a><br>light and sound - activate the windows | <a href="http://pobox.com/~warmerdam" target="_blank">http://pobox.com/~warmerdam</a><br>and watch the world go round - Rush | Geospatial Software Developer<br></div>
</div></div>