<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Amit, this is great news! Before I reach out to USGS, should we use the extra account to acquire any other scenes that we missed throughout the year?
<div class=""><br class="">
</div>
<div class="">Also, are we still getting a lot of errors?<br class="">
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Oct 26, 2015, at 6:04 AM, Amit Kapadia <<a href="mailto:amit@planet.com" class="">amit@planet.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">Good news. We've finished ingesting the ~53,000 reprocessed scenes.
<div class=""><br class="">
</div>
<div class="">Jed - you can follow up with USGS to revoke the extra account.</div>
<div class=""><br class="">
</div>
<div class="">Cheers,</div>
<div class="">Amit</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Fri, Oct 2, 2015 at 4:46 PM, Amit Kapadia <span dir="ltr" class="">
<<a href="mailto:amit@planet.com" target="_blank" class="">amit@planet.com</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="">Jed - I can't give a definitive answer, but I suspect we'll start to fall behind. I just checked our ingestion from September, and we're doing well. All images released in September were uploaded to S3 by Oct 1. To keep this pace, we
do have a machine running all the time. Our ingestion job has started to fail about 1/3 of the time due to the new rate limiting. It would be nice to understand the full scope of these constraints. Ideally we'd be able to talk to one of the developers to better
understand how best to operate.</div>
<div class="HOEnZb">
<div class="h5">
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Wed, Sep 30, 2015 at 12:50 PM, Sundwall, Jed <span dir="ltr" class="">
<<a href="mailto:jed@amazon.com" target="_blank" class="">jed@amazon.com</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">Thanks for the update, Amit. Is it possible that this new limit could cause us to fall behind in acquiring all new scenes as they’re produced each day?
<div class="">
<div class="">
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Sep 30, 2015, at 12:00 PM, Amit Kapadia <<a href="mailto:amit@planet.com" target="_blank" class="">amit@planet.com</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" class="">Hey Jed,
<div class=""><br class="">
</div>
<div class="">Thanks for reaching out to them. Looks like we have another rate-limiting error to handle:
<div class="">
<div class=""><br class="">
</div>
<div class=""><font face="monospace, monospace" class="">usgs.USGSError: RATE_LIMIT: Rate limit exceeded - cannot support simultaneous requests.</font><br class="">
</div>
<div class=""><font face="monospace, monospace" class=""><br class="">
</font></div>
<div class="">According to the changelog of the USGS inventory service:<font face="monospace, monospace" class=""><br class="">
</font></div>
<div class=""><br class="">
</div>
<div class="">
<div class="">August 2015</div>
<div class=""><br class="">
</div>
<div class=""> * Implemented single-stream rate limiting</div>
<div class=""> * Added FGDC Metadata URL to search and metadata responses</div>
<div class=""> * API Key is now required for all requests</div>
</div>
<div class=""><br class="">
</div>
<div class="">Despite the change being made in August, we're only now starting to see this error. Previously, we were allowed 2 simultaneous downloads per machine. This has been cut in half. To keep up with the flow of Landsat scenes, we need simultaneous requests.
This error is cropping up periodically in our re-ingestion of the ~53,000 scenes, as well as our daily ingestion.</div>
<div class=""><br class="">
</div>
<div class="">Enforcing single-stream per machine is a terrible waste of computing resources.</div>
<div class=""><br class="">
</div>
<div class="">Also note, the need of an API key for all requests. Previously, anyone was able to programmatically access metadata. This is no longer possible.</div>
</div>
<div class=""><br class="">
</div>
<div class="">Any help would be appreciated.</div>
<div class=""><br class="">
</div>
<div class="">Cheers,</div>
<div class="">Amit</div>
</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Mon, Sep 28, 2015 at 3:38 PM, Sundwall, Jed <span dir="ltr" class="">
<<a href="mailto:jed@amazon.com" target="_blank" class="">jed@amazon.com</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">I’ve reached out to USGS to ask if we can increase the limit.
<div class=""><br class="">
</div>
<div class="">Thanks for the update, Amit!
<div class="">
<div class=""><br class="">
<div class=""></div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Sep 28, 2015, at 12:16 PM, Amit Kapadia <<a href="mailto:amit@planet.com" target="_blank" class="">amit@planet.com</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" class="">Another update on the reingestion of these ~53,000 scenes. We've moving along faster than the initial few weeks. Currently we have ~28,500 scenes left to reprocess. This is taking a bit of time, mostly because USGS rate limits the number
of scenes that can be simultaneously downloaded.
<div class=""><br class="">
</div>
<div class="">Jed - we often hit an error of this sort:</div>
<div class=""><br class="">
</div>
<div class=""><font face="monospace, monospace" class="">DOWNLOAD_RATE_LIMIT: User currently has more than 10 downloads that have not been attempted in the past 10 minutes.</font><br class="">
</div>
<div class=""><font face="monospace, monospace" class=""><br class="">
</font></div>
<div class="">If there's a way we can work with USGS on getting this type of rate-limiting lifted, I'll be able to spin up additional workers, breaking through this 10 scene limit. No big deal if that's not possible.<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Cheers,</div>
<div class="">Amit</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Tue, Sep 15, 2015 at 10:58 AM, Amit Kapadia <span dir="ltr" class="">
<<a href="mailto:amit@planet.com" target="_blank" class="">amit@planet.com</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="">
<div class="">We're ingesting about 1.35 scenes per minute (~2000 scenes per day). With 44,200 scenes remaining, this work should be complete in 22 - 23 days.
<div class=""><br class="">
</div>
<div class="">The additional worker has kicked up the rate. I'm learning more about the rate-limiting that USGS imposes, and it seems that a single machine is limited to 2 concurrent downloads (we already knew this). However, we have 3 machines running, so
the rate-limiting appears to be a combination between IP address and EROS account.</div>
<div class=""><br class="">
</div>
<div class="">Cheers,</div>
<div class="">Amit</div>
<div class=""><br class="">
</div>
</div>
</div>
<div class="">
<div class="">
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Mon, Sep 14, 2015 at 3:58 PM, Sundwall, Jed <span dir="ltr" class="">
<<a href="mailto:jed@amazon.com" target="_blank" class="">jed@amazon.com</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">Thanks for the update, Amit. Could you please let us know if you see that the extra workers have upped our rate? Also, if you can estimate when this would be done?
<div class=""><br class="">
</div>
<div class="">Thank you very much for your work on this!</div>
<span class=""><font color="#888888" class="">
<div class=""><br class="">
</div>
<div class="">Jed.</div>
</font></span>
<div class="">
<div class="">
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Sep 14, 2015, at 2:16 PM, Amit Kapadia <<a href="mailto:amit@planet.com" target="_blank" class="">amit@planet.com</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" class="">Hi all - an update to the ingestion of these reprocessed Landsat scenes. Using the additional bandwidth that Jed locked down, we've ingested ~8,000 of the 52,877 scenes. This has been moving a little slow, so I've bumped up the number
of workers.
<div class=""><br class="">
</div>
<div class="">In the past we've been restricted to 2 concurrent downloads from USGS servers, but it now seems that we're able to get 4 concurrent downloads. I'll try our luck with one more worker (2 more downloads) to see if we're allowed this luxury.</div>
<div class=""><br class="">
</div>
<div class="">Ingestion of new Landsat scenes continues as normal.</div>
<div class=""><br class="">
</div>
<div class="">Cheers,</div>
<div class="">Amit</div>
<div class=""> </div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Wed, Aug 26, 2015 at 12:08 PM, Sundwall, Jed <span dir="ltr" class="">
<<a href="mailto:jed@amazon.com" target="_blank" class="">jed@amazon.com</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">Quick update:
<div class=""><br class="">
</div>
<div class="">We have been granted additional bandwidth to acquire Landsat data from EROS and will use it to reacquire 53,206 scenes that have been reprocessed with updated TIRS data as described at <a href="http://landsat.usgs.gov/calibration_notices.php" target="_blank" class="">http://landsat.usgs.gov/calibration_notices.php</a> </div>
<div class=""><br class="">
</div>
<div class="">We will also use this opportunity to check for any scenes that we may have failed to acquire throughout 2015. Another user of the data recently pointed out that "USGS states that there are 149307 scenes so far in 2015, but AWS claims to host only
145746 of them. As a percentage, that is 97.6% - IOW 2.4% are missing.” These scenes may be missing from the bucket or they may merely be missing from the scene_list.gz file. </div>
<div class=""><br class="">
</div>
<div class="">I’ll update the list once the reacquisition is complete.</div>
<div class=""><br class="">
</div>
<div class="">Thanks, </div>
<br class="">
<div class="">
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word" class="">
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word" class="">
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word" class="">
Jed Sundwall – Open Data – Amazon Web Services<br class="">
<br class="">
cell: <a href="tel:801-949-1482" value="+18019491482" target="_blank" class="">801-949-1482</a><br class="">
office: <a href="tel:206-435-3104" value="+12064353104" target="_blank" class="">
206-435-3104</a></div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word" class="">
<br class="">
</div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word" class="">
<a href="https://aws.amazon.com/opendata/" target="_blank" class="">https://aws.amazon.com/opendata/</a></div>
</div>
</div>
</div>
<br class="">
</div>
<br class="">
_______________________________________________<br class="">
Landsat-pds mailing list<br class="">
<a href="mailto:Landsat-pds@lists.osgeo.org" target="_blank" class="">Landsat-pds@lists.osgeo.org</a><br class="">
<a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds" rel="noreferrer" target="_blank" class="">http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds</a><br class="">
<br class="">
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</body>
</html>