[Landsat-pds] Upcoming updates to Landsat on AWS data

[Amit Kapadia]

We're ingesting about 1.35 scenes per minute (~2000 scenes per day). With
44,200 scenes remaining, this work should be complete in 22 - 23 days.

The additional worker has kicked up the rate. I'm learning more about the
rate-limiting that USGS imposes, and it seems that a single machine is
limited to 2 concurrent downloads (we already knew this). However, we have
3 machines running, so the rate-limiting appears to be a combination
between IP address and EROS account.

Cheers,
Amit


On Mon, Sep 14, 2015 at 3:58 PM, Sundwall, Jed <jed at amazon.com> wrote:

> Thanks for the update, Amit. Could you please let us know if you see that
> the extra workers have upped our rate? Also, if you can estimate when this
> would be done?
>
> Thank you very much for your work on this!
>
> Jed.
>
> On Sep 14, 2015, at 2:16 PM, Amit Kapadia <amit at planet.com> wrote:
>
> Hi all - an update to the ingestion of these reprocessed Landsat scenes.
> Using the additional bandwidth that Jed locked down, we've ingested ~8,000
> of the 52,877 scenes. This has been moving a little slow, so I've bumped up
> the number of workers.
>
> In the past we've been restricted to 2 concurrent downloads from USGS
> servers, but it now seems that we're able to get 4 concurrent downloads.
> I'll try our luck with one more worker (2 more downloads) to see if we're
> allowed this luxury.
>
> Ingestion of new Landsat scenes continues as normal.
>
> Cheers,
> Amit
>
>
> On Wed, Aug 26, 2015 at 12:08 PM, Sundwall, Jed <jed at amazon.com> wrote:
>
>> Quick update:
>>
>> We have been granted additional bandwidth to acquire Landsat data from
>> EROS and will use it to reacquire 53,206 scenes that have been reprocessed
>> with updated TIRS data as described at
>> http://landsat.usgs.gov/calibration_notices.php
>>
>> We will also use this opportunity to check for any scenes that we may
>> have failed to acquire throughout 2015. Another user of the data recently
>> pointed out that "USGS states that there are 149307 scenes so far in 2015,
>> but AWS claims to host only 145746 of them. As a percentage, that is 97.6%
>> - IOW 2.4% are missing.” These scenes may be missing from the bucket or
>> they may merely be missing from the scene_list.gz file.
>>
>> I’ll update the list once the reacquisition is complete.
>>
>> Thanks,
>>
>> Jed Sundwall – Open Data – Amazon Web Services
>>
>> cell: 801-949-1482
>> office: 206-435-3104
>>
>> https://aws.amazon.com/opendata/
>>
>>
>> _______________________________________________
>> Landsat-pds mailing list
>> Landsat-pds at lists.osgeo.org
>> http://lists.osgeo.org/cgi-bin/mailman/listinfo/landsat-pds
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/landsat-pds/attachments/20150915/c7f26f43/attachment-0001.html>