[Landsat-pds] Estimated lag-time between USGS and AWS

[Amit Kapadia]

Hi Robert,

It turns out that we had a bug that affected only T1 scenes. Due to the
processing latency of T1 data (observed up to ~20 days), we have to query
against USGS's API, paginated in one-week intervals. It is set to query for
scenes with an acquisition date up to one month back, leaving a buffer in
case some scenes take longer than 20 days to become available. Only one of
these paginated results were being queued for ingestion, leaving out a good
portion of scenes. This is now fixed, so we should observe more T1 scenes
being made available. The example scene you linked to
(LC81370412017161LGN00) is queued up right now.

Thanks for bringing this issue to our attention.

Cheers,
Amit


On Tue, Jun 13, 2017 at 11:37 AM, Amit Kapadia <amit at planet.com> wrote:

> Hi Robert,
>
> For Tier 1 data we poll USGS once per day. We used to poll twice per day,
> but found that they tend to release one large batch each day rather than a
> continuous stream. For RT data we continue to poll twice per day.
>
> We've observed a high variance in when T1 data becomes available. It can
> range between 0 - 20 days due to, I believe, lunar models that are made
> available on a roughly bi-weekly interval.
>
> Those are two sources of latency. To your question, I've run on the
> assumption that as soon as a scene is made available through EarthExplorer,
> it's also made available through USGS's API. This might not be the case as
> demonstrated by the example scene you shared. I've poked the USGS API, and
> it does appear it will be queued for ingestion later today. I'll keep an
> eye on it as there might be problem on our end.
>
> Thanks for reaching out.
>
> Amit
>
> On Tue, Jun 13, 2017 at 10:54 AM, Robert Hildebrand <
> robert_hildebrand at trimble.com> wrote:
>
>> Hello,
>>
>> I am wondering if there is a known time gap between when Tier 1
>> collection data is available via the USGS and when it is subsequently
>> replicated into AWS.
>>
>> As of 2017-06-13, the most recent Tier 1 data is from 2017-05-23.
>>
>> The Real Time data is replicated into AWS much faster - the most recent
>> RT data is from 2017-06-12.
>>
>> Here is an example of Tier 1 data that is available via the USGS, but has
>> not been replicated into AWS: https://earthexplorer.usg
>> s.gov/metadata/12864/LC81370412017161LGN00/
>>
>> Thank you for any timing information that you can provide.
>>
>>
>> _______________________________________________
>> Landsat-pds mailing list
>> Landsat-pds at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/landsat-pds
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/landsat-pds/attachments/20170614/bb53d7dd/attachment.html>