[Landsat-pds] usgs service issues?

[Amit Kapadia]

Hi Frank,

We pass multiple scene ids in each download request. This minimizes the
number of requests made to USGS's servers, keeping us on friendly terms
with them. We've found that one download request to get multiple download
URLs, followed by parallel downloads is effective in avoiding 500s.
Parallel downloads should be throttled; we happen to limit at 4 concurrent
downloads.

Despite having built redundancy for connections errors (e.g. 503s), we
still miss up to 4% of the scenes each night.

I'm wondering if it would be easier to use the code that we've already
written to get this task done. It would be minimal effort to point the
tarballs to the public bucket. We would have to add the additional
functionality that you've built (e.g. splitting into individual scenes,
preview images, index page).

Cheers,
Amit


On Sat, Jan 10, 2015 at 11:07 AM, Frank Warmerdam <warmerdam at pobox.com>
wrote:

> Amit,
>
> I was only downloading one scene on that connection, but other processes
> were potentially processing others at the same time.
>
> Currently I only ever pass one scene id to the request to get download
> urls though it was using, not requesting the download url that failed.
>
> Best regards,
> Frank
>
> On Sat, Jan 10, 2015 at 7:35 AM, Amit Kapadia <amit at mapbox.com> wrote:
>
>> Hi Frank,
>>
>> Documentation says each download request takes up to 50,000 scene ids. In
>> practice the limit is lower, where the limiting factor is the size of the
>> request. How many scenes where you requesting?
>>
>> Cheers,
>> Amit
>>
>>
>> On Sat, Jan 10, 2015 at 12:16 AM, Frank Warmerdam <warmerdam at pobox.com>
>> wrote:
>>
>>> Amit,
>>>
>>> I ran into this once when using a download url for the usgs service:
>>>
>>> requests.exceptions.HTTPError: 503 Server Error: Service Temporarily
>>> Unavailable
>>>
>>> Are there limits on the number of download urls I should be fetching at
>>> once?  Is this a common problem?
>>>
>>> I haven't run enough at once to know if this is going to be a broader
>>> problem, but if so it will mean I need to reconsider my approach which can
>>> result in quite a few parallel downloads.
>>>
>>> Best regards,
>>> --
>>>
>>> ---------------------------------------+--------------------------------------
>>> I set the clouds in motion - turn up   | Frank Warmerdam,
>>> warmerdam at pobox.com
>>> light and sound - activate the windows | http://pobox.com/~warmerdam
>>> and watch the world go round - Rush    | Geospatial Software Developer
>>>
>>
>>
>
>
> --
>
> ---------------------------------------+--------------------------------------
> I set the clouds in motion - turn up   | Frank Warmerdam,
> warmerdam at pobox.com
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush    | Geospatial Software Developer
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/landsat-pds/attachments/20150110/27eb5d89/attachment.html>