<div dir="ltr">Amit,<div><br></div><div>OK, thanks for the feedback. I guess I will segment my ingestion into two steps. One that is run on a singleton VM to pull files from usgs, and push them into a working location in S3, and the other that unpacks and repacks and can be run massively parallel. It is disappointing, but I suppose not surprising that the USGS servers aren't very well setup to handle large numbers of connections at the same time. </div><div><br></div><div>Best regards,</div><div>Frank</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 10, 2015 at 11:02 AM, Amit Kapadia <span dir="ltr"><<a href="mailto:amit@mapbox.com" target="_blank">amit@mapbox.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Frank,<br></div><div><br></div><div>We pass multiple scene ids in each download request. This minimizes the number of requests made to USGS's servers, keeping us on friendly terms with them. We've found that one download request to get multiple download URLs, followed by parallel downloads is effective in avoiding 500s. Parallel downloads should be throttled; we happen to limit at 4 concurrent downloads.</div><div><br></div><div>Despite having built redundancy for connections errors (e.g. 503s), we still miss up to 4% of the scenes each night.</div><div><br></div><div>I'm wondering if it would be easier to use the code that we've already written to get this task done. It would be minimal effort to point the tarballs to the public bucket. We would have to add the additional functionality that you've built (e.g. splitting into individual scenes, preview images, index page).</div><div><br></div><div>Cheers,</div><div>Amit</div><div><div class="h5"><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 10, 2015 at 11:07 AM, Frank Warmerdam <span dir="ltr"><<a href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Amit,<br><br>I was only downloading one scene on that connection, but other processes were potentially processing others at the same time.<br><br></div>Currently I only ever pass one scene id to the request to get download urls though it was using, not requesting the download url that failed.<br><br></div>Best regards,<br>Frank<br></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 10, 2015 at 7:35 AM, Amit Kapadia <span dir="ltr"><<a href="mailto:amit@mapbox.com" target="_blank">amit@mapbox.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Frank,<div><br></div><div>Documentation says each download request takes up to 50,000 scene ids. In practice the limit is lower, where the limiting factor is the size of the request. How many scenes where you requesting?</div><div><br></div><div>Cheers,</div><div>Amit</div><div><br></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 10, 2015 at 12:16 AM, Frank Warmerdam <span dir="ltr"><<a href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Amit, <div><br></div><div>I ran into this once when using a download url for the usgs service:</div><div><br></div><div><div>requests.exceptions.HTTPError: 503 Server Error: Service Temporarily Unavailable</div></div><div><br></div><div>Are there limits on the number of download urls I should be fetching at once? Is this a common problem?</div><div><br></div><div>I haven't run enough at once to know if this is going to be a broader problem, but if so it will mean I need to reconsider my approach which can result in quite a few parallel downloads. </div><div><br></div><div>Best regards,</div><span><font color="#888888"><div>-- <br><div>---------------------------------------+--------------------------------------<br>I set the clouds in motion - turn up | Frank Warmerdam, <a href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a><br>light and sound - activate the windows | <a href="http://pobox.com/~warmerdam" target="_blank">http://pobox.com/~warmerdam</a><br>and watch the world go round - Rush | Geospatial Software Developer<br></div>
</div></font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div>---------------------------------------+--------------------------------------<br>I set the clouds in motion - turn up | Frank Warmerdam, <a href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a><br>light and sound - activate the windows | <a href="http://pobox.com/~warmerdam" target="_blank">http://pobox.com/~warmerdam</a><br>and watch the world go round - Rush | Geospatial Software Developer<br></div>
</div>
</div></div></blockquote></div><br></div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">---------------------------------------+--------------------------------------<br>I set the clouds in motion - turn up | Frank Warmerdam, <a href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a><br>light and sound - activate the windows | <a href="http://pobox.com/~warmerdam" target="_blank">http://pobox.com/~warmerdam</a><br>and watch the world go round - Rush | Geospatial Software Developer<br></div>
</div>