[GRASS-user] Re: off-topis: bulk downloading of Landsat tiles from GLCF

[Hamish]

maning:
> > How do I download in bulk a number of Landsat tiles from GLCF.
> > for example:
> ftp://ftp.glcf.umiacs.umd.edu/glcf/Landsat/WRS2/p114/r052/p114r052_7x20010909.ETM-EarthSat-Orthorectified/
> > I want to download only the bands 1-5 and 7and the
> > metadata.
> >
> > I need to do this for 52+ plus Landsat tiles.

a little wget shell script loop is given here:
  http://grass.osgeo.org/wiki/MODIS#Download

maybe you could adapt that?


> Just read the esdi warning:
> http://glcf.umiacs.umd.edu/esdi2-help/expert.html
> "Do not use these layers to automate a bulk download
> (over 50 scenes) without contacting us first. Violations will
> result in the ban of your IP address or domain."

The idea is to not abuse their server, so others are not denied access by a single greedy use hogging the system. I assume that they are working with finite resources and so forth, and FTP usually only has a small number of connections allowed at once.


> Maybe ten at a time

maybe ten a day, and put a "sleep <number of seconds>" after each call to wget to give someone else the chance to connect. The ban sounds automated, and thus strict. No idea how long it takes for the download log to cycle, it could be once a day, it could be once a week. But a 10 minute break between batches is probably not enough.

or, as they request, drop them an friendly email saying you'd like to download 52+ images in a way that suits them.


Hamish


p.s. there are other solutions for them: http://geotorrent.org/