[gdal-dev] Distributed processing

[Matthew Snape]

Hi,

We currently do parallel execution of ArGIS geoprocessing tools,
although not yet with GDAL.  We use IPython to execute normal commands
in parallel.  IPython is a shell replacement that has parallel commands
that can be used from the shell or within code.  

The biggest difficulty I have found is using resources efficiently.  You
will inevitably hit a bottleneck of either cpu speed, disk bandwidth,
network bandwidth or memory.  It is worth spending some time thinking
about what resources a particular process will consume, and where
bottlenecks are likely to occur.  



-----Original Message-----
From: gdal-dev-bounces at lists.osgeo.org
[mailto:gdal-dev-bounces at lists.osgeo.org] On Behalf Of John Donovan
Sent: 12 June 2009 12:06
To: gdal-dev at lists.osgeo.org
Subject: [gdal-dev] Distributed processing

Hi,
We currently have a stand-alone app that converts, mosaicks and scales
GDAL-supported images to a proprietary format. It works well, but we
handle tens of thousands of source files at a time, which can slow the
process down to a crawl.

So we're investigating parallelising this process over several machines,
and I was wondering if anyone has any experience of this that they'd be
willing to share? It's still early days yet, so we're open to all
suggestions.

Regards,
John Donovan - Programmer, Virtalis Ltd.

______________________________________________________________________
This email has been scanned for Virtalis Ltd by the MessageLabs Email
Security System.
______________________________________________________________________
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
This e-mail message and any attached file is the property of the sender and is sent in confidence to the addressee only. The contents are not to be disclosed to anyone other than
the addressee. Unauthorised recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission. If you experience difficulty 
with opening any attachments to this message, or with sending a reply by email, please telephone on + 44-(0)1235 438151 or fax on + 44-(0)1235 438188. 
Any advice contained in this e-mail or any accompanying file attached hereto is for information purposes only. RPS do not take any responsibility for differences between the original 
and the transmission copy or any amendments made thereafter. If the addressee requires RPS to be responsible for the contents of this e-mail, RPS will be pleased to issue a signed 
hard copy of the document upon request.
RPS Planning and Development Limited, company number: 02947164 (England). Registered office: Centurion Court, 85 Milton Park Abingdon Oxfordshire OX14 4RY.
RPS Group Plc web link: <http://www.rpsgroup.com>