[gdal-dev] Distributed processing

[Klokan Petr Přidal]

Hello John,

I developed the Parallel GDAL2Tiles application (info at
http://www.maptiler.com/) and I am using the ParallelPython module for
GDAL automation on a computer cluster - have a look at
http://www.parallelpython.com/.

It runs automatically on SMP and multi-core computers and it is simple
to setup in a cluster mode as well.
Parallel programming for simultaneously running python processes with
this module is quite straight forward. I can just recommend that - it
runs well in several production environments, including Amazon EC2
cluster (I made some patches for that).

I was also testing the light-weight open-source implementation of
Google's MapReduce from Nokia research lab available at
http://discoproject.org/. This is also usable for GDAL processing of
huge datasets.

I would like to present the MapTiler at FOSS4G in Sydney in case I get
a time slot...
BTW You can vote for my presentation at:
http://dev.extranet.lisasoft.com/foss4g/itemrate/list/all/gdal/

Best regards,

Klokan Petr Pridal

On Wed, Jun 17, 2009 at 8:00 PM, Trent M Hare<thare at usgs.gov> wrote:
>
> huh - processing using Make - clever.
>
> We are researching several different approaches for this also but I imagine
> we will use Conductor, created by (UofA). Another of our commercial systems
> uses Condor (open source - http://www.cs.wisc.edu/condor/ ) .
>
> Currently the UofA Conductor HiRISE pipeline uses our ISIS image processing
> software which is not yet multi-threaded. Thus this system should also work
> well for GDAL processing. They also have another tool that monitors
> Conductor called Maestro. Don't jump into this lightly as I imagine it will
> be tricky to setup (it also requires a database - MySQL).
> http://pirlwww.lpl.arizona.edu/software/Conductor.shtml BTW, they are a
> completely *nix house (Solaris, Mac OS-X, FreeBSD and Linux).
>
> here is an abstract (by the author B. Castalia)
> http://www.lsstmail.org/meet_coll/ongoing/Software/
>
> Regards,
> Trent
>
>
> From: Fabrice LALLAURET - PSV <fabrice.lallauret at external.thalesgroup.com>
> To:
> gdal-dev at lists.osgeo.org
> Date: 06/16/2009 08:32 AM
> Subject: Re: [gdal-dev] Distributed processing
> Sent by:
> gdal-dev-bounces at lists.osgeo.org
> ________________________________
>
>
> John Donovan a écrit :
>> Hi,
>> We currently have a stand-alone app that converts, mosaicks and scales
>> GDAL-supported images to a proprietary format. It works well, but we
>> handle tens of thousands of source files at a time, which can slow the
>> process down to a crawl.
>>
>> So we're investigating parallelising this process over several machines,
>> and I was wondering if anyone has any experience of this that they'd be
>> willing to share? It's still early days yet, so we're open to all
>> suggestions.
>>
>> Regards,
>> John Donovan - Programmer, Virtalis Ltd.
>>
>> ______________________________________________________________________
>> This email has been scanned for Virtalis Ltd by the MessageLabs Email
>> Security System.
>> ______________________________________________________________________
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> Personally, I create Makefile and use distmake for the distribution purpose.
>
> Regards
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>



-- 
http://blog.klokan.cz/
http://www.maptiler.org/
http://www.oldmapsonline.org/