[gdal-dev] Distributed processing

[Trent M Hare]

huh - processing using Make - clever.

We are researching several different approaches for this also but I 
imagine we will use Conductor, created by (UofA). Another of our 
commercial systems uses Condor (open source - 
http://www.cs.wisc.edu/condor/ ) .

Currently the UofA Conductor HiRISE pipeline uses our ISIS image 
processing software which is not yet multi-threaded. Thus this system 
should also work well for GDAL processing. They also have another tool 
that monitors Conductor called Maestro. Don't jump into this lightly as I 
imagine it will be tricky to setup (it also requires a database - MySQL). 
http://pirlwww.lpl.arizona.edu/software/Conductor.shtml BTW, they are a 
completely *nix house (Solaris, Mac OS-X, FreeBSD and Linux).

here is an abstract (by the author B. Castalia)
http://www.lsstmail.org/meet_coll/ongoing/Software/

Regards,
Trent



From:
Fabrice LALLAURET - PSV <fabrice.lallauret at external.thalesgroup.com>
To:
gdal-dev at lists.osgeo.org
Date:
06/16/2009 08:32 AM
Subject:
Re: [gdal-dev] Distributed processing
Sent by:
gdal-dev-bounces at lists.osgeo.org



John Donovan a écrit :
> Hi,
> We currently have a stand-alone app that converts, mosaicks and scales
> GDAL-supported images to a proprietary format. It works well, but we
> handle tens of thousands of source files at a time, which can slow the
> process down to a crawl.
>
> So we're investigating parallelising this process over several machines,
> and I was wondering if anyone has any experience of this that they'd be
> willing to share? It's still early days yet, so we're open to all
> suggestions.
>
> Regards,
> John Donovan - Programmer, Virtalis Ltd.
>
> ______________________________________________________________________
> This email has been scanned for Virtalis Ltd by the MessageLabs Email 
Security System. 
______________________________________________________________________
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
Personally, I create Makefile and use distmake for the distribution 
purpose.

Regards
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20090617/2b50e93c/attachment.html