[Gdal-dev] running gdal_translate, gdaladdo, and gdaltindex simultaneously

[Frank Warmerdam]

John Mitchell wrote:
> Hi,
> 
> I have set up 4 .bat files each which run the following over and over again:
> file1.bat                        file2.bat                          
> file3.bat                           file4.bat
> gdal_translate               gdal_translate                
> gdal_translate                  gdal_translate
> gdaladdo                      gdaladdo                       
> gdaladdo                          gdaladdo
> gdaltindex                    gdaltindex                      
> gdaltindex                        gdaltindex
> 
> What I would like to do is run all 4 .bat files at the same time so as 
> to save time, but I am concerned that their may be a conflict since all 
> 4 files will be running gdaltindex to the same .shp file.  What if say 
> file1.bat and file3.bat happen to call gdaltindex at the same time will 
> they both fail to update the .shp file; will only one update the .shp or 
> will both of them successfully update the .shp file.  Is their any 
> parameter that I should add when running gdal_translate, gdaladdo or 
> gdaltindex to make it more efficient or stable?

John,

This is indeed unsafe and could result in corruption of the index.  I
would suggest doing the gdal_translate, and gdaladdo "in parallel", but
doing the index creation in a single run afterwards. Index creation is
very fast compared to translation and adding overviews.

Note, if you are seeing surprisingly slow performance in gdal_translate
or gdaladdo it may be suffering from "cache thrashing", in which case
setting the GDAL_CACHEMAX environment variable to a reasonable size may
help.

eg
set GDAL_CACHEMAX=200

(for 200MB).

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org