[gdal-dev] does gdal support multiple simultaneous writers to raster

[Kennedy, Paul]

thanks for the prompt response.  It would have been nice to use the GDAL
API in a multi-process (parallelised) world if feasible.  We are
increasingly seeing the need to move to multi-core (parallel) access so
we can spread the processing load across multiple CPU's, but we do like
to keep our data in a single, cohesive raster file (often with billions
of pixels) so we can avoid stitching errors at the seams.  
The use of pyramids means the visualisation of huge rasters is not an
issue but single write access is starting to be a problem.
We are thinking of a 'master' process running which can server out tiles
to different processes and keep track of them to avoid write collision.
Almost a Tile Map Service with write capabilities.
would you think this is feasible with GDAL as the backend service?



-----Original Message-----
From: fwarmerdam at gmail.com [mailto:fwarmerdam at gmail.com] On Behalf Of
Frank Warmerdam
Sent: Friday, 11 January 2013 9:33 AM
To: Kennedy, Paul
Cc: gdal dev
Subject: Re: [gdal-dev] does gdal support multiple simultaneous writers
to raster

Paul,

Generally speaking - no - GDAL does not now support multi-threaded
writing.  Part of this is due to the way stuff gets write-flushed out of
the block cache and partly it is just that none of GDAL supports
multiple threading on a single dataset.

This statement is based on the assumption it is all in one process.

If you want to run different processes with the output to a pre-existing
"fixed location" format like uncompressed GeoTIFF with tiles that is all
preinitialized to some real value then you could certainly use one
output file for many writers as long as case is taken to not have
multiple processes write to the same tile.  But GDAL isn't doing
anything to make this easy.

Best regards,
Frank

On Thu, Jan 10, 2013 at 4:12 PM, Kennedy, Paul <P.Kennedy at fugro.com.au>
wrote:
> I am looking to parallelize a long duration gridding process, and was 
> wondering if GDAL can support multiple simultaneous writers from
processess?
>
>
>
> I am not sure if any of the underlying raster formats support this?
>
>
>
> thanks in advance.
>
>
>
> regards
>
> pk
>
>
>
>
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev



-- 
---------------------------------------+--------------------------------
---------------------------------------+------
I set the clouds in motion - turn up   | Frank Warmerdam,
warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Software Developer