[gdal-dev] multiple access to a file

[Frank Warmerdam]

Jose,

I'm sorry, but I don't think this will work trivially.

If you write a small part of an image, GDAL is likely to read a larger
area, update that and write it back.  If other programs happen to try and
update the same larger area at the same time some data is likely to be
lost.

If you are careful to write on tile/block boundaries, and use an output
file that preinitializes everything then it might work but it is likely to
be somewhat fragile.

Best regards,
Frank



On Mon, May 20, 2013 at 3:46 PM, Jose Gomez-Dans <jgomezdans at gmail.com>wrote:

> Hi,
> I have a rather demanding processing scheme (read data from files, and
> process pixel per pixel). It's trivial to parallelise (just do pixels in
> different machines/cores), and I intend to run this on several machines on
> our cluster.
>
> I'm unsure whether I can use GDAL (python bindings) like this. My idea
> would be to first create the output files. After this, the scene is split
> into several windows, and different works start processing the different
> windows. Once a worker is done, it gets a handle to the created file and
> writes a chunk to it, closes the handle an exits.
>
> Questions are: does this "create and fill" make sense? Do I need to lock
> the output files, or can I just write to the file handles (the windows
> don't overlap)? Is this format dependent? Am I missing something? I'd
> rather avoid creating lots of intermediate files to then stitch up.
>
> Thanks!
> Jose
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>



-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam,
warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Software Developer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20130520/4f64b7a7/attachment.html>