[gdal-dev] does gdal support multiple simultaneous writers to raster

[Jan Hartmann]

I would like to make a further suggestion: what about an option for 
gdalwarp to create *external* tiles, instead of one big output file? 
That's what I am doing now: just create one big georeferenced raster, 
and split it up in tiles, mostly 2000*2000. Accessing those tiles with a 
tileindex is really fast, although I was surprised to see how fast an 
internallly tiled geotiff already is. My impression is that that would 
not be very difficult to implement, and for big datasets tiles are a 
must anyway. I'm not sure how this would interact with the already 
existing tiling applications, though.

Jan

On 01/11/2013 11:53 AM, Even Rouault wrote:
> Hi,
>
> This is an intersting topic, with many "intersecting" issues to deal with at
> different levels.
>
> First, are you confident that in the use cases you imagine that I/O access won't
> be the limiting factor, in which case serialization of I/O could be acceptable
> and this would just require an API with a dataset level mutex.
>
> There are several places where parallel write should be addressed :
> - The GDAL core mechanisms that deal with the block cache
> - Each GDAL driver where parallel write would be supported. I guess that GDAL
> drivers should advertize a specific capability
> - The low-level library used by the driver. In the case of GDAL, libtiff
>
> And finally, as Frank underlined, there are intrinsic limitations due to the
> format itself. For a compressed TIFF, at some point, you have to serialize the
> writing of the tile, because you cannot kown in advance the size of the
> compressed data, or at least have some coordination of the writers so that a
> "next offset available" is properly synchronized between them. The compression
> itself could be serialized.
>
> I'm not sure however if what Jan mentionned, different process, writing the same
> dataset is doable.
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20130111/dc067949/attachment-0001.html>