<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Hi</div><div>Simultaneous writers would be a better long term solution as we often improve the raster following initial creation. This improvement may well be a filter run on a sub region ( eg a despeckle) or updating a piece of the dtm with better information or even some manual edits as a last resort. </div><div>I can imagine a Hadoop style map/reduce would fit nicely into your sub window idea. <br><br>Regards<div>pk</div></div><div><br>On 12/01/2013, at 11:16 PM, "Even Rouault" <<a href="mailto:even.rouault@mines-paris.org">even.rouault@mines-paris.org</a>> wrote:<br><br></div><blockquote type="cite"><div>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="MS Exchange Server version 6.5.7654.1">
<title>Re: [gdal-dev] does gdal support multiple simultaneous writers to raster</title>
<!-- Converted from text/plain format -->
<p><font size="2">Le samedi 12 janvier 2013 02:38:55, Kennedy, Paul a écrit :<br>
> Hi,<br>
> Yes, we are pretty sure we will see a significant benefit. The processing<br>
> algorithms are CPU bound not io bound. Our digital terrain model<br>
> interpolations often run for many hours ( we do them overnight) but the<br>
> underlying file is only a few gigabytes.<br>
<br>
OK, my understanding is that you don't really need writers to write<br>
simultaneously. You need to compute tiles or subwindows of the whole raster in<br>
parallel, but the writing itself of the result of that computation could be<br>
well done in a serialized way.<br>
<br>
That's a bit what is done with gdalwarp -wo NUM_THREADS=xxxx . Having<br>
parallelized I/O could perhaps give some extra performance when you have so<br>
many threads that the time spent in I/O becomes of the same order of magnitude<br>
than the time spent in computing, but at the expense of probably a significant<br>
complexity in GDAL core and drivers.<br>
<br>
> If we split them into multiple<br>
> files of tiles and run each on a dedicated process the whole thing is<br>
> quicker, but this is messy and results in a stitching error.<br>
><br>
> Another example is gdalwarp. It takes quite some time with a large data set<br>
> and would be. A good candidate for parallelisation, as would gdaladdo.<br>
><br>
> I believe slower cores but more of them in pcs are the future. My pc has 8<br>
> but they rarely get used to their potential.<br>
><br>
> I am certain there are some challenges here, that's why it is interesting;)<br>
><br>
> Regards<br>
> pk<br>
><br>
> On 11/01/2013, at 6:54 PM, "Even Rouault" <<a href="mailto:even.rouault@mines-paris.org">even.rouault@mines-paris.org</a>><br>
wrote:<br>
> > Hi,<br>
> ><br>
> > This is an intersting topic, with many "intersecting" issues to deal with<br>
> > at different levels.<br>
> ><br>
> > First, are you confident that in the use cases you imagine that I/O<br>
> > access won't be the limiting factor, in which case serialization of I/O<br>
> > could be acceptable and this would just require an API with a dataset<br>
> > level mutex.<br>
> ><br>
> > There are several places where parallel write should be addressed :<br>
> > - The GDAL core mechanisms that deal with the block cache<br>
> > - Each GDAL driver where parallel write would be supported. I guess that<br>
> > GDAL drivers should advertize a specific capability<br>
> > - The low-level library used by the driver. In the case of GDAL, libtiff<br>
> ><br>
> > And finally, as Frank underlined, there are intrinsic limitations due to<br>
> > the format itself. For a compressed TIFF, at some point, you have to<br>
> > serialize the writing of the tile, because you cannot kown in advance<br>
> > the size of the compressed data, or at least have some coordination of<br>
> > the writers so that a "next offset available" is properly synchronized<br>
> > between them. The compression itself could be serialized.<br>
> ><br>
> > I'm not sure however if what Jan mentionned, different process, writing<br>
> > the same dataset is doable.<br>
</font>
</p>
</div></blockquote></body></html>