[gdal-dev] RE: progressive rendering

[Even Rouault]

Tamas,

The only thing that was in Adam's proposal and that I don't find in yours is 
that, when the RasterIO in async mode returns something valid, the user has 
no way to know which part of the buffer has been updated by the driver. Thus, 
it introduces some extra cost for progressive image display since you then 
need to resend the whole buffer to your display window (before the first 
call, you must initialize the buffer to some default value too).

Question : do we expect that it is going to ruin performance and/or interest 
of progressive rendering ?

I agree that the driver implementation may not be multi-threaded. But it's 
just a choice of implementation from the driver writer. The LockBuffer() / 
UnlockBuffer() API doesn't necessarily imply that the driver must use an 
auxiliary thread. An empty implementation would do for a non multithreaded 
driver. It just makes it possible for the driver to update-in-place the user 
buffer without the need for an internal buffer. 
But... I agree that the need for the LockBuffer()/UnlockBuffer() API is  
arguable since you need the 3 following conditions to be met for it to be 
really usefull :
- a multi-threaded driver
- AND it does not use a temporary buffer
- AND the user doesn't lock for too long an area of the buffer that is going 
to be updated by the driver, otherwise the thread of the driver will be 
stalled by the mutex on the buffer.
All other cases do not need that API.

We could also imagine that the API exists but that the documentation of the 
driver explicitely says whether it is necessary to explictely Lock() / 
Unlock() the buffer, depending on how it is implemented.

Hum, I'm thinking that --enable-threading must be explicitely passed to 
configure. So it means that a multi-threaded driver should be disabled if 
threading is not enabled. Maybe not a big deal but until now, no GDAL driver 
had this requirement.


Le Friday 29 August 2008 21:01:22 Tamas Szekeres, vous avez écrit :
> 2008/8/29 Adam Nowacki <nowak at xpam.de>:
> > In my proposal the user could:
> > 1) LockBuffer()
> > 2) display the current snapshot
> > 3) UnlockBuffer()
> > 4) sleep for 100ms ignoring the update messages (but
> > NextAsyncRasterIOMessage() still has to be called to get the
> > GARM_COMPLETE message)
>
> Adam,
>
> It seems you consider a multi threaded driver behaviour by default,
> Why should the driver tamper the contents the buffer of the user in
> the meantime at all. Wouldn't it be sufficient to update the buffer
> contents in the NextAsyncRasterIOMessage call? I've mentioned that I'm
> not totally sure about the necessity of using multiple threads, but
> the driver could eventually collect the data in an internal temporary
> buffer and copy the contents to the user buffer upon the subsequent
> NextAsyncRasterIOMessage. However for collecting the data in an
> internal buffer I don't think we should duplicate what the TCP/IP
> driver is actually doing at the OS level.
>
> > The RasterIO function is not renamed. A new function is added
> > (AsyncRasterIO) with different behavior than the current RasterIO
> > function. RasterIO function doesnt change, a full backwards
> > compatibility.
>
> I see no problem if the original RasterIO behaves differently in some
> cases controlled explicitly by the user. By specifying a global
> setting that RasterIO will operate in asynchronous mode at dataset
> level would be enough (like using the dataset creation option or a new
> flag in the dataset). I guess the programmer should take care of the
> fact how the dataset have been created before. We should indeed use
> the synchronous mode as the default setting.
>
> >> Related to the statements above - in my opinion - only the behaviour
> >> of the existing RasterIO should be altered, which would provide an
> >> alternative option (in a Win32 overlapped IO fashion), according to
> >> the following example:
> >>
> >> 1. During the dataset creation the asynchronous behaviour of the
> >> driver for this dataset could be specified as a new dataset creation
> >> option (like RASTERIO_MODE=ASYNC)
> >
> > With my proposal you can mix normal blocking RasterIO calls with
> > AsyncRasterIO calls.
>
> Hmmm. I don't see the related use case when the the RasterIO should be
> used in synchronous and asyncronous mode with the same dataset in
> parallel. However the user have the freedom to create 2 datasets
> specify different IO mode and use them simultaneously.
>
> >> 2. The user would use the existing RasterIO method to initiate the
> >> operation as well as to fetch the next available data. The RasterIO
> >> would return immediately if the buffer contains a proper set of the
> >> intermediary data. In this case RasterIO would return a special error
> >> code (IO_PENDING) to denote that more data will be available and the
> >> user will have to initiate a subsequent call to RasterIO. (If we don't
> >> like to introduce new error codes we could also use a separate
> >> function like GetAsyncResult for this purpose).
> >>
> >> 3. The RasterIO  (or GetAsyncResult) would return no error if the
> >> operation have been finished and no more new update will be available
> >> in the buffer.
> >>
> >> 4. We could introduce a separate function like CancelIO to allow the
> >> user to cancel the pending IO operation at any time.
> >
> > While I like the simplicity there are a few 'problems'. Identifying
> > subsequent calls to RasterIO belonging to the same operation. Would it be
> > the buffer pointer, together with the offset and size maybe? Possibly a
> > lot of variables that have to be remembered by both the driver and user
> > side. Searching a list of all open async raster io operations ? Would the
> > buffer be updated with data received since last RasterIO call or a
> > current snapshot ? Each open async raster io would also require its own
> > data buffer, later copied in whole (or only updated regions) into user
> > buffer.
>
> I consider the IO context should be stored in the dataset as member
> variables by the driver. I don't see a pressing reason to support
> multiple raster IO operations at the same time with the same dataset.
> I think a similar operation could safely be utilized by creating 2 or
> more datasets and invoke their rasterIO simultaneously.
> In case if the user specifies a different buffer or image section in
> RasterIO then the driver would return with an error or gracefully
> initiate a new sequence to the server implicitly.
>
>
> Best regards,
>
> Tamas
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev