[gdal-dev] RE: progressive rendering

[Adam Nowacki]

Tamas Szekeres wrote:
> 2008/8/29 Adam Nowacki <nowak at xpam.de>:
>> In my proposal the user could:
>> 1) LockBuffer()
>> 2) display the current snapshot
>> 3) UnlockBuffer()
>> 4) sleep for 100ms ignoring the update messages (but
>> NextAsyncRasterIOMessage() still has to be called to get the GARM_COMPLETE
>> message)
> It seems you consider a multi threaded driver behaviour by default,
> Why should the driver tamper the contents the buffer of the user in
> the meantime at all. Wouldn't it be sufficient to update the buffer
> contents in the NextAsyncRasterIOMessage call? I've mentioned that I'm
> not totally sure about the necessity of using multiple threads, but
> the driver could eventually collect the data in an internal temporary
> buffer and copy the contents to the user buffer upon the subsequent
> NextAsyncRasterIOMessage. However for collecting the data in an
> internal buffer I don't think we should duplicate what the TCP/IP
> driver is actually doing at the OS level.

Im trying to design a interface with lowest overhead possible. Driver 
doesnt have to keep its own buffer and later copy, received data can be 
directly dumped into user buffer.

>> The RasterIO function is not renamed. A new function is added
>> (AsyncRasterIO) with different behavior than the current RasterIO function.
>> RasterIO function doesnt change, a full backwards compatibility.
>>
> 
> I see no problem if the original RasterIO behaves differently in some
> cases controlled explicitly by the user. By specifying a global
> setting that RasterIO will operate in asynchronous mode at dataset
> level would be enough (like using the dataset creation option or a new
> flag in the dataset). I guess the programmer should take care of the
> fact how the dataset have been created before. We should indeed use
> the synchronous mode as the default setting.

The RasterIO function has been there for years. Changing its behavior by 
some hidden variable will definitely make it easier to use (or read) it 
the wrong way. Im trying to protect programmer's foots :)

>>> Related to the statements above - in my opinion - only the behaviour
>>> of the existing RasterIO should be altered, which would provide an
>>> alternative option (in a Win32 overlapped IO fashion), according to
>>> the following example:
>>>
>>> 1. During the dataset creation the asynchronous behaviour of the
>>> driver for this dataset could be specified as a new dataset creation
>>> option (like RASTERIO_MODE=ASYNC)
>> With my proposal you can mix normal blocking RasterIO calls with
>> AsyncRasterIO calls.
> 
> Hmmm. I don't see the related use case when the the RasterIO should be
> used in synchronous and asyncronous mode with the same dataset in
> parallel. However the user have the freedom to create 2 datasets
> specify different IO mode and use them simultaneously.

Being able to mix both blocking and async RasterIO calls sure would help 
to 'progressively' upgrade code from RasterIO to AsyncRasterIO.
Opening 2 datasets would be a real bad thing: no shared cache.

>>> 2. The user would use the existing RasterIO method to initiate the
>>> operation as well as to fetch the next available data. The RasterIO
>>> would return immediately if the buffer contains a proper set of the
>>> intermediary data. In this case RasterIO would return a special error
>>> code (IO_PENDING) to denote that more data will be available and the
>>> user will have to initiate a subsequent call to RasterIO. (If we don't
>>> like to introduce new error codes we could also use a separate
>>> function like GetAsyncResult for this purpose).
>>>
>>> 3. The RasterIO  (or GetAsyncResult) would return no error if the
>>> operation have been finished and no more new update will be available
>>> in the buffer.
>>>
>>> 4. We could introduce a separate function like CancelIO to allow the
>>> user to cancel the pending IO operation at any time.
>> While I like the simplicity there are a few 'problems'. Identifying
>> subsequent calls to RasterIO belonging to the same operation. Would it be
>> the buffer pointer, together with the offset and size maybe? Possibly a lot
>> of variables that have to be remembered by both the driver and user side.
>> Searching a list of all open async raster io operations ? Would the buffer
>> be updated with data received since last RasterIO call or a current snapshot
>> ? Each open async raster io would also require its own data buffer, later
>> copied in whole (or only updated regions) into user buffer.
>>
> 
> I consider the IO context should be stored in the dataset as member
> variables by the driver. I don't see a pressing reason to support
> multiple raster IO operations at the same time with the same dataset.
> I think a similar operation could safely be utilized by creating 2 or
> more datasets and invoke their rasterIO simultaneously.
> In case if the user specifies a different buffer or image section in
> RasterIO then the driver would return with an error or gracefully
> initiate a new sequence to the server implicitly.

Consider a simple image browsing app with a overview box in a corner. 
Having 2 simultaneously running asynchronous raster io operations on the 
same dataset would allow the driver to optimize requests to the server 
possibly even eliminating separate requests for the overview. Or after 
zooming in the previously cached data for main view could be used to 
service the overview view.