[gdal-dev] [Qgis-developer] QGIS Multi-threaded Rendering

[Martin Dobias]

Hi Even

thanks for your thoughts.

On Thu, Dec 12, 2013 at 8:30 PM, Even Rouault
<even.rouault at mines-paris.org> wrote:
>> - rasters - currently we do not have API to cancel requests for raster
>> blocks. This means that currently we need to wait until the raster
>> block is fully read even when we cancel the rendering job. GDAL has
>> some support for asynchronous requests - anyone has some experience
>> with it?
>
> Indeed there's an API in GDAL for asynchronous requests :
> http://trac.osgeo.org/gdal/wiki/rfc24_progressive_data_support
> Note that it has a "real" implementation only in the JPIPKAK driver
> (driver that implement the JPIP protocol - JPEG2000 trough the network -
> with the Kakadu library). For other drivers that don't have a dedicated
> implementation, a default implementation will just turn the async requests
> as a sync request (more precisely GetNextUpdatedRegion() will just do
> a RasterIO(), other methods mentionned in the RFC are no-op).
> Real cancellation / async request is generally hard to implement since the
> low level libraries used by drivers must support it. The analysis must be
> done on a per-driver basis (determine which drivers are important for you,
> which ones are slow, see if it can be remedied in the general synchronous
> case, otherwise imagine how async request could be implemented, ...)
> We could imagine that the default GetNextUpdatedRegion(), instead of
> issuing a single RasterIO() could split it up into several RasterIO()
> if the request window crosses several tiles (in the case of a tiled
> dataset) to allow cancellation between the calls to GetNextUpdatedRegion().
> But that's no always a beneficial strategy. Drivers can use the fact that
> a RasterIO() is done on a "big" window to optimize things a bit, w.r.t
> issuing several RasterIO() on smaller windows. In GDAL 1.10, I've for example
> implemented multi-threaded block decoding in the OpenJPEG driver : if the
> window of a RasterIO() call spans over several tiles and you have several
> cores, then each tile will be decoded in parallel.
> Regarding the current API of asynchronous requests, I feel there's perhaps
> a lack of a way of signaling that data is ready. GetNextUpdatedRegion()
> works on a pulling startegy. Perhaps a callback mechanism could be usefull.
> That topic needs some thinking...

So obviously adding async API support for other common formats would
take a lot of time... I was secretly hoping that the remark in rfc24
mentioning only jpipkak driver was just a temporary state of affairs
:-)

However I realize that actually I do not really need to have truly
asynchronous requests. The requests from QGIS are being made from
within a worker thread, so it is not a problem if GDAL block that
thread. The only thing I need is to be able to tell GDAL to stop
reading data as soon as possible once the rendering gets cancelled.
API-wise, I would like to have a method in GDALDataset, using which I
could register my custom function returning cancellation status of the
request (just true/false). The drivers would be responsible for
calling that function whenever appropriate during RasterIO to check
whether the request has been cancelled or not. It would be probably
much less hassle to implement this functionality in drivers - although
I understand that still many drivers using custom libraries won't be
able to provide that functionality.

Regards
Martin