[gdal-dev] [Qgis-developer] QGIS Multi-threaded Rendering

Fri Dec 13 11:11:10 PST 2013

Le vendredi 13 décembre 2013 08:25:55, Dmitriy Baryshnikov a écrit :
> Hi,
> 
> Why not reading small chunks (i.e. 256 x 256) with RasterIO
> <http://www.gdal.org/classGDALDataset.html#ae077c53268d2272eebed10b891a0574
> 3> from big raster? We'll get less memory usage.

I think QGIS currently does a single RasterIO for the widget. I guess it is a 
matter of simplicity in comparison to issuing small chunks and compositing 
them. And for some formats (such as ECW), it is actually better to issue a 
single RasterIO for the "big" window rather than doing several smaller ones.

> The program can count the percentage using the size of GDALDataset and
> size of chunks and cancel on any chunk.
> 
> By the way, can GDALDataset RasterIO different chunks from separate
> threads?

The dataset and rasterband objects are not supposed to be called from 
concurrent threads, so the generic implementation of 
GDALRasterBand::IRasterIO() in gcore/rasterio.cpp cannot do that. For example 
a IReadBlock() implementation will generally do a fseek() and fread() : if 2 
threads call it, the seek and read can be interleaved in a funny fashion ! 
Rasterband and dataset objects can also have state variables modified by 
IReadBlock() to do some caching, etc...
However driver implementations are perfectly free to do what they want. That's 
the case of the OpenJPEG driver I mentionned in my initial post.

> 
> Best regards,
>      Dmitry
> 
> 12.12.2013 19:28, Even Rouault ?????:
> > Selon Martin Dobias <wonder.sk at gmail.com>:
> >> Hi Even
> >> 
> >> thanks for your thoughts.
> >> 
> >> On Thu, Dec 12, 2013 at 8:30 PM, Even Rouault
> >> 
> >> <even.rouault at mines-paris.org> wrote:
> >>>> - rasters - currently we do not have API to cancel requests for raster
> >>>> blocks. This means that currently we need to wait until the raster
> >>>> block is fully read even when we cancel the rendering job. GDAL has
> >>>> some support for asynchronous requests - anyone has some experience
> >>>> with it?
> >>> 
> >>> Indeed there's an API in GDAL for asynchronous requests :
> >>> http://trac.osgeo.org/gdal/wiki/rfc24_progressive_data_support
> >>> Note that it has a "real" implementation only in the JPIPKAK driver
> >>> (driver that implement the JPIP protocol - JPEG2000 trough the network
> >>> - with the Kakadu library). For other drivers that don't have a
> >>> dedicated implementation, a default implementation will just turn the
> >>> async requests as a sync request (more precisely
> >>> GetNextUpdatedRegion() will just do a RasterIO(), other methods
> >>> mentionned in the RFC are no-op).
> >>> Real cancellation / async request is generally hard to implement since
> >>> the low level libraries used by drivers must support it. The analysis
> >>> must be done on a per-driver basis (determine which drivers are
> >>> important for you, which ones are slow, see if it can be remedied in
> >>> the general synchronous case, otherwise imagine how async request
> >>> could be implemented, ...) We could imagine that the default
> >>> GetNextUpdatedRegion(), instead of issuing a single RasterIO() could
> >>> split it up into several RasterIO() if the request window crosses
> >>> several tiles (in the case of a tiled dataset) to allow cancellation
> >>> between the calls to GetNextUpdatedRegion(). But that's no always a
> >>> beneficial strategy. Drivers can use the fact that a RasterIO() is
> >>> done on a "big" window to optimize things a bit, w.r.t issuing several
> >>> RasterIO() on smaller windows. In GDAL 1.10, I've for
> >> 
> >> example
> >> 
> >>> implemented multi-threaded block decoding in the OpenJPEG driver : if
> >>> the window of a RasterIO() call spans over several tiles and you have
> >>> several cores, then each tile will be decoded in parallel.
> >>> Regarding the current API of asynchronous requests, I feel there's
> >>> perhaps a lack of a way of signaling that data is ready.
> >>> GetNextUpdatedRegion() works on a pulling startegy. Perhaps a callback
> >>> mechanism could be usefull. That topic needs some thinking...
> >> 
> >> So obviously adding async API support for other common formats would
> >> take a lot of time... I was secretly hoping that the remark in rfc24
> >> mentioning only jpipkak driver was just a temporary state of affairs
> >> 
> >> :-)
> >> 
> >> However I realize that actually I do not really need to have truly
> >> asynchronous requests. The requests from QGIS are being made from
> >> within a worker thread, so it is not a problem if GDAL block that
> >> thread. The only thing I need is to be able to tell GDAL to stop
> >> reading data as soon as possible once the rendering gets cancelled.
> >> API-wise, I would like to have a method in GDALDataset, using which I
> >> could register my custom function returning cancellation status of the
> >> request (just true/false). The drivers would be responsible for
> >> calling that function whenever appropriate during RasterIO to check
> >> whether the request has been cancelled or not. It would be probably
> >> much less hassle to implement this functionality in drivers - although
> >> I understand that still many drivers using custom libraries won't be
> >> able to provide that functionality.
> > 
> > Yes, that's a reasonable possibility. The callback could also take
> > as parameter a percentage of completion. That could actually be a
> > GDALProgressFunc (
> > http://trac.osgeo.org/gdal/browser/trunk/gdal/port/cpl_progress.h )
> > 
> >> Regards
> >> Martin
> > 
> > _______________________________________________
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
Geospatial professional services
http://even.rouault.free.fr/services.html