[QGIS-Developer] How to speed up rendering
Andreas Neumann
andreas at qgis.org
Tue May 30 07:15:15 PDT 2023
Hi,
See
https://docs.qgis.org/3.28/en/docs/user_manual/working_with_raster/raster_properties.html#pyramids-properties
(for a QGIS based solution)
or https://gdal.org/programs/gdaladdo.html (for a command line or python
version).
Andreas
On Tue, 30 May 2023 at 16:00, afernandez via QGIS-Developer <
qgis-developer at lists.osgeo.org> wrote:
> Hello Hannes,
> Sorry to say that I'm at a loss here. I had not previously heard of
> 'overviews' and am having a hard time finding much information. I couldn't
> find anything in the PyQGIS (v3.28) developer cookbook, which is my main
> reference in this QGIS journey. The found information is at
> https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/gdal/rastermiscellaneous.html
> but didn't exactly clarify how to implement overviews with a script or why
> this would sidestep reading the file several times. I don't quite
> understand this last point either, as the file is large but the size of the
> band (after everything is prepped and before rendering) is relatively
> small, and should fit amply into the available memory.
> Thanks,
> Arturo
>
>
> Johannes Kröger (WhereGroup) via QGIS-Developer wrote:
>
>
> Hi Arturo,
>
> make sure you create overviews on your big rasters or QGIS will re-read
> the whole file all the time to render an appropriate representation in the
> map.
>
> Cheers, Hannes
> Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer:
>
> Hello Martin,
> I'm trying to understand what you're saying but my background is in
> numerical methods where tasks are split into threads for acceleration
> purposes. If a single layer is rendered by a single core (as you write and
> I was observing), how would a 1,000 x 1,000 (or a 5,000 x 5,000) layer be
> handled? Wouldn't it take too long? (However, you want to define 'too
> long').
> As far as more details about the process, the tensor 'var' (n x m x index
> where I have tried different combinations but let's say n x m is 100 x 200)
> of floating point numbers is manipulated according to the following
> pseudo-code:
> # Initial manipulations
> dims = var.dimensions
> shape = var.shape
> driver_name = 'GTIFF'
> driver = gdal.GetDriverByName(driver_name)
> np_dtype = var.dtype
> type_code = gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)
> gdal_ds = driver.Create(_my_path_, cols, rows, 1,
> gdal.GDT_UInt16)
> gdal_ds.SetProjection(_my_projection_)
> gdal_ds.SetGeoTransform(_my_transformation_)
> # Creation of the bands and scaled matrix
> band = gdal_ds.GetRasterBand(1)
> data = var[_chosen_index_]
> data = ma.getdata(data)
> data_scaled = np.interp(data, (data.min(), data.max()), (0,
> 255))
> data_scaled2 = data_scaled.astype(int) # This is to rescale
> into integers so that it can color the layer
> # *** Lines to set up the color palette ***
> # Write the array to band once everything has been rescaled
> band.WriteArray(data_scaled2)
> gdal_ds.FlushCache()
>
> This procedure seems to work well and takes a reasonable amount of time.
> Then, the time is consumed when doing the rendering itself. I'm not
> entirely sure what you mean by 'sub-optimal raster format' or 'thre are
> overviews missing' so I cannot address them specifically.
> Thanks,
> Arturo
>
>
>
>
> Martin Dobias wrote:
>
>
> Hi
>
> On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer <
> qgis-developer at lists.osgeo.org> wrote:
>
>> Hello,
>> One of my project would greatly benefit from a much faster rendering.
>> After some testing, the bulk (>99%) of the time is spent at rendering the
>> layer with QgsRasterLayer(...,...). What I have also noticed is that
>> rendering time does not change with the number of cores, which suggests
>> that the renderer might be using a single thread. I checked the API
>> documentation but couldn't find anything about multithreading or some other
>> procedure to accelerate rendering. I was wondering if anyone has any
>> suggestion or has faced a similar difficulty.
>>
>
> Multi-threaded rendering is done at the granularity of whole map layers,
> so a single layer is rendered by a single CPU core. Therefore if you have
> just one layer that is slow to render, the multi-threading is not going to
> help. It is best to do some testing how you can improve the layer's
> rendering - for example, it could be that you are using some sub-optimal
> raster format, or there are overviews missing, or something else... one
> would need more details.
>
> Regards
> Martin
>
>
> _______________________________________________
> QGIS-Developer mailing listQGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
> --
> Johannes Kröger / GIS-Entwickler/-Berater
>
> ---------------------------------------------
> Aufwind durch Wissen!
> Web-Seminare und Online-Schulungen
> bei der www.foss-academy.com
> ---------------------------------------------
>
> WhereGroup GmbH
> c/o KK03 GmbH
> Lange Reihe 29
> 20099 Hamburg
> Germany
>
> Tel: +49 (0)228 / 90 90 38 - 36
> Fax: +49 (0)228 / 90 90 38 - 11
> johannes.kroeger at wheregroup.comwww.wheregroup.com
> Geschäftsführer:
> Olaf Knopp, Peter Stamm
> Amtsgericht Bonn, HRB 9885
> -------------------------------
>
> Hi Arturo,
>
> make sure you create overviews on your big rasters or QGIS will re-read
> the whole file all the time to render an appropriate representation in the
> map.
>
> Cheers, Hannes
> Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer:
>
> Hello Martin,
> I'm trying to understand what you're saying but my background is in
> numerical methods where tasks are split into threads for acceleration
> purposes. If a single layer is rendered by a single core (as you write and
> I was observing), how would a 1,000 x 1,000 (or a 5,000 x 5,000) layer be
> handled? Wouldn't it take too long? (However, you want to define 'too
> long').
> As far as more details about the process, the tensor 'var' (n x m x index
> where I have tried different combinations but let's say n x m is 100 x 200)
> of floating point numbers is manipulated according to the following
> pseudo-code:
> # Initial manipulations
> dims = var.dimensions
> shape = var.shape
> driver_name = 'GTIFF'
> driver = gdal.GetDriverByName(driver_name)
> np_dtype = var.dtype
> type_code = gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)
> gdal_ds = driver.Create(_my_path_, cols, rows, 1,
> gdal.GDT_UInt16)
> gdal_ds.SetProjection(_my_projection_)
> gdal_ds.SetGeoTransform(_my_transformation_)
> # Creation of the bands and scaled matrix
> band = gdal_ds.GetRasterBand(1)
> data = var[_chosen_index_]
> data = ma.getdata(data)
> data_scaled = np.interp(data, (data.min(), data.max()), (0,
> 255))
> data_scaled2 = data_scaled.astype(int) # This is to rescale
> into integers so that it can color the layer
> # *** Lines to set up the color palette ***
> # Write the array to band once everything has been rescaled
> band.WriteArray(data_scaled2)
> gdal_ds.FlushCache()
>
> This procedure seems to work well and takes a reasonable amount of time.
> Then, the time is consumed when doing the rendering itself. I'm not
> entirely sure what you mean by 'sub-optimal raster format' or 'thre are
> overviews missing' so I cannot address them specifically.
> Thanks,
> Arturo
>
>
>
>
> Martin Dobias wrote:
>
>
> Hi
>
> On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer <
> qgis-developer at lists.osgeo.org> wrote:
>
>> Hello,
>> One of my project would greatly benefit from a much faster rendering.
>> After some testing, the bulk (>99%) of the time is spent at rendering the
>> layer with QgsRasterLayer(...,...). What I have also noticed is that
>> rendering time does not change with the number of cores, which suggests
>> that the renderer might be using a single thread. I checked the API
>> documentation but couldn't find anything about multithreading or some other
>> procedure to accelerate rendering. I was wondering if anyone has any
>> suggestion or has faced a similar difficulty.
>>
>
> Multi-threaded rendering is done at the granularity of whole map layers,
> so a single layer is rendered by a single CPU core. Therefore if you have
> just one layer that is slow to render, the multi-threading is not going to
> help. It is best to do some testing how you can improve the layer's
> rendering - for example, it could be that you are using some sub-optimal
> raster format, or there are overviews missing, or something else... one
> would need more details.
>
> Regards
> Martin
>
>
> _______________________________________________
> QGIS-Developer mailing listQGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
> --
> Johannes Kröger / GIS-Entwickler/-Berater
>
> ---------------------------------------------
> Aufwind durch Wissen!
> Web-Seminare und Online-Schulungen
> bei der www.foss-academy.com
> ---------------------------------------------
>
> WhereGroup GmbH
> c/o KK03 GmbH
> Lange Reihe 29
> 20099 Hamburg
> Germany
>
> Tel: +49 (0)228 / 90 90 38 - 36
> Fax: +49 (0)228 / 90 90 38 - 11
> johannes.kroeger at wheregroup.comwww.wheregroup.com
> Geschäftsführer:
> Olaf Knopp, Peter Stamm
> Amtsgericht Bonn, HRB 9885
> -------------------------------
>
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
--
--
Andreas Neumann
QGIS.ORG board member (treasurer)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20230530/5fee4c6e/attachment-0001.htm>
More information about the QGIS-Developer
mailing list