[QGIS-Developer] How to speed up rendering
Ian Turton
ijturton at gmail.com
Wed May 31 01:17:11 PDT 2023
A good introduction to fast rasters is Paul Ramsey's GeoTiff compression
for dummies blog post -
https://blog.cleverelephant.ca/2015/02/geotiff-compression-for-dummies.html
it takes you through all the steps needed using gdal.
Ian
On Wed, 31 May 2023 at 08:47, Johannes Kröger (WhereGroup) via
QGIS-Developer <qgis-developer at lists.osgeo.org> wrote:
> Sorry Arturo,
>
> I totally missed that Martin had already recommended it and that you said
> you didn't know about it. I didn't mean to come across pushy or anything :))
>
> Andreas' links are great to see how to create them. Here is a nice post
> that explains the concept:
> https://developers.planet.com/docs/planetschool/an-introduction-to-cloud-optimized-geotiffs-cogs-part-1-overview/
>
> As you can see there you probably also want to created "tiled" TIFFs,
> where the data internal to the TIFF is arranged in a way that fits spatial
> queries better.
>
> If you can, simply specify that you want "cloud-optimized geotiffs", those
> have all these optimisations and are also "typical spatial analysis
> optimized geotiffs".
>
> Cheers, Hannes
> Am 30.05.23 um 14:37 schrieb afernandez via QGIS-Developer:
>
> Hello Hannes,
> Sorry to say that I'm at a loss here. I had not previously heard of
> 'overviews' and am having a hard time finding much information. I couldn't
> find anything in the PyQGIS (v3.28) developer cookbook, which is my main
> reference in this QGIS journey. The found information is at
> https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/gdal/rastermiscellaneous.html
> but didn't exactly clarify how to implement overviews with a script or why
> this would sidestep reading the file several times. I don't quite
> understand this last point either, as the file is large but the size of the
> band (after everything is prepped and before rendering) is relatively
> small, and should fit amply into the available memory.
> Thanks,
> Arturo
>
>
> Johannes Kröger (WhereGroup) via QGIS-Developer wrote:
>
>
> Hi Arturo,
>
> make sure you create overviews on your big rasters or QGIS will re-read
> the whole file all the time to render an appropriate representation in the
> map.
>
> Cheers, Hannes
> Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer:
>
> Hello Martin,
> I'm trying to understand what you're saying but my background is in
> numerical methods where tasks are split into threads for acceleration
> purposes. If a single layer is rendered by a single core (as you write and
> I was observing), how would a 1,000 x 1,000 (or a 5,000 x 5,000) layer be
> handled? Wouldn't it take too long? (However, you want to define 'too
> long').
> As far as more details about the process, the tensor 'var' (n x m x index
> where I have tried different combinations but let's say n x m is 100 x 200)
> of floating point numbers is manipulated according to the following
> pseudo-code:
> # Initial manipulations
> dims = var.dimensions
> shape = var.shape
> driver_name = 'GTIFF'
> driver = gdal.GetDriverByName(driver_name)
> np_dtype = var.dtype
> type_code = gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)
> gdal_ds = driver.Create(_my_path_, cols, rows, 1,
> gdal.GDT_UInt16)
> gdal_ds.SetProjection(_my_projection_)
> gdal_ds.SetGeoTransform(_my_transformation_)
> # Creation of the bands and scaled matrix
> band = gdal_ds.GetRasterBand(1)
> data = var[_chosen_index_]
> data = ma.getdata(data)
> data_scaled = np.interp(data, (data.min(), data.max()), (0,
> 255))
> data_scaled2 = data_scaled.astype(int) # This is to rescale
> into integers so that it can color the layer
> # *** Lines to set up the color palette ***
> # Write the array to band once everything has been rescaled
> band.WriteArray(data_scaled2)
> gdal_ds.FlushCache()
>
> This procedure seems to work well and takes a reasonable amount of time.
> Then, the time is consumed when doing the rendering itself. I'm not
> entirely sure what you mean by 'sub-optimal raster format' or 'thre are
> overviews missing' so I cannot address them specifically.
> Thanks,
> Arturo
>
>
>
>
> Martin Dobias wrote:
>
>
> Hi
>
> On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer <
> qgis-developer at lists.osgeo.org> wrote:
>
>> Hello,
>> One of my project would greatly benefit from a much faster rendering.
>> After some testing, the bulk (>99%) of the time is spent at rendering the
>> layer with QgsRasterLayer(...,...). What I have also noticed is that
>> rendering time does not change with the number of cores, which suggests
>> that the renderer might be using a single thread. I checked the API
>> documentation but couldn't find anything about multithreading or some other
>> procedure to accelerate rendering. I was wondering if anyone has any
>> suggestion or has faced a similar difficulty.
>>
>
> Multi-threaded rendering is done at the granularity of whole map layers,
> so a single layer is rendered by a single CPU core. Therefore if you have
> just one layer that is slow to render, the multi-threading is not going to
> help. It is best to do some testing how you can improve the layer's
> rendering - for example, it could be that you are using some sub-optimal
> raster format, or there are overviews missing, or something else... one
> would need more details.
>
> Regards
> Martin
>
>
> _______________________________________________
> QGIS-Developer mailing listQGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
> --
> Johannes Kröger / GIS-Entwickler/-Berater
>
> ---------------------------------------------
> Aufwind durch Wissen!
> Web-Seminare und Online-Schulungen
> bei der www.foss-academy.com
> ---------------------------------------------
>
> WhereGroup GmbH
> c/o KK03 GmbH
> Lange Reihe 29
> 20099 Hamburg
> Germany
>
> Tel: +49 (0)228 / 90 90 38 - 36
> Fax: +49 (0)228 / 90 90 38 - 11
> johannes.kroeger at wheregroup.comwww.wheregroup.com
> Geschäftsführer:
> Olaf Knopp, Peter Stamm
> Amtsgericht Bonn, HRB 9885
> -------------------------------
>
> Hi Arturo,
>
> make sure you create overviews on your big rasters or QGIS will re-read
> the whole file all the time to render an appropriate representation in the
> map.
>
> Cheers, Hannes
> Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer:
>
> Hello Martin,
> I'm trying to understand what you're saying but my background is in
> numerical methods where tasks are split into threads for acceleration
> purposes. If a single layer is rendered by a single core (as you write and
> I was observing), how would a 1,000 x 1,000 (or a 5,000 x 5,000) layer be
> handled? Wouldn't it take too long? (However, you want to define 'too
> long').
> As far as more details about the process, the tensor 'var' (n x m x index
> where I have tried different combinations but let's say n x m is 100 x 200)
> of floating point numbers is manipulated according to the following
> pseudo-code:
> # Initial manipulations
> dims = var.dimensions
> shape = var.shape
> driver_name = 'GTIFF'
> driver = gdal.GetDriverByName(driver_name)
> np_dtype = var.dtype
> type_code = gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)
> gdal_ds = driver.Create(_my_path_, cols, rows, 1,
> gdal.GDT_UInt16)
> gdal_ds.SetProjection(_my_projection_)
> gdal_ds.SetGeoTransform(_my_transformation_)
> # Creation of the bands and scaled matrix
> band = gdal_ds.GetRasterBand(1)
> data = var[_chosen_index_]
> data = ma.getdata(data)
> data_scaled = np.interp(data, (data.min(), data.max()), (0,
> 255))
> data_scaled2 = data_scaled.astype(int) # This is to rescale
> into integers so that it can color the layer
> # *** Lines to set up the color palette ***
> # Write the array to band once everything has been rescaled
> band.WriteArray(data_scaled2)
> gdal_ds.FlushCache()
>
> This procedure seems to work well and takes a reasonable amount of time.
> Then, the time is consumed when doing the rendering itself. I'm not
> entirely sure what you mean by 'sub-optimal raster format' or 'thre are
> overviews missing' so I cannot address them specifically.
> Thanks,
> Arturo
>
>
>
>
> Martin Dobias wrote:
>
>
> Hi
>
> On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer <
> qgis-developer at lists.osgeo.org> wrote:
>
>> Hello,
>> One of my project would greatly benefit from a much faster rendering.
>> After some testing, the bulk (>99%) of the time is spent at rendering the
>> layer with QgsRasterLayer(...,...). What I have also noticed is that
>> rendering time does not change with the number of cores, which suggests
>> that the renderer might be using a single thread. I checked the API
>> documentation but couldn't find anything about multithreading or some other
>> procedure to accelerate rendering. I was wondering if anyone has any
>> suggestion or has faced a similar difficulty.
>>
>
> Multi-threaded rendering is done at the granularity of whole map layers,
> so a single layer is rendered by a single CPU core. Therefore if you have
> just one layer that is slow to render, the multi-threading is not going to
> help. It is best to do some testing how you can improve the layer's
> rendering - for example, it could be that you are using some sub-optimal
> raster format, or there are overviews missing, or something else... one
> would need more details.
>
> Regards
> Martin
>
>
> _______________________________________________
> QGIS-Developer mailing listQGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
> --
> Johannes Kröger / GIS-Entwickler/-Berater
>
> ---------------------------------------------
> Aufwind durch Wissen!
> Web-Seminare und Online-Schulungen
> bei der www.foss-academy.com
> ---------------------------------------------
>
> WhereGroup GmbH
> c/o KK03 GmbH
> Lange Reihe 29
> 20099 Hamburg
> Germany
>
> Tel: +49 (0)228 / 90 90 38 - 36
> Fax: +49 (0)228 / 90 90 38 - 11
> johannes.kroeger at wheregroup.comwww.wheregroup.com
> Geschäftsführer:
> Olaf Knopp, Peter Stamm
> Amtsgericht Bonn, HRB 9885
> -------------------------------
>
>
> _______________________________________________
> QGIS-Developer mailing listQGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
> --
> Johannes Kröger / GIS-Entwickler/-Berater
>
> ---------------------------------------------
> Aufwind durch Wissen!
> Web-Seminare und Online-Schulungen
> bei der www.foss-academy.com
> ---------------------------------------------
>
> WhereGroup GmbH
> c/o KK03 GmbH
> Lange Reihe 29
> 20099 Hamburg
> Germany
>
> Tel: +49 (0)228 / 90 90 38 - 36
> Fax: +49 (0)228 / 90 90 38 - 11
> johannes.kroeger at wheregroup.comwww.wheregroup.com
> Geschäftsführer:
> Olaf Knopp, Peter Stamm
> Amtsgericht Bonn, HRB 9885
> -------------------------------
>
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
--
Ian Turton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20230531/49715286/attachment-0001.htm>
More information about the QGIS-Developer
mailing list