[QGIS-Developer] How to speed up rendering
Johannes Kröger (WhereGroup)
johannes.kroeger at wheregroup.com
Wed May 31 00:46:51 PDT 2023
Sorry Arturo,
I totally missed that Martin had already recommended it and that you
said you didn't know about it. I didn't mean to come across pushy or
anything :))
Andreas' links are great to see how to create them. Here is a nice post
that explains the concept:
https://developers.planet.com/docs/planetschool/an-introduction-to-cloud-optimized-geotiffs-cogs-part-1-overview/
As you can see there you probably also want to created "tiled" TIFFs,
where the data internal to the TIFF is arranged in a way that fits
spatial queries better.
If you can, simply specify that you want "cloud-optimized geotiffs",
those have all these optimisations and are also "typical spatial
analysis optimized geotiffs".
Cheers, Hannes
Am 30.05.23 um 14:37 schrieb afernandez via QGIS-Developer:
> Hello Hannes,
> Sorry to say that I'm at a loss here. I had not previously heard of
> 'overviews' and am having a hard time finding much information. I
> couldn't find anything in the PyQGIS (v3.28) developer cookbook, which
> is my main reference in this QGIS journey. The found information is at
> https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/gdal/rastermiscellaneous.html
> but didn't exactly clarify how to implement overviews with a script or
> why this would sidestep reading the file several times. I don't quite
> understand this last point either, as the file is large but the size
> of the band (after everything is prepped and before rendering) is
> relatively small, and should fit amply into the available memory.
> Thanks,
> Arturo
>
>
> Johannes Kröger (WhereGroup) via QGIS-Developer wrote:
>
>
>> Hi Arturo,
>>
>> make sure you create overviews on your big rasters or QGIS will
>> re-read the whole file all the time to render an appropriate
>> representation in the map.
>>
>> Cheers, Hannes
>>
>> Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer:
>>> Hello Martin,
>>> I'm trying to understand what you're saying but my background is in
>>> numerical methods where tasks are split into threads for
>>> acceleration purposes. If a single layer is rendered by a single
>>> core (as you write and I was observing), how would a 1,000 x 1,000
>>> (or a 5,000 x 5,000) layer be handled? Wouldn't it take too long?
>>> (However, you want to define 'too long').
>>> As far as more details about the process, the tensor 'var' (n x m x
>>> index where I have tried different combinations but let's say n x m
>>> is 100 x 200) of floating point numbers is manipulated according to
>>> the following pseudo-code:
>>> # Initial manipulations
>>> dims = var.dimensions
>>> shape = var.shape
>>> driver_name = 'GTIFF'
>>> driver = gdal.GetDriverByName(driver_name)
>>> np_dtype = var.dtype
>>> type_code =
>>> gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)
>>> gdal_ds = driver.Create(_my_path_, cols, rows, 1,
>>> gdal.GDT_UInt16)
>>> gdal_ds.SetProjection(_my_projection_)
>>> gdal_ds.SetGeoTransform(_my_transformation_)
>>> # Creation of the bands and scaled matrix
>>> band = gdal_ds.GetRasterBand(1)
>>> data = var[_chosen_index_]
>>> data = ma.getdata(data)
>>> data_scaled = np.interp(data, (data.min(), data.max()),
>>> (0, 255))
>>> data_scaled2 = data_scaled.astype(int) # This is to
>>> rescale into integers so that it can color the layer
>>> # *** Lines to set up the color palette ***
>>> # Write the array to band once everything has been rescaled
>>> band.WriteArray(data_scaled2)
>>> gdal_ds.FlushCache()
>>>
>>> This procedure seems to work well and takes a reasonable amount of
>>> time. Then, the time is consumed when doing the rendering itself.
>>> I'm not entirely sure what you mean by 'sub-optimal raster format'
>>> or 'thre are overviews missing' so I cannot address them specifically.
>>> Thanks,
>>> Arturo
>>>
>>>
>>>
>>>
>>> Martin Dobias wrote:
>>>
>>>
>>>> Hi
>>>>
>>>> On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer
>>>> <qgis-developer at lists.osgeo.org> wrote:
>>>>
>>>> Hello,
>>>> One of my project would greatly benefit from a much faster
>>>> rendering. After some testing, the bulk (>99%) of the time is
>>>> spent at rendering the layer with QgsRasterLayer(...,...). What
>>>> I have also noticed is that rendering time does not change with
>>>> the number of cores, which suggests that the renderer might be
>>>> using a single thread. I checked the API documentation but
>>>> couldn't find anything about multithreading or some other
>>>> procedure to accelerate rendering. I was wondering if anyone
>>>> has any suggestion or has faced a similar difficulty.
>>>>
>>>>
>>>> Multi-threaded rendering is done at the granularity of whole map
>>>> layers, so a single layer is rendered by a single CPU core.
>>>> Therefore if you have just one layer that is slow to render, the
>>>> multi-threading is not going to help. It is best to do some testing
>>>> how you can improve the layer's rendering - for example, it could
>>>> be that you are using some sub-optimal raster format, or there are
>>>> overviews missing, or something else... one would need more details.
>>>>
>>>> Regards
>>>> Martin
>>>>
>>>
>>> _______________________________________________
>>> QGIS-Developer mailing list
>>> QGIS-Developer at lists.osgeo.org
>>> List info:https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>> Unsubscribe:https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> --
>> Johannes Kröger / GIS-Entwickler/-Berater
>>
>> ---------------------------------------------
>> Aufwind durch Wissen!
>> Web-Seminare und Online-Schulungen
>> bei derwww.foss-academy.com
>> ---------------------------------------------
>>
>> WhereGroup GmbH
>> c/o KK03 GmbH
>> Lange Reihe 29
>> 20099 Hamburg
>> Germany
>>
>> Tel: +49 (0)228 / 90 90 38 - 36
>> Fax: +49 (0)228 / 90 90 38 - 11
>>
>> johannes.kroeger at wheregroup.com
>> www.wheregroup.com
>> Geschäftsführer:
>> Olaf Knopp, Peter Stamm
>> Amtsgericht Bonn, HRB 9885
>> -------------------------------
>>
>> Hi Arturo,
>>
>> make sure you create overviews on your big rasters or QGIS will
>> re-read the whole file all the time to render an appropriate
>> representation in the map.
>>
>> Cheers, Hannes
>>
>> Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer:
>>> Hello Martin,
>>> I'm trying to understand what you're saying but my background is in
>>> numerical methods where tasks are split into threads for
>>> acceleration purposes. If a single layer is rendered by a single
>>> core (as you write and I was observing), how would a 1,000 x 1,000
>>> (or a 5,000 x 5,000) layer be handled? Wouldn't it take too long?
>>> (However, you want to define 'too long').
>>> As far as more details about the process, the tensor 'var' (n x m x
>>> index where I have tried different combinations but let's say n x m
>>> is 100 x 200) of floating point numbers is manipulated according to
>>> the following pseudo-code:
>>> # Initial manipulations
>>> dims = var.dimensions
>>> shape = var.shape
>>> driver_name = 'GTIFF'
>>> driver = gdal.GetDriverByName(driver_name)
>>> np_dtype = var.dtype
>>> type_code =
>>> gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)
>>> gdal_ds = driver.Create(_my_path_, cols, rows, 1,
>>> gdal.GDT_UInt16)
>>> gdal_ds.SetProjection(_my_projection_)
>>> gdal_ds.SetGeoTransform(_my_transformation_)
>>> # Creation of the bands and scaled matrix
>>> band = gdal_ds.GetRasterBand(1)
>>> data = var[_chosen_index_]
>>> data = ma.getdata(data)
>>> data_scaled = np.interp(data, (data.min(), data.max()),
>>> (0, 255))
>>> data_scaled2 = data_scaled.astype(int) # This is to
>>> rescale into integers so that it can color the layer
>>> # *** Lines to set up the color palette ***
>>> # Write the array to band once everything has been rescaled
>>> band.WriteArray(data_scaled2)
>>> gdal_ds.FlushCache()
>>>
>>> This procedure seems to work well and takes a reasonable amount of
>>> time. Then, the time is consumed when doing the rendering itself.
>>> I'm not entirely sure what you mean by 'sub-optimal raster format'
>>> or 'thre are overviews missing' so I cannot address them specifically.
>>> Thanks,
>>> Arturo
>>>
>>>
>>>
>>>
>>> Martin Dobias wrote:
>>>
>>>
>>>> Hi
>>>>
>>>> On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer
>>>> <qgis-developer at lists.osgeo.org> wrote:
>>>>
>>>> Hello,
>>>> One of my project would greatly benefit from a much faster
>>>> rendering. After some testing, the bulk (>99%) of the time is
>>>> spent at rendering the layer with QgsRasterLayer(...,...). What
>>>> I have also noticed is that rendering time does not change with
>>>> the number of cores, which suggests that the renderer might be
>>>> using a single thread. I checked the API documentation but
>>>> couldn't find anything about multithreading or some other
>>>> procedure to accelerate rendering. I was wondering if anyone
>>>> has any suggestion or has faced a similar difficulty.
>>>>
>>>>
>>>> Multi-threaded rendering is done at the granularity of whole map
>>>> layers, so a single layer is rendered by a single CPU core.
>>>> Therefore if you have just one layer that is slow to render, the
>>>> multi-threading is not going to help. It is best to do some testing
>>>> how you can improve the layer's rendering - for example, it could
>>>> be that you are using some sub-optimal raster format, or there are
>>>> overviews missing, or something else... one would need more details.
>>>>
>>>> Regards
>>>> Martin
>>>>
>>>
>>> _______________________________________________
>>> QGIS-Developer mailing list
>>> QGIS-Developer at lists.osgeo.org
>>> List info:https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>> Unsubscribe:https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> --
>> Johannes Kröger / GIS-Entwickler/-Berater
>>
>> ---------------------------------------------
>> Aufwind durch Wissen!
>> Web-Seminare und Online-Schulungen
>> bei derwww.foss-academy.com
>> ---------------------------------------------
>>
>> WhereGroup GmbH
>> c/o KK03 GmbH
>> Lange Reihe 29
>> 20099 Hamburg
>> Germany
>>
>> Tel: +49 (0)228 / 90 90 38 - 36
>> Fax: +49 (0)228 / 90 90 38 - 11
>>
>> johannes.kroeger at wheregroup.com
>> www.wheregroup.com
>> Geschäftsführer:
>> Olaf Knopp, Peter Stamm
>> Amtsgericht Bonn, HRB 9885
>> -------------------------------
>
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info:https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe:https://lists.osgeo.org/mailman/listinfo/qgis-developer
--
Johannes Kröger / GIS-Entwickler/-Berater
---------------------------------------------
Aufwind durch Wissen!
Web-Seminare und Online-Schulungen
bei derwww.foss-academy.com
---------------------------------------------
WhereGroup GmbH
c/o KK03 GmbH
Lange Reihe 29
20099 Hamburg
Germany
Tel: +49 (0)228 / 90 90 38 - 36
Fax: +49 (0)228 / 90 90 38 - 11
johannes.kroeger at wheregroup.com
www.wheregroup.com
Geschäftsführer:
Olaf Knopp, Peter Stamm
Amtsgericht Bonn, HRB 9885
-------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20230531/89f03225/attachment-0001.htm>
More information about the QGIS-Developer
mailing list