<div dir="ltr"><div>Hi,</div><div><br></div><div>See <a href="https://docs.qgis.org/3.28/en/docs/user_manual/working_with_raster/raster_properties.html#pyramids-properties">https://docs.qgis.org/3.28/en/docs/user_manual/working_with_raster/raster_properties.html#pyramids-properties</a> (for a QGIS based solution)<br></div><div>or <a href="https://gdal.org/programs/gdaladdo.html">https://gdal.org/programs/gdaladdo.html</a> (for a command line or python version).</div><div><br></div><div>Andreas<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 30 May 2023 at 16:00, afernandez via QGIS-Developer <<a href="mailto:qgis-developer@lists.osgeo.org">qgis-developer@lists.osgeo.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>
<div id="m_-5713556291054507349compose-body-wrapper" dir="auto"><div dir="auto"><div dir="auto">Hello Hannes,</div><div dir="auto">Sorry to say that I'm at a loss here. I had not previously heard of 'overviews' and am having a hard time finding much information. I couldn't find anything in the PyQGIS (v3.28) developer cookbook, which is my main reference in this QGIS journey. The found information is at <a href="https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/gdal/rastermiscellaneous.html" target="_blank">https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/gdal/rastermiscellaneous.html</a> but didn't exactly clarify how to implement overviews with a script or why this would sidestep reading the file several times. I don't quite understand this last point either, as the file is large but the size of the band (after everything is prepped and before rendering) is relatively small, and should fit amply into the available memory.</div><div dir="auto">Thanks,</div><div dir="auto">Arturo</div><div><br></div></div><div dir="auto"><br></div></div><div dir="auto">Johannes Kröger (WhereGroup) via QGIS-Developer wrote:</div><br><br><div><blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<p>Hi Arturo,</p>
<p>make sure you create overviews on your big rasters or QGIS will
re-read the whole file all the time to render an appropriate
representation in the map.</p>
<p>Cheers, Hannes<br>
</p>
<div>Am 30.05.23 um 04:35 schrieb afernandez
via QGIS-Developer:<br>
</div>
<blockquote type="cite">
<div>
<div id="m_-5713556291054507349compose-body-wrapper" dir="auto">
<div dir="auto">
<div dir="auto">Hello Martin,</div>
<div dir="auto">I'm trying to understand what you're saying
but my background is in numerical methods where tasks are
split into threads for acceleration purposes. If a single
layer is rendered by a single core (as you write and I was
observing), how would a 1,000 x 1,000 (or a 5,000 x 5,000)
layer be handled? Wouldn't it take too long? (However, you
want to define 'too long'). </div>
<div dir="auto">As far as more details about the process,
the tensor 'var' (n x m x index where I have tried
different combinations but let's say n x m is 100 x 200)
of floating point numbers is manipulated according to the
following pseudo-code:</div>
<div dir="auto"># Initial manipulations</div>
<div dir="auto"> dims = var.dimensions</div>
<div dir="auto"> shape = var.shape</div>
<div dir="auto"> driver_name = 'GTIFF'</div>
<div dir="auto"> driver =
gdal.GetDriverByName(driver_name) </div>
<div dir="auto"> np_dtype = var.dtype</div>
<div dir="auto"> type_code =
gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)</div>
<div dir="auto"> gdal_ds =
driver.Create(_my_path_, cols, rows, 1, gdal.GDT_UInt16) </div>
<div dir="auto">
gdal_ds.SetProjection(_my_projection_)</div>
<div dir="auto">
gdal_ds.SetGeoTransform(_my_transformation_)</div>
<div dir="auto"># Creation of the bands and scaled matrix </div>
<div dir="auto"> band = gdal_ds.GetRasterBand(1) </div>
<div dir="auto"> data = var[_chosen_index_]</div>
<div dir="auto"> data = ma.getdata(data)</div>
<div dir="auto"> data_scaled = np.interp(data,
(data.min(), data.max()), (0, 255))</div>
<div dir="auto"> data_scaled2 =
data_scaled.astype(int) # This is to rescale into integers
so that it can color the layer</div>
<div dir="auto"># *** Lines to set up the color palette ***</div>
<div dir="auto"># Write the array to band once everything
has been rescaled</div>
<div dir="auto"> band.WriteArray(data_scaled2)</div>
<div dir="auto"> gdal_ds.FlushCache() </div>
<div dir="auto"><br>
</div>
<div dir="auto">This procedure seems to work well and takes
a reasonable amount of time. Then, the time is consumed
when doing the rendering itself. I'm not entirely sure
what you mean by 'sub-optimal raster format' or 'thre are
overviews missing' so I cannot address them specifically.</div>
<div dir="auto">Thanks,</div>
<div dir="auto">Arturo</div>
<div><br>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto" id="m_-5713556291054507349tmjah_g_1299"><br>
</div>
<br>
</div>
<div dir="auto">Martin Dobias wrote:</div>
<br>
<br>
<div>
<blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Hi<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, May 26, 2023
at 11:30 PM afernandez via QGIS-Developer <<a href="mailto:qgis-developer@lists.osgeo.org" target="_blank">qgis-developer@lists.osgeo.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>
<div id="m_-5713556291054507349m_-6129067434502316114compose-body-wrapper" dir="auto">
<div dir="auto">
<div dir="auto">Hello,</div>
<div dir="auto">One of my project would
greatly benefit from a much faster
rendering. After some testing, the bulk
(>99%) of the time is spent at rendering
the layer with QgsRasterLayer(...,...). What
I have also noticed is that rendering time
does not change with the number of cores,
which suggests that the renderer might be
using a single thread. I checked the API
documentation but couldn't find anything
about multithreading or some other procedure
to accelerate rendering. I was wondering if
anyone has any suggestion or has faced a
similar difficulty. </div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Multi-threaded rendering is done
at the granularity of whole map layers, so a single
layer is rendered by a single CPU core. Therefore if you
have just one layer that is slow to render, the
multi-threading is not going to help. It is best to do
some testing how you can improve the layer's rendering -
for example, it could be that you are using some
sub-optimal raster format, or there are overviews
missing, or something else... one would need more
details.</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Regards</div>
<div class="gmail_quote">Martin</div>
<div class="gmail_quote"><br>
</div>
</div>
</blockquote>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
QGIS-Developer mailing list
<a href="mailto:QGIS-Developer@lists.osgeo.org" target="_blank">QGIS-Developer@lists.osgeo.org</a>
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a>
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a>
</pre>
</blockquote>
<pre cols="72">--
Johannes Kröger / GIS-Entwickler/-Berater
---------------------------------------------
Aufwind durch Wissen!
Web-Seminare und Online-Schulungen
bei der <a href="http://www.foss-academy.com" target="_blank">www.foss-academy.com</a>
---------------------------------------------
WhereGroup GmbH
c/o KK03 GmbH
Lange Reihe 29
20099 Hamburg
Germany
Tel: +49 (0)228 / 90 90 38 - 36
Fax: +49 (0)228 / 90 90 38 - 11
<a href="mailto:johannes.kroeger@wheregroup.com" target="_blank">johannes.kroeger@wheregroup.com</a>
<a href="http://www.wheregroup.com" target="_blank">www.wheregroup.com</a>
Geschäftsführer:
Olaf Knopp, Peter Stamm
Amtsgericht Bonn, HRB 9885
-------------------------------
</pre>
</blockquote></div><div><blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<p>Hi Arturo,</p>
<p>make sure you create overviews on your big rasters or QGIS will
re-read the whole file all the time to render an appropriate
representation in the map.</p>
<p>Cheers, Hannes<br>
</p>
<div>Am 30.05.23 um 04:35 schrieb afernandez
via QGIS-Developer:<br>
</div>
<blockquote type="cite">
<div>
<div id="m_-5713556291054507349compose-body-wrapper" dir="auto">
<div dir="auto">
<div dir="auto">Hello Martin,</div>
<div dir="auto">I'm trying to understand what you're saying
but my background is in numerical methods where tasks are
split into threads for acceleration purposes. If a single
layer is rendered by a single core (as you write and I was
observing), how would a 1,000 x 1,000 (or a 5,000 x 5,000)
layer be handled? Wouldn't it take too long? (However, you
want to define 'too long'). </div>
<div dir="auto">As far as more details about the process,
the tensor 'var' (n x m x index where I have tried
different combinations but let's say n x m is 100 x 200)
of floating point numbers is manipulated according to the
following pseudo-code:</div>
<div dir="auto"># Initial manipulations</div>
<div dir="auto"> dims = var.dimensions</div>
<div dir="auto"> shape = var.shape</div>
<div dir="auto"> driver_name = 'GTIFF'</div>
<div dir="auto"> driver =
gdal.GetDriverByName(driver_name) </div>
<div dir="auto"> np_dtype = var.dtype</div>
<div dir="auto"> type_code =
gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)</div>
<div dir="auto"> gdal_ds =
driver.Create(_my_path_, cols, rows, 1, gdal.GDT_UInt16) </div>
<div dir="auto">
gdal_ds.SetProjection(_my_projection_)</div>
<div dir="auto">
gdal_ds.SetGeoTransform(_my_transformation_)</div>
<div dir="auto"># Creation of the bands and scaled matrix </div>
<div dir="auto"> band = gdal_ds.GetRasterBand(1) </div>
<div dir="auto"> data = var[_chosen_index_]</div>
<div dir="auto"> data = ma.getdata(data)</div>
<div dir="auto"> data_scaled = np.interp(data,
(data.min(), data.max()), (0, 255))</div>
<div dir="auto"> data_scaled2 =
data_scaled.astype(int) # This is to rescale into integers
so that it can color the layer</div>
<div dir="auto"># *** Lines to set up the color palette ***</div>
<div dir="auto"># Write the array to band once everything
has been rescaled</div>
<div dir="auto"> band.WriteArray(data_scaled2)</div>
<div dir="auto"> gdal_ds.FlushCache() </div>
<div dir="auto"><br>
</div>
<div dir="auto">This procedure seems to work well and takes
a reasonable amount of time. Then, the time is consumed
when doing the rendering itself. I'm not entirely sure
what you mean by 'sub-optimal raster format' or 'thre are
overviews missing' so I cannot address them specifically.</div>
<div dir="auto">Thanks,</div>
<div dir="auto">Arturo</div>
<div><br>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto" id="m_-5713556291054507349tmjah_g_1299"><br>
</div>
<br>
</div>
<div dir="auto">Martin Dobias wrote:</div>
<br>
<br>
<div>
<blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Hi<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, May 26, 2023
at 11:30 PM afernandez via QGIS-Developer <<a href="mailto:qgis-developer@lists.osgeo.org" target="_blank">qgis-developer@lists.osgeo.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>
<div id="m_-5713556291054507349m_-6129067434502316114compose-body-wrapper" dir="auto">
<div dir="auto">
<div dir="auto">Hello,</div>
<div dir="auto">One of my project would
greatly benefit from a much faster
rendering. After some testing, the bulk
(>99%) of the time is spent at rendering
the layer with QgsRasterLayer(...,...). What
I have also noticed is that rendering time
does not change with the number of cores,
which suggests that the renderer might be
using a single thread. I checked the API
documentation but couldn't find anything
about multithreading or some other procedure
to accelerate rendering. I was wondering if
anyone has any suggestion or has faced a
similar difficulty. </div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Multi-threaded rendering is done
at the granularity of whole map layers, so a single
layer is rendered by a single CPU core. Therefore if you
have just one layer that is slow to render, the
multi-threading is not going to help. It is best to do
some testing how you can improve the layer's rendering -
for example, it could be that you are using some
sub-optimal raster format, or there are overviews
missing, or something else... one would need more
details.</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Regards</div>
<div class="gmail_quote">Martin</div>
<div class="gmail_quote"><br>
</div>
</div>
</blockquote>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
QGIS-Developer mailing list
<a href="mailto:QGIS-Developer@lists.osgeo.org" target="_blank">QGIS-Developer@lists.osgeo.org</a>
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a>
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a>
</pre>
</blockquote>
<pre cols="72">--
Johannes Kröger / GIS-Entwickler/-Berater
---------------------------------------------
Aufwind durch Wissen!
Web-Seminare und Online-Schulungen
bei der <a href="http://www.foss-academy.com" target="_blank">www.foss-academy.com</a>
---------------------------------------------
WhereGroup GmbH
c/o KK03 GmbH
Lange Reihe 29
20099 Hamburg
Germany
Tel: +49 (0)228 / 90 90 38 - 36
Fax: +49 (0)228 / 90 90 38 - 11
<a href="mailto:johannes.kroeger@wheregroup.com" target="_blank">johannes.kroeger@wheregroup.com</a>
<a href="http://www.wheregroup.com" target="_blank">www.wheregroup.com</a>
Geschäftsführer:
Olaf Knopp, Peter Stamm
Amtsgericht Bonn, HRB 9885
-------------------------------
</pre>
</blockquote></div>
</div></div>_______________________________________________<br>
QGIS-Developer mailing list<br>
<a href="mailto:QGIS-Developer@lists.osgeo.org" target="_blank">QGIS-Developer@lists.osgeo.org</a><br>
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a><br>
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a><br>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><br>--<br>Andreas Neumann<br></div><a href="http://QGIS.ORG" target="_blank">QGIS.ORG</a> board member (treasurer)<br></div></div>