<div dir="ltr"><div>A good introduction to fast rasters is Paul Ramsey's GeoTiff compression for dummies blog post - <a href="https://blog.cleverelephant.ca/2015/02/geotiff-compression-for-dummies.html">https://blog.cleverelephant.ca/2015/02/geotiff-compression-for-dummies.html</a> it takes you through all the steps needed using gdal.</div><div><br></div><div>Ian<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 31 May 2023 at 08:47, Johannes Kröger (WhereGroup) via QGIS-Developer <<a href="mailto:qgis-developer@lists.osgeo.org">qgis-developer@lists.osgeo.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Sorry Arturo,</p>
<p>I totally missed that Martin had already recommended it and that
you said you didn't know about it. I didn't mean to come across
pushy or anything :))</p>
<p>Andreas' links are great to see how to create them. Here is a
nice post that explains the concept:
<a href="https://developers.planet.com/docs/planetschool/an-introduction-to-cloud-optimized-geotiffs-cogs-part-1-overview/" target="_blank">https://developers.planet.com/docs/planetschool/an-introduction-to-cloud-optimized-geotiffs-cogs-part-1-overview/</a></p>
<p>As you can see there you probably also want to created "tiled"
TIFFs, where the data internal to the TIFF is arranged in a way
that fits spatial queries better.</p>
<p>If you can, simply specify that you want "cloud-optimized
geotiffs", those have all these optimisations and are also
"typical spatial analysis optimized geotiffs".<br>
</p>
<p>Cheers, Hannes<br>
</p>
<div>Am 30.05.23 um 14:37 schrieb afernandez
via QGIS-Developer:<br>
</div>
<blockquote type="cite">
<div>
<div id="m_794229472922089744compose-body-wrapper" dir="auto">
<div dir="auto">
<div dir="auto">Hello Hannes,</div>
<div dir="auto">Sorry to say that I'm at a loss here. I had
not previously heard of 'overviews' and am having a hard
time finding much information. I couldn't find anything in
the PyQGIS (v3.28) developer cookbook, which is my main
reference in this QGIS journey. The found information is
at
<a href="https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/gdal/rastermiscellaneous.html" target="_blank">https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/gdal/rastermiscellaneous.html</a>
but didn't exactly clarify how to implement overviews with
a script or why this would sidestep reading the file
several times. I don't quite understand this last point
either, as the file is large but the size of the band
(after everything is prepped and before rendering) is
relatively small, and should fit amply into the available
memory.</div>
<div dir="auto">Thanks,</div>
<div dir="auto">Arturo</div>
<div><br>
</div>
</div>
<div dir="auto"><br>
</div>
</div>
<div dir="auto">Johannes Kröger (WhereGroup)
via QGIS-Developer wrote:</div>
<br>
<br>
<div>
<blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<p>Hi Arturo,</p>
<p>make sure you create overviews on your big rasters or
QGIS will re-read the whole file all the time to render an
appropriate representation in the map.</p>
<p>Cheers, Hannes<br>
</p>
<div>Am 30.05.23 um 04:35 schrieb
afernandez via QGIS-Developer:<br>
</div>
<blockquote type="cite">
<div>
<div id="m_794229472922089744compose-body-wrapper" dir="auto">
<div dir="auto">
<div dir="auto">Hello Martin,</div>
<div dir="auto">I'm trying to understand what you're
saying but my background is in numerical methods
where tasks are split into threads for
acceleration purposes. If a single layer is
rendered by a single core (as you write and I was
observing), how would a 1,000 x 1,000 (or a 5,000
x 5,000) layer be handled? Wouldn't it take too
long? (However, you want to define 'too long'). </div>
<div dir="auto">As far as more details about the
process, the tensor 'var' (n x m x index where I
have tried different combinations but let's say n
x m is 100 x 200) of floating point numbers is
manipulated according to the following
pseudo-code:</div>
<div dir="auto"># Initial manipulations</div>
<div dir="auto"> dims = var.dimensions</div>
<div dir="auto"> shape = var.shape</div>
<div dir="auto"> driver_name = 'GTIFF'</div>
<div dir="auto"> driver =
gdal.GetDriverByName(driver_name) </div>
<div dir="auto"> np_dtype = var.dtype</div>
<div dir="auto"> type_code =
gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)</div>
<div dir="auto"> gdal_ds =
driver.Create(_my_path_, cols, rows, 1,
gdal.GDT_UInt16) </div>
<div dir="auto">
gdal_ds.SetProjection(_my_projection_)</div>
<div dir="auto">
gdal_ds.SetGeoTransform(_my_transformation_)</div>
<div dir="auto"># Creation of the bands and scaled
matrix </div>
<div dir="auto"> band =
gdal_ds.GetRasterBand(1) </div>
<div dir="auto"> data =
var[_chosen_index_]</div>
<div dir="auto"> data = ma.getdata(data)</div>
<div dir="auto"> data_scaled =
np.interp(data, (data.min(), data.max()), (0,
255))</div>
<div dir="auto"> data_scaled2 =
data_scaled.astype(int) # This is to rescale into
integers so that it can color the layer</div>
<div dir="auto"># *** Lines to set up the color
palette ***</div>
<div dir="auto"># Write the array to band once
everything has been rescaled</div>
<div dir="auto">
band.WriteArray(data_scaled2)</div>
<div dir="auto"> gdal_ds.FlushCache() </div>
<div dir="auto"><br>
</div>
<div dir="auto">This procedure seems to work well
and takes a reasonable amount of time. Then, the
time is consumed when doing the rendering itself.
I'm not entirely sure what you mean by
'sub-optimal raster format' or 'thre are overviews
missing' so I cannot address them specifically.</div>
<div dir="auto">Thanks,</div>
<div dir="auto">Arturo</div>
<div><br>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto" id="m_794229472922089744tmjah_g_1299"><br>
</div>
<br>
</div>
<div dir="auto">Martin Dobias wrote:</div>
<br>
<br>
<div>
<blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Hi<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, May
26, 2023 at 11:30 PM afernandez via
QGIS-Developer <<a href="mailto:qgis-developer@lists.osgeo.org" target="_blank">qgis-developer@lists.osgeo.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>
<div id="m_794229472922089744m_-6129067434502316114compose-body-wrapper" dir="auto">
<div dir="auto">
<div dir="auto">Hello,</div>
<div dir="auto">One of my project
would greatly benefit from a much
faster rendering. After some
testing, the bulk (>99%) of the
time is spent at rendering the layer
with QgsRasterLayer(...,...). What I
have also noticed is that rendering
time does not change with the number
of cores, which suggests that the
renderer might be using a single
thread. I checked the API
documentation but couldn't find
anything about multithreading or
some other procedure to accelerate
rendering. I was wondering if anyone
has any suggestion or has faced a
similar difficulty. </div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Multi-threaded rendering
is done at the granularity of whole map layers,
so a single layer is rendered by a single CPU
core. Therefore if you have just one layer that
is slow to render, the multi-threading is not
going to help. It is best to do some testing how
you can improve the layer's rendering - for
example, it could be that you are using some
sub-optimal raster format, or there are
overviews missing, or something else... one
would need more details.</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Regards</div>
<div class="gmail_quote">Martin</div>
<div class="gmail_quote"><br>
</div>
</div>
</blockquote>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
QGIS-Developer mailing list
<a href="mailto:QGIS-Developer@lists.osgeo.org" target="_blank">QGIS-Developer@lists.osgeo.org</a>
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a>
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a>
</pre>
</blockquote>
<pre cols="72">--
Johannes Kröger / GIS-Entwickler/-Berater
---------------------------------------------
Aufwind durch Wissen!
Web-Seminare und Online-Schulungen
bei der <a href="http://www.foss-academy.com" target="_blank">www.foss-academy.com</a>
---------------------------------------------
WhereGroup GmbH
c/o KK03 GmbH
Lange Reihe 29
20099 Hamburg
Germany
Tel: +49 (0)228 / 90 90 38 - 36
Fax: +49 (0)228 / 90 90 38 - 11
<a href="mailto:johannes.kroeger@wheregroup.com" target="_blank">johannes.kroeger@wheregroup.com</a>
<a href="http://www.wheregroup.com" target="_blank">www.wheregroup.com</a>
Geschäftsführer:
Olaf Knopp, Peter Stamm
Amtsgericht Bonn, HRB 9885
-------------------------------
</pre>
</blockquote>
</div>
<div>
<blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<p>Hi Arturo,</p>
<p>make sure you create overviews on your big rasters or
QGIS will re-read the whole file all the time to render an
appropriate representation in the map.</p>
<p>Cheers, Hannes<br>
</p>
<div>Am 30.05.23 um 04:35 schrieb
afernandez via QGIS-Developer:<br>
</div>
<blockquote type="cite">
<div>
<div id="m_794229472922089744compose-body-wrapper" dir="auto">
<div dir="auto">
<div dir="auto">Hello Martin,</div>
<div dir="auto">I'm trying to understand what you're
saying but my background is in numerical methods
where tasks are split into threads for
acceleration purposes. If a single layer is
rendered by a single core (as you write and I was
observing), how would a 1,000 x 1,000 (or a 5,000
x 5,000) layer be handled? Wouldn't it take too
long? (However, you want to define 'too long'). </div>
<div dir="auto">As far as more details about the
process, the tensor 'var' (n x m x index where I
have tried different combinations but let's say n
x m is 100 x 200) of floating point numbers is
manipulated according to the following
pseudo-code:</div>
<div dir="auto"># Initial manipulations</div>
<div dir="auto"> dims = var.dimensions</div>
<div dir="auto"> shape = var.shape</div>
<div dir="auto"> driver_name = 'GTIFF'</div>
<div dir="auto"> driver =
gdal.GetDriverByName(driver_name) </div>
<div dir="auto"> np_dtype = var.dtype</div>
<div dir="auto"> type_code =
gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)</div>
<div dir="auto"> gdal_ds =
driver.Create(_my_path_, cols, rows, 1,
gdal.GDT_UInt16) </div>
<div dir="auto">
gdal_ds.SetProjection(_my_projection_)</div>
<div dir="auto">
gdal_ds.SetGeoTransform(_my_transformation_)</div>
<div dir="auto"># Creation of the bands and scaled
matrix </div>
<div dir="auto"> band =
gdal_ds.GetRasterBand(1) </div>
<div dir="auto"> data =
var[_chosen_index_]</div>
<div dir="auto"> data = ma.getdata(data)</div>
<div dir="auto"> data_scaled =
np.interp(data, (data.min(), data.max()), (0,
255))</div>
<div dir="auto"> data_scaled2 =
data_scaled.astype(int) # This is to rescale into
integers so that it can color the layer</div>
<div dir="auto"># *** Lines to set up the color
palette ***</div>
<div dir="auto"># Write the array to band once
everything has been rescaled</div>
<div dir="auto">
band.WriteArray(data_scaled2)</div>
<div dir="auto"> gdal_ds.FlushCache() </div>
<div dir="auto"><br>
</div>
<div dir="auto">This procedure seems to work well
and takes a reasonable amount of time. Then, the
time is consumed when doing the rendering itself.
I'm not entirely sure what you mean by
'sub-optimal raster format' or 'thre are overviews
missing' so I cannot address them specifically.</div>
<div dir="auto">Thanks,</div>
<div dir="auto">Arturo</div>
<div><br>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto" id="m_794229472922089744tmjah_g_1299"><br>
</div>
<br>
</div>
<div dir="auto">Martin Dobias wrote:</div>
<br>
<br>
<div>
<blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Hi<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, May
26, 2023 at 11:30 PM afernandez via
QGIS-Developer <<a href="mailto:qgis-developer@lists.osgeo.org" target="_blank">qgis-developer@lists.osgeo.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>
<div id="m_794229472922089744m_-6129067434502316114compose-body-wrapper" dir="auto">
<div dir="auto">
<div dir="auto">Hello,</div>
<div dir="auto">One of my project
would greatly benefit from a much
faster rendering. After some
testing, the bulk (>99%) of the
time is spent at rendering the layer
with QgsRasterLayer(...,...). What I
have also noticed is that rendering
time does not change with the number
of cores, which suggests that the
renderer might be using a single
thread. I checked the API
documentation but couldn't find
anything about multithreading or
some other procedure to accelerate
rendering. I was wondering if anyone
has any suggestion or has faced a
similar difficulty. </div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Multi-threaded rendering
is done at the granularity of whole map layers,
so a single layer is rendered by a single CPU
core. Therefore if you have just one layer that
is slow to render, the multi-threading is not
going to help. It is best to do some testing how
you can improve the layer's rendering - for
example, it could be that you are using some
sub-optimal raster format, or there are
overviews missing, or something else... one
would need more details.</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Regards</div>
<div class="gmail_quote">Martin</div>
<div class="gmail_quote"><br>
</div>
</div>
</blockquote>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
QGIS-Developer mailing list
<a href="mailto:QGIS-Developer@lists.osgeo.org" target="_blank">QGIS-Developer@lists.osgeo.org</a>
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a>
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a>
</pre>
</blockquote>
<pre cols="72">--
Johannes Kröger / GIS-Entwickler/-Berater
---------------------------------------------
Aufwind durch Wissen!
Web-Seminare und Online-Schulungen
bei der <a href="http://www.foss-academy.com" target="_blank">www.foss-academy.com</a>
---------------------------------------------
WhereGroup GmbH
c/o KK03 GmbH
Lange Reihe 29
20099 Hamburg
Germany
Tel: +49 (0)228 / 90 90 38 - 36
Fax: +49 (0)228 / 90 90 38 - 11
<a href="mailto:johannes.kroeger@wheregroup.com" target="_blank">johannes.kroeger@wheregroup.com</a>
<a href="http://www.wheregroup.com" target="_blank">www.wheregroup.com</a>
Geschäftsführer:
Olaf Knopp, Peter Stamm
Amtsgericht Bonn, HRB 9885
-------------------------------
</pre>
</blockquote>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
QGIS-Developer mailing list
<a href="mailto:QGIS-Developer@lists.osgeo.org" target="_blank">QGIS-Developer@lists.osgeo.org</a>
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a>
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a>
</pre>
</blockquote>
<pre cols="72">--
Johannes Kröger / GIS-Entwickler/-Berater
---------------------------------------------
Aufwind durch Wissen!
Web-Seminare und Online-Schulungen
bei der <a href="http://www.foss-academy.com" target="_blank">www.foss-academy.com</a>
---------------------------------------------
WhereGroup GmbH
c/o KK03 GmbH
Lange Reihe 29
20099 Hamburg
Germany
Tel: +49 (0)228 / 90 90 38 - 36
Fax: +49 (0)228 / 90 90 38 - 11
<a href="mailto:johannes.kroeger@wheregroup.com" target="_blank">johannes.kroeger@wheregroup.com</a>
<a href="http://www.wheregroup.com" target="_blank">www.wheregroup.com</a>
Geschäftsführer:
Olaf Knopp, Peter Stamm
Amtsgericht Bonn, HRB 9885
-------------------------------
</pre>
</div>
_______________________________________________<br>
QGIS-Developer mailing list<br>
<a href="mailto:QGIS-Developer@lists.osgeo.org" target="_blank">QGIS-Developer@lists.osgeo.org</a><br>
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a><br>
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-developer</a><br>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">Ian Turton<br></div>