[Qgis-user] Handling a large number of raster layers with Qgis architectural limitations

[Falk Huettmann]

Dear Colleagues, Patrick,

thanks, this is a VERY relevant issue, and might be the future concept for
GIS, of sorts (=more data volumes).
While I have no direct answer to your architecture question (sorry),
I have a sample dataset and application on the topic and can share it with
people as they see fit.
Here the manuscript and data, unpublished
https://www.earth-syst-sci-data-discuss.net/essd-2016-65/

The 104 GIS data layers (rasters) I can share via a google drive etc with
people on request and for more
progress and completion of the topic.

Arguably,
people are shopping around in the GIS world for a solution and platform
for such type of (massive multi) layer questions/analysis, and classic R
has usually a memory issue,
and supercomputing and 'the cloud' lacks the 'usual' GIS platform etc etc.
So here comes QGIS to the rescue, more or less
(consider Metadata creation in such projects also).
I am happy to learn.

Please keep me posted on this topic, offline as needed.

Thanks so much, very best
    Falk Huettmann

On Wed, Nov 27, 2019 at 6:35 PM Patrick Dunford <enzedrailmaps at gmail.com>
wrote:

> Hello,
>
> the issue is not with the performance drawing the raster, it is with the
> system only being able to have a certain number of file based rasters
> loaded in a project before it will crash.
> On 22/11/19 11:08 AM, Alexandre Neto wrote:
>
> AFAIK,
>
> WMST creates pyramids in cache as needed, hence it may do a better balance
> between disk usage and performance. You can control the number of pyramid
> layers you create and the compression format of it to save some space. But
> having raster files with 4000x4000 pixels with no pyramids will make a full
> read of the raster file to show even only a bit of it... That will make
> QGIS performance really poor.
>
> Thereºs this blog post from 2010 that does some comparison between several
> formats and their sizes and performances. It's in portuguese, but I think
> you can find ways of translating it.
>
>
> https://blog.viasig.com/2010/01/mosaicos-de-imagens-em-mapserver-com-gdal/
>
> Hope it helps
>
> Alexandre Neto
>
> On Thu, Nov 21, 2019 at 2:05 PM Patrick Dunford <enzedrailmaps at gmail.com>
> wrote:
>
>> Building pyramids gobbles up heaps of disk space. I tried pyramids on 157
>> MB layer, which created an extra file 1.6 GiB.
>>
>> I can't see the WMTS building lots of temporary file pyramids using up
>> disk
>> On 18/11/19 10:50 PM, Alexandre Neto wrote:
>>
>> Hi Patrick,
>>
>> Sorry for asking, did you create pyramids (overlays) for your raster
>> "tiles"?
>>
>> See more information about pyramids on QGIS official documentation:
>>
>>
>> https://docs.qgis.org/3.4/en/docs/user_manual/working_with_raster/raster_properties.html#pyramids-properties
>>
>> To create pyramids on many files, it's probably better to use GDAL
>> directly to process all files in a folder:
>>
>> https://gdal.org/programs/gdaladdo.html
>>
>> My feeling is that the WMS service is doing that for you, and that's the
>> reason why it works well with a service.
>>
>> Alexandre Neto
>> QGIS Support
>> www.qcooperative.net
>>
>> _______________________________________________
>> Qgis-user mailing list
>> Qgis-user at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20191127/a25b8236/attachment.html>