[Qgis-user] Handling a large number of raster layers with Qgis architectural limitations

[Nicolas Cadieux]

Hi,

I ran into same problems when dealing with a few thousand files.  The idea was to load the very small shp files created by another process and then merge then.  I solved the problem by loading and merge it then I SAGA.  It generally loads everything into memory and probably does not keep a file handle once the file are open.  I believe this issue is created by Windows and not QGIS.  I remember being told to go on Linux as this maximum operating system file max can be modified with a script.

Nicolas

> Le 18 nov. 2019 à 04:59, Jésahel Benoist <djes1975 at gmail.com> a écrit :
> 
> From my experience, GeoTIFF has a long history and is a more
> appropriate format to handle multiple large rasters. As a container,
> it could handle misc compression format (JPEG an other), misc
> representation at different scales (resolution is not a problem), misc
> color modes with raster/vectorial alpha layer, and so on. In one of my
> projects I handle more than 400 raster files (4000x4000x32) without
> any problem.
> Of course, a better and final choice would be to tile everything, but
> it is sometimes difficult with older maps.
> 
>> Le lun. 18 nov. 2019 à 10:12, Patrick Dunford
>> <enzedrailmaps at gmail.com> a écrit :
>> 
>> Good day to all
>> 
>> One of the user experiences I have had from using the Qgis software has been with projects using large numbers of raster tile layers. These layers are generally tiles that have a size of 4800x7200 pixels in GeoJPEG format and have either been downloaded directly from tile servers to these locally stored files, or created from downloaded tiles with other layers overlaid in Gimp projects.
>> 
>> There appears to be some architectural limit in Qgis desktop software relating to either the total number of raster layer [files] in a project or to the total number of pixels in raster layer [files] in a project. This is unrelated to the number of layers or pixels currently enabled for display in the map canvas. In practice, the appearance of this limit is that it is kicking in long before the host computer's own physical resources are anywhere near fully engaged. Map digitising and editing is done on systems with 32 GB of physical memory (RAM) and 200 GB of SSD-based virtual memory (swap) and these systems are able to edit very large Gimp projects for user tile creation that often engage all of the system's physical memory and around 100 GB of the virtual memory without problems. But these types of numbers are in practice never seen with Qgis projects when the raster layer limit is being seen.
>> 
>> The appearance of a raster layer limit is generally experienced in older versions of the software by layers being displayed on the canvas as garbage, and in newer versions by the software crashing. It will only start working again if raster layers are removed from the project. However, when layers are loaded from WMTS servers, no appe