[QGIS-Developer] Handling a large number of raster layers with Qgis architectural limitations

[Patrick Dunford]

Good day to all

One of the user experiences I have had from using the Qgis software has 
been with projects using large numbers of raster tile layers. These 
layers are generally tiles that have a size of 4800x7200 pixels in 
GeoJPEG format and have either been downloaded directly from tile 
servers to these locally stored files, or created from downloaded tiles 
with other layers overlaid in Gimp projects.

There appears to be some architectural limit in Qgis desktop software 
relating to either the total number of raster layer [files] in a project 
or to the total number of pixels in raster layer [files] in a project. 
This is unrelated to the number of layers or pixels currently enabled 
for display in the map canvas. In practice, the appearance of this limit 
is that it is kicking in long before the host computer's own physical 
resources are anywhere near fully engaged. Map digitising and editing is 
done on systems with 32 GB of physical memory (RAM) and 200 GB of 
SSD-based virtual memory (swap) and these systems are able to edit very 
large Gimp projects for user tile creation that often engage all of the 
system's physical memory and around 100 GB of the virtual memory without 
problems. But these types of numbers are in practice never seen with 
Qgis projects when the raster layer limit is being seen.

The appearance of a raster layer limit is generally experienced in older 
versions of the software by layers being displayed on the canvas as 
garbage, and in newer versions by the software crashing. It will only 
start working again if raster layers are removed from the project. 
However, when layers are loaded from WMTS servers, no appearance of 
limitation is seen.

The question to be answered, then, is which of any possible range of 
resolutions would be appropriate or useful to this predicament. With 
only limited understanding of the architectural design of the software, 
it would seem the following options exist:

 1. File a bug report for the software concerning a possible issue with
    the design of the product
 2. Amalgamate smaller tiles into larger ones (e.g. 48 tiles at
    4800x7200 can be put into one tile at 57600x28800). This only works
    if the software issue is related to the number of file based tile
    layers and not to the total number of pixels in those layers.
 3. Post a feature request for sub-project capabilities. This would
    allow a project that combines vector and raster layers, to be split
    into one project containing the vector layers and a number of
    projects each of which contains the vector project as a subproject
    and a certain subset of all the raster layers that is smaller than
    the observable limit.
 4. Set up my own local WMTS server to serve all the raster layers to my
    map editing projects.
 5. Explore the possibilities of preconfigured limits in the operating
    system that may need to be increased to overcome file based layer
    limits in projects (such as the NOFILES limit in Linux, currently
    set at 10,000 hard and soft on map editing computers)

Is anyone who is knowledgeable about the architecture of the Qgis 
desktop software able to comment with some detail about possible 
resolutions.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20191118/b88e732a/attachment-0001.html>