[Qgis-user] Handling a large number of raster layers with Qgis architectural limitations

[Jésahel Benoist]

>From my experience, GeoTIFF has a long history and is a more
appropriate format to handle multiple large rasters. As a container,
it could handle misc compression format (JPEG an other), misc
representation at different scales (resolution is not a problem), misc
color modes with raster/vectorial alpha layer, and so on. In one of my
projects I handle more than 400 raster files (4000x4000x32) without
any problem.
Of course, a better and final choice would be to tile everything, but
it is sometimes difficult with older maps.

Le lun. 18 nov. 2019 à 10:12, Patrick Dunford
<enzedrailmaps at gmail.com> a écrit :
>
> Good day to all
>
> One of the user experiences I have had from using the Qgis software has been with projects using large numbers of raster tile layers. These layers are generally tiles that have a size of 4800x7200 pixels in GeoJPEG format and have either been downloaded directly from tile servers to these locally stored files, or created from downloaded tiles with other layers overlaid in Gimp projects.
>
> There appears to be some architectural limit in Qgis desktop software relating to either the total number of raster layer [files] in a project or to the total number of pixels in raster layer [files] in a project. This is unrelated to the number of layers or pixels currently enabled for display in the map canvas. In practice, the appearance of this limit is that it is kicking in long before the host computer's own physical resources are anywhere near fully engaged. Map digitising and editing is done on systems with 32 GB of physical memory (RAM) and 200 GB of SSD-based virtual memory (swap) and these systems are able to edit very large Gimp projects for user tile creation that often engage all of the system's physical memory and around 100 GB of the virtual memory without problems. But these types of numbers are in practice never seen with Qgis projects when the raster layer limit is being seen.
>
> The appearance of a raster layer limit is generally experienced in older versions of the software by layers being displayed on the canvas as garbage, and in newer versions by the software crashing. It will only start working again if raster layers are removed from the project. However, when layers are loaded from WMTS servers, no appearance of limitation is seen.
>
> The question to be answered, then, is which of any possible range of resolutions would be appropriate or useful to this predicament. With only limited understanding of the architectural design of the software, it would seem the following options exist:
>
> File a bug report for the software concerning a possible issue with the design of the product
> Amalgamate smaller tiles into larger ones (e.g. 48 tiles at 4800x7200 can be put into one tile at 57600x28800). This only works if the software issue is related to the number of file based tile layers and not to the total number of pixels in those layers.
> Post a feature request for sub-project capabilities. This would allow a project that combines vector and raster layers, to be split into one project containing the vector layers and a number of projects each of which contains the vector project as a subproject and a certain subset of all the raster layers that is smaller than the observable limit.
> Set up my own local WMTS server to serve all the raster layers to my map editing projects.
> Explore the possibilities of preconfigured limits in the operating system that may need to be increased to overcome file based layer limits in projects (such as the NOFILES limit in Linux, currently set at 10,000 hard and soft on map editing computers)
>
> Is anyone who is knowledgeable about the architecture of the Qgis desktop software able to comment with some detail about possible resolutions.
>
>
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user