[QGIS-Developer] [Qgis-user] Handling a large number of raster layers with Qgis architectural limitations

[Alexandre Neto]

Hi Patrick,

Sorry for asking, did you create pyramids (overlays) for your raster
"tiles"?

See more information about pyramids on QGIS official documentation:

https://docs.qgis.org/3.4/en/docs/user_manual/working_with_raster/raster_properties.html#pyramids-properties

To create pyramids on many files, it's probably better to use GDAL directly
to process all files in a folder:

https://gdal.org/programs/gdaladdo.html

My feeling is that the WMS service is doing that for you, and that's the
reason why it works well with a service.

Alexandre Neto
QGIS Support
www.qcooperative.net

A segunda, 18/11/2019, 09:11, Patrick Dunford <enzedrailmaps at gmail.com>
escreveu:

> Good day to all
>
> One of the user experiences I have had from using the Qgis software has
> been with projects using large numbers of raster tile layers. These layers
> are generally tiles that have a size of 4800x7200 pixels in GeoJPEG format
> and have either been downloaded directly from tile servers to these locally
> stored files, or created from downloaded tiles with other layers overlaid
> in Gimp projects.
>
> There appears to be some architectural limit in Qgis desktop software
> relating to either the total number of raster layer [files] in a project or
> to the total number of pixels in raster layer [files] in a project. This is
> unrelated to the number of layers or pixels currently enabled for display
> in the map canvas. In practice, the appearance of this limit is that it is
> kicking in long before the host computer's own physical resources are
> anywhere near fully engaged. Map digitising and editing is done on systems
> with 32 GB of physical memory (RAM) and 200 GB of SSD-based virtual memory
> (swap) and these systems are able to edit very large Gimp projects for user
> tile creation that often engage all of the system's physical memory and
> around 100 GB of the virtual memory without problems. But these types of
> numbers are in practice never seen with Qgis projects when the raster layer
> limit is being seen.
>
> The appearance of a raster layer limit is generally experienced in older
> versions of the software by layers being displayed on the canvas as
> garbage, and in newer versions by the software crashing. It will only start
> working again if raster layers are removed from the project. However, when
> layers are loaded from WMTS servers, no appearance of limitation is seen.
>
> The question to be answered, then, is which of any possible range of
> resolutions would be appropriate or useful to this predicament. With only
> limited understanding of the architectural design of the software, it would
> seem the following options exist:
>
>    1. File a bug report for the software concerning a possible issue with
>    the design of the product
>    2. Amalgamate smaller tiles into larger ones (e.g. 48 tiles at
>    4800x7200 can be put into one tile at 57600x28800). This only works if the
>    software issue is related to the number of file based tile layers and not
>    to the total number of pixels in those layers.
>    3. Post a feature request for sub-project capabilities. This would
>    allow a project that combines vector and raster layers, to be split into
>    one project containing the vector layers and a number of projects each of
>    which contains the vector project as a subproject and a certain subset of
>    all the raster layers that is smaller than the observable limit.
>    4. Set up my own local WMTS server to serve all the raster layers to
>    my map editing projects.
>    5. Explore the possibilities of preconfigured limits in the operating
>    system that may need to be increased to overcome file based layer limits in
>    projects (such as the NOFILES limit in Linux, currently set at 10,000 hard
>    and soft on map editing computers)
>
> Is anyone who is knowledgeable about the architecture of the Qgis desktop
> software able to comment with some detail about possible resolutions.
>
>
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20191118/523519d5/attachment.html>