[Qgis-user] File Descriptor Limits in Linux

[Patrick Dunford]

Good day.

The steps listed below are for the configuration of hard and soft file 
limits.

The listed configuration for systemd only configures the hard limit. It 
is apparent that some additional process is required to configure the 
soft limit in this instance. This has not been observed before because 
the prlimit command output in both Debian 12 and Debian 13 shows that 
the soft limit is set by default to a value of 1024. Therefore issues 
are only being encountered when there are more than 1000 layers in a 
Qgis project.

I use qgis to process large numbers of raster layers that are loaded 
from files and at time there are more than 1000 of these layers in a 
particular series therefore this situation often arises for me.

I will be investigating how to configure the soft limit on these 
operating systems.

On 17/09/2024 01:31, Patrick Dunford wrote:
> Good day
>
> For many years it has been known that applications such as Qgis 
> running under the Linux operating system have been subject to 
> configuration of maximum number of file descriptors for open files. 
> Generally the default is set to something like 1024 and will affect 
> Qgis projects with more than 1024 layers.
>
> There are currently two limits in Linux that may affect users of Qgis 
> with a large number of layers in their project. These can be checked 
> using the ulimit -n command which will display the maximum number of 
> open files permitted in the current configuration.
>
> 1. PAM session limits are configured in the file 
> /etc/security/limits.conf
>
> 2. On some distros, the limits also have to be separately configured 
> for systemd.
>
> Ubuntu up to and including 22.04 requires only step 1 to be completed.
>
> Debian 12 on the other hand requires step 2 to be completed. It is not 
> clear if this is a requirement on any other releases of Debian. 
> However experience suggests this step has not been a requirement on 
> most earlier Debian releases.
>
> The specifics as far as I can determine for step 2 suggest that a 
> .conf file is placed in the following path:
>
> /etc/systemd/user.conf.d
>
> This file can be any name chosen by a user (e.g. limits.conf). The 
> file was copied from the user.conf file placed in /etc/systemd and it 
> is also the case that editing this user.conf file in its default path 
> would have the same effect, however the subdirectory usage is 
> recommended.
>
> This file must at a minimum contain the following:
>
> [Manager]
>
> DefaultLimitNOFILE=xxxxxx
>
>
> This is a per-user limit and it does not apply to the root user. There 
> is a different procedure for the root user, which I do not have 
> documentation for. The result of ulimit -n run under a root shell will 
> display a different result from running it as a non-root user.
>
>