[Qgis-user] reduce number of open files in pyqgis script

[Martin Landa]

Dear all,

I have a standalone pyqgis script which processes multiple layers stored in
a single GPKG. The first part of the script loops over layers, performs
clipping by extent and stores the result in a memory layer.  The problem
which I am facing is the increasing number of open files which easily reach
the OS limit. The sample script:

"""
import sys
import psutil, os
import gc

from qgis.core import QgsProviderRegistry, QgsVectorLayer, QgsRectangle,
QgsFeatureRequest, QgsWkbTypes, QgsFeature, QgsGeometry

def open_files_count():
    return len(process.open_files())

def clip_layer(layer: QgsVectorLayer, extent: QgsRectangle, layer_name:
str) -> QgsVectorLayer:
    extent_geom = QgsGeometry.fromRect(extent)
    clipped_layer = QgsVectorLayer(

f"{QgsWkbTypes.displayString(layer.wkbType())}?crs={layer.crs().authid()}",
        layer_name,
        "memory"
    )

    clipped_layer.dataProvider().addAttributes(layer.fields())
    clipped_layer.updateFields()

    for feature in
layer.getFeatures(QgsFeatureRequest().setFilterRect(extent)):
        geom = feature.geometry()
        if geom.intersects(extent_geom):
            clipped_feature = QgsFeature()
            clipped_feature.setGeometry(geom.intersection(extent_geom))
            clipped_feature.setAttributes(feature.attributes())
            clipped_layer.dataProvider().addFeature(clipped_feature)
            clipped_feature = None

    clipped_layer.commitChanges()

    return clipped_layer

process = psutil.Process(os.getpid())

gpkg_path = "/home/martin/geodata/ZABAGED20251006.gpkg"

layers = QgsProviderRegistry.instance().querySublayers(gpkg_path)

extent = QgsRectangle(-653547, -1065864, -653225, -1065600)
clipped_layers = []
for sublayer in layers:
    name = sublayer.name()
    layer = QgsVectorLayer(sublayer.uri(), 'ogr')

    clipped_layer = clip_layer(layer, extent, layer.name())

    del layer

    print(name, open_files_count())
    if clipped_layer.featureCount() == 0:
        del clipped_layer
        continue
    clipped_layers.append(clipped_layer)
"""

When launching the script number of open file is increasing regardless any
attempts to release data sources (del layer, layer = None, gc.collect(),
...):

ZeleznicniStaniceZastavka 3
ZeleznicniPrejezd_b 4
ZdrojPodzemnichVod 5
...
BazinaMocal 172
ArealZeleznicniStaniceZastavky 173
ArealUceloveZastavby 174

Please how to release data sources correctly to avoid increasing the number
of open files?

Thanks in advance, Martin

-- 
Martin Landa
https://geomatics.fsv.cvut.cz/en/employees/martin-landa/
https://gismentors.cz/mentors/landa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20251226/c034289f/attachment.htm>