[Qgis-user] reduce number of open files in pyqgis script

[Ludwig Kniprath]

Hi Martin,

I just fiddled with QgsProviderRegistry and noticed, that 
"querySublayers" returns a bunch of GPKG-internal-views and -indices, 
that are processed in your script, although not containing any clippable 
vector data.

Workaround 1: Use osgeo.ogr.Open() to get a list of real 
vector-sub-layers and reduce the number of layer-creations, 
clip_layer-calls and open files

Workaround 2: Use osge.ogr to do the layer-creation and clip 
(sadly beyond my knowledge...)

"""
gpkg_path = "/home/martin/geodata/ZABAGED20251006.gpkg"

# variant with QgsProviderRegistry.instance().querySublayers(...)
gpkg_sub_layer_names = [gpkg_sub_layer_details.name() for 
gpkg_sub_layer_details in 
QgsProviderRegistry.instance().querySublayers(gpkg_path)]
print(len(gpkg_sub_layer_names))
print(gpkg_sub_layer_names)

# variant with osgeo.ogr.Open(...)
from osgeo import ogr
gpkg_real_sub_layer_names = [gpkg_sub_layer.GetName() for gpkg_sub_layer 
in ogr.Open(gpkg_path)]

print(len(gpkg_real_sub_layer_names))
print(gpkg_real_sub_layer_names)
"""

hth

Ludwig

Am 26.12.25 um 12:24 schrieb Martin Landa via QGIS-User:
> Dear all,
>
> I have a standalone pyqgis script which processes multiple layers 
> stored in a single GPKG. The first part of the script loops over 
> layers, performs clipping by extent and stores the result in a 
> memory layer.  The problem which I am facing is the increasing number 
> of open files which easily reach the OS limit. The sample script:
>
> """
> import sys
> import psutil, os
> import gc
>
> from qgis.core import QgsProviderRegistry, QgsVectorLayer, 
> QgsRectangle, QgsFeatureRequest, QgsWkbTypes, QgsFeature, QgsGeometry
>
> def open_files_count():
>     return len(process.open_files())
>
> def clip_layer(layer: QgsVectorLayer, extent: QgsRectangle, 
> layer_name: str) -> QgsVectorLayer:
>     extent_geom = QgsGeometry.fromRect(extent)
>     clipped_layer = QgsVectorLayer(
> f"{QgsWkbTypes.displayString(layer.wkbType())}?crs={layer.crs().authid()}",
>         layer_name,
>         "memory"
>     )
>
>     clipped_layer.dataProvider().addAttributes(layer.fields())
>     clipped_layer.updateFields()
>
>     for feature in 
> layer.getFeatures(QgsFeatureRequest().setFilterRect(extent)):
>         geom = feature.geometry()
>         if geom.intersects(extent_geom):
>             clipped_feature = QgsFeature()
> clipped_feature.setGeometry(geom.intersection(extent_geom))
> clipped_feature.setAttributes(feature.attributes())
> clipped_layer.dataProvider().addFeature(clipped_feature)
>             clipped_feature = None
>
>     clipped_layer.commitChanges()
>
>     return clipped_layer
>
> process = psutil.Process(os.getpid())
>
> gpkg_path = "/home/martin/geodata/ZABAGED20251006.gpkg"
>
> layers = QgsProviderRegistry.instance().querySublayers(gpkg_path)
>
> extent = QgsRectangle(-653547, -1065864, -653225, -1065600)
> clipped_layers = []
> for sublayer in layers:
>     name = sublayer.name <http://sublayer.name>()
>     layer = QgsVectorLayer(sublayer.uri(), 'ogr')
>
>     clipped_layer = clip_layer(layer, extent, layer.name 
> <http://layer.name>())
>
>     del layer
>
>     print(name, open_files_count())
>     if clipped_layer.featureCount() == 0:
>         del clipped_layer
>         continue
>     clipped_layers.append(clipped_layer)
> """
>
> When launching the script number of open file is increasing regardless 
> any attempts to release data sources (del layer, layer = None, 
> gc.collect(), ...):
>
> ZeleznicniStaniceZastavka 3
> ZeleznicniPrejezd_b 4
> ZdrojPodzemnichVod 5
> ...
> BazinaMocal 172
> ArealZeleznicniStaniceZastavky 173
> ArealUceloveZastavby 174
>
> Please how to release data sources correctly to avoid increasing the 
> number of open files?
>
> Thanks in advance, Martin
>
> -- 
> Martin Landa
> https://geomatics.fsv.cvut.cz/en/employees/martin-landa/
> https://gismentors.cz/mentors/landa
>
> _______________________________________________
> QGIS-User mailing list
> QGIS-User at lists.osgeo.org
> List info:https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe:https://lists.osgeo.org/mailman/listinfo/qgis-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20251228/84f59961/attachment-0001.htm>