[gdal-dev] Faster alternative to GetFeatureCount?

Thu Jan 20 07:59:43 PST 2022

Hi Sean,

That seems to be the best option. I was testing on Shapefile initially, but after I posted, I tried a memory layer and got quite different results. Ultimately, our output format is determined by clients (it's usually Shapefile or File GDB), so we'll just have to let them know there are performance implications with some formats.

Thanks,

Jon

e: Jon.Morris at jbarisk.com
t: +44 (0)1756 799919
www.jbarisk.com
All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. 
JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire, BD23 3FD, England.
From: gdal-dev <gdal-dev-bounces at lists.osgeo.org> On Behalf Of Sean Gillies
Sent: 20 January 2022 15:44
To: gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] Faster alternative to GetFeatureCount?

CAUTION: This email originated from outside of JBA and contains one or more links and one or more attachments. DO NOT click links or open attachments unless you recognise the sender's email address and are absolutely certain that the content is safe.
See the Phishing page on IMS on SP for more information about how to spot and report suspicious messages.
WARNING: This email contains one or more attachments with links within the attachment(s). Please take extra care when handling the attachments.
Hi Jon,

The performance of GetFeatureCount, filters, and GetNextFeature depends a lot on the data format. If your data is in Postgres, for example, these operations are super fast because the database keeps the number of rows (features) in memory and has highly optimized views of selected rows. If your data is in a CSV file, GDAL has to read every line to count the number of features and when getting filtered features has to read all the lines in between the ones you want. See https://github.com/OSGeo/gdal/blob/master/ogr/ogrsf_frmts/csv/ogrcsvlayer.cpp#L1667.

Can you switch to a more optimized format?

On Thu, Jan 20, 2022 at 2:02 AM Jon Morris <Jon.Morris at jbarisk.com<mailto:Jon.Morris at jbarisk.com>> wrote:
I'm writing applications using the GDAL Python bindings and when I profile for performance, GetFeatureCount frequently comes out near the top. I'm often using it to check whether a spatial or attribute filter has returned any features and don't need the full count. When the layer contains millions of features, there would be a big performance improvement if we could exit the count early.

Is there a better way of doing this? I've tried using GetNextFeature instead, but there must be quite a lot of overhead in that function as it is much slower. All I need to know is if the layer has has 0, 1 or >1 features, I don't need the actual count. Can anyone suggest the fastest way of doing this in Python? I'm using GDAL 3.3.1 at the moment but could upgrade if there is new functionality that would help.

Thanks,

Jon

Jon Morris
Software Developer

e:
Jon.Morris at jbarisk.com<mailto:Jon.Morris at jbarisk.com>
t:
+44 (0)1756 799919<tel:+44%20(0)1756%20799919>
www.jbarisk.com<http://www.jbarisk.com/>
[cid:image001.png at 01D80E16.BC39E5D0]
[Facebook]<https://www.facebook.com/TheFloodPeople>
[LinkedIn]<https://www.linkedin.com/company/jba-risk-management/>
[Twitter]<https://twitter.com/JBARisk>
[YouTube]<https://www.youtube.com/channel/UC0iatom2jYbW96voW0rlpCw>

All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.
JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire, BD23 3FD, England.

--
Sean Gillies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/69617c3c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 9886 bytes
Desc: image001.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/69617c3c/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 651 bytes
Desc: image002.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/69617c3c/attachment-0011.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 715 bytes
Desc: image003.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/69617c3c/attachment-0012.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 966 bytes
Desc: image004.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/69617c3c/attachment-0013.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 784 bytes
Desc: image005.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/69617c3c/attachment-0014.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image633772.png
Type: image/png
Size: 9886 bytes
Desc: image633772.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/69617c3c/attachment-0015.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image076485.png
Type: image/png
Size: 651 bytes
Desc: image076485.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/69617c3c/attachment-0016.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image695600.png
Type: image/png
Size: 715 bytes
Desc: image695600.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/69617c3c/attachment-0017.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image601417.png
Type: image/png
Size: 966 bytes
Desc: image601417.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/69617c3c/attachment-0018.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image345894.png
Type: image/png
Size: 784 bytes
Desc: image345894.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/69617c3c/attachment-0019.png>