[gdal-dev] Faster alternative to GetFeatureCount?

Mike mswope at gmail.com
Thu Jan 20 21:03:11 PST 2022


In the case if FileGDB, which driver you use to read the FileGDB makes a
big difference.

On Thu, Jan 20, 2022 at 8:59 AM Jon Morris <Jon.Morris at jbarisk.com> wrote:

> Hi Sean,
>
>
>
> That seems to be the best option. I was testing on Shapefile initially,
> but after I posted, I tried a memory layer and got quite different results.
> Ultimately, our output format is determined by clients (it's usually
> Shapefile or File GDB), so we'll just have to let them know there are
> performance implications with some formats.
>
>
>
> Thanks,
>
>
>
> Jon
>
>
> e: ​ *Jon.Morris at jbarisk.com* <Jon.Morris at jbarisk.com>
> t:  *+44 (0)1756 799919* <+44%20(0)1756%20799919>
> *www.jbarisk.com* <http://www.jbarisk.com/>
> [image: Facebook] <https://www.facebook.com/TheFloodPeople>
> [image: LinkedIn] <https://www.linkedin.com/company/jba-risk-management/>
> [image: Twitter] <https://twitter.com/JBARisk>
> [image: YouTube]
> <https://www.youtube.com/channel/UC0iatom2jYbW96voW0rlpCw>
>
>
> All JBA Risk Management's email messages contain confidential information
> and are intended only for the individual(s) named. If you are not the named
> addressee you should not disseminate, distribute or copy this e-mail.
> Please notify the sender immediately by email if you have received this
> email by mistake and delete this email from your system.
> JBA Risk Management Limited is registered in England, company number
> 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North
> Yorkshire, BD23 3FD, England.
>
> *From:* gdal-dev <gdal-dev-bounces at lists.osgeo.org> *On Behalf Of *Sean
> Gillies
> *Sent:* 20 January 2022 15:44
> *To:* gdal-dev at lists.osgeo.org
> *Subject:* Re: [gdal-dev] Faster alternative to GetFeatureCount?
>
>
>
> *CAUTION:* This email originated from outside of JBA and *contains one or
> more links and one or more attachments*. DO NOT click links or open
> attachments unless you recognise the sender's email address and are
> absolutely certain that the content is safe.
> See the Phishing page on IMS on SP for more information about how to spot
> and report suspicious messages.
>
> *WARNING: This email contains one or more attachments with links within
> the attachment(s)*. Please take extra care when handling the attachments.
>
> Hi Jon,
>
>
>
> The performance of GetFeatureCount, filters, and GetNextFeature depends a
> lot on the data format. If your data is in Postgres, for example, these
> operations are super fast because the database keeps the number of rows
> (features) in memory and has highly optimized views of selected rows. If
> your data is in a CSV file, GDAL has to read every line to count the number
> of features and when getting filtered features has to read all the lines in
> between the ones you want. See
> https://github.com/OSGeo/gdal/blob/master/ogr/ogrsf_frmts/csv/ogrcsvlayer.cpp#L1667
> .
>
>
>
> Can you switch to a more optimized format?
>
>
>
> On Thu, Jan 20, 2022 at 2:02 AM Jon Morris <Jon.Morris at jbarisk.com> wrote:
>
> I'm writing applications using the GDAL Python bindings and when I profile
> for performance, GetFeatureCount frequently comes out near the top. I'm
> often using it to check whether a spatial or attribute filter has returned
> any features and don't need the full count. When the layer contains
> millions of features, there would be a big performance improvement if we
> could exit the count early.
>
>
>
> Is there a better way of doing this? I've tried using GetNextFeature
> instead, but there must be quite a lot of overhead in that function as it
> is much slower. All I need to know is if the layer has has 0, 1 or >1
> features, I don't need the actual count. Can anyone suggest the fastest way
> of doing this in Python? I'm using GDAL 3.3.1 at the moment but could
> upgrade if there is new functionality that would help.
>
>
>
> Thanks,
>
>
>
> Jon
>
>
>
> *Jon Morris*
>
> *Software Developer*
>
>
>
>
>
> *e: *
>
> Jon.Morris at jbarisk.com
>
> *t: *
>
> +44 (0)1756 799919 <+44%20(0)1756%20799919>
>
> www.jbarisk.com
>
> [image: Facebook] <https://www.facebook.com/TheFloodPeople>
>
> [image: LinkedIn] <https://www.linkedin.com/company/jba-risk-management/>
>
> [image: Twitter] <https://twitter.com/JBARisk>
>
> [image: YouTube]
> <https://www.youtube.com/channel/UC0iatom2jYbW96voW0rlpCw>
>
>
>
>
>
>
>
> All JBA Risk Management's email messages contain confidential information
> and are intended only for the individual(s) named. If you are not the named
> addressee you should not disseminate, distribute or copy this e-mail.
> Please notify the sender immediately by email if you have received this
> email by mistake and delete this email from your system.
> JBA Risk Management Limited is registered in England, company number
> 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North
> Yorkshire, BD23 3FD, England.
>
>
>
> --
>
> Sean Gillies
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/b09e4bd0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 9886 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/b09e4bd0/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 651 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/b09e4bd0/attachment-0011.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 715 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/b09e4bd0/attachment-0012.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 966 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/b09e4bd0/attachment-0013.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 784 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/b09e4bd0/attachment-0014.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image633772.png
Type: image/png
Size: 9886 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/b09e4bd0/attachment-0015.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image076485.png
Type: image/png
Size: 651 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/b09e4bd0/attachment-0016.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image695600.png
Type: image/png
Size: 715 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/b09e4bd0/attachment-0017.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image601417.png
Type: image/png
Size: 966 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/b09e4bd0/attachment-0018.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image345894.png
Type: image/png
Size: 784 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220120/b09e4bd0/attachment-0019.png>


More information about the gdal-dev mailing list