[fdo-internals] Counting rows in managed IFeatureReader

Traian Stanev traian.stanev at autodesk.com
Wed Apr 23 11:42:22 EDT 2008 

I was just talking about the general case, where there could be an arbitrary WHERE clause.

Yes, SelectAggregates with Count() is the solution that would give optimal results, and that was already discussed. Maksim didn't like that solution (because not all providers support it, according to him), so I just explained why using a regular Select is not a great idea.

Traian


> -----Original Message-----
> From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-
> bounces at lists.osgeo.org] On Behalf Of Robert Fortin
> Sent: Wednesday, April 23, 2008 11:37 AM
> To: FDO Internals Mail List
> Subject: RE: [fdo-internals] Counting rows in managed IFeatureReader
>
> Traian,
>
> Just want to make sure you are not suggesting 2 identical select to get
> the count and then process them.  This is a lot of back/forth and
> possibly memory consumption just to get the number of rows if you do it
> through a ReadNext() loop and a counter.
>
> The 1st select should be a selectaggregate call using function Count()
> where it is available.  This will be optimized for most provider, will
> avoid a full table scan and will limit the number of trip to the
> server.
> The 2nd is to process those rows as needed.
>
> RF
>
> -----Original Message-----
> From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-
> bounces at lists.osgeo.org] On Behalf Of Traian Stanev
> Sent: Wednesday, April 23, 2008 10:05 AM
> To: fdo-internals at lists.osgeo.org
> Subject: RE: [fdo-internals] Counting rows in managed IFeatureReader
>
>
> Hi Maksim,
>
> In general (and you are looking for a general solution) it is *not
> possible* to know how many results are returned by an FdoIFeatureReader
> in advance. The reason why it is *impossible* is because for some FDO
> providers, the select query is executed incrementally, as you are
> reading the reader. When you execute the Select command, it does not
> actually execute your query, if it involves a table scan. So it has no
> way of knowing how many things are returned.
>
> Is it clear now?
>
> To get the count the way you want, you will need to do two identical
> selects -- one where you just call ReadNext() to count how many items
> you get back and another, to do the processing on those items.
>
>
> Traian
>
>
> > -----Original Message-----
> > From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-
> > bounces at lists.osgeo.org] On Behalf Of Maksim Sestic
> > Sent: Wednesday, April 23, 2008 9:21 AM
> > To: fdo-internals at lists.osgeo.org
> > Subject: RE: [fdo-internals] Counting rows in managed IFeatureReader
> >
> >
> > Hi Orest,
> >
> > Actually, I was thinking of Count() avoiding duplicate selects by
> > cloning
> > once-instantiated internal reader object, saving a roadtrip to server
> > (which
> > is, naturally, an overkill). I'm already doing it on managed level,
> but
> > I'm
> > not sure how managed object clones relate to their unmanaged twins
> > underneath (you know, managed bindings disposing patterns and other
> > obscure
> > stuff) :-) Why do you think that another instance (not reference) of
> a
> > reader, a copy of it actually, would affect the original?
> >
> > I thought it would be nice to be able to inspect total count of
> > features
> > being fetched via ISelect, before processing actual reader contents.
> > Another
> > thing - ISelect is supported by every provider, and I need a generic
> > solution. I'm using it heavily during both reader instance pre-
> > processing
> > and in-processing (i.e. progress bar, counters)...
> >
> > Regards,
> > Maksim Sestic
> >
> >
> >
> > Orest Halustchak wrote:
> > >
> > > Hi Maksim,
> > >
> > > I think the point around performance issues with Count() on the
> > feature
> > > reader is not related to side effect if it isn't used but that if
> it
> > is
> > > used it may be much less efficient than using Count() with Select
> > > Aggregates which may use a native Count() implementation. Selecting
> > all
> > > the records just to get a count and then selecting them all over
> > again is
> > > duplicated processing.
> > >
> > > Making an internal copy of a reader may be more of an issue than a
> > simple
> > > clone of that reader object. The reader is generated as a result of
> > the
> > > execution of a select. If you start reading through the results of
> > the
> > > reader, to be able to start reading again, in most cases it means
> the
> > > provider needs to re-execute the select (exception may be providers
> > that
> > > support a scrollable reader, but not all have that sort of
> > capability).
> > > You can re-execute the select yourself and will get a separate
> > reader.
> > >
> > > Could you explain your application scenario where you need this
> > ability
> > > and maybe we can think of other alternatives?
> > >
> > > Thanks,
> > > Orest.
> > >
> > > -----Original Message-----
> > > From: fdo-internals-bounces at lists.osgeo.org
> > > [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Maksim
> > Sestic
> > > Sent: Tuesday, April 22, 2008 8:02 AM
> > > To: fdo-internals at lists.osgeo.org
> > > Subject: RE: [fdo-internals] Counting rows in managed
> IFeatureReader
> > >
> > >
> > > Hi Barbara,
> > >
> > > I'm aware of it, but Count would not get calculated on
> IFeatureReader
> > > instantiation (i.e. via ISelect.Execute). The property would act as
> a
> > > function, when explicitly called (get). If called, it should create
> > an
> > > internal copy of underlying reader, traversing it to return total
> > number
> > > of
> > > feature instances. If it's not calculated on IFeatureReader
> > instantiation
> > > then there're no performance side effects. A private member could
> be
> > used
> > > for storing once-calculated features count for faster consequent
> > calls to
> > > Count() function of the same instance of IFeatureReader.
> > >
> > > Now, why IsEmpty when there's a Count? Because IsEmpty traverses
> only
> > the
> > > first record of possibly many records, returning False if ReadNext
> > doesn't
> > > return Nothing. IsEmpty() works on the same premises as above
> > described
> > > Count() mechanism.
> > >
> > > So user can chain calls to get the results with less overhead:
> > >
> > > If Not reader.IsEmpty
> > >    Dim cnt As Integer = reader.Count
> > > ...
> > > End If
> > >
> > > I think that it's better to have it implemented on unmanaged tier,
> > since
> > > doing it using managed bindings may sometimes lead to unexpected
> > results.
> > >
> > > Robert's proposal is OK, but the downside is that SelectAggregates
> > can't
> > > lock the records being fetched.
> > >
> > > Regards,
> > > Maksim Sestic
> > >
> > >
> > >
> > > Barbara Zoladek wrote:
> > >>
> > >> Maksim,
> > >> Adding Count to IFeatureReader will have performance side effects.
> > There
> > >> is no way to find out the number of rows without either fetching
> all
> > the
> > >> rows or explicitly executing something like 'select count(*)' .
> > >>
> > >> Robert's suggestion looks better.
> > >>
> > >> Thanks,
> > >> Barbara.
> > >>
> > >> -----Original Message-----
> > >> From: fdo-internals-bounces at lists.osgeo.org
> > >> [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Maksim
> > Sestic
> > >> Sent: Thursday, April 17, 2008 2:56 AM
> > >> To: fdo-internals at lists.osgeo.org
> > >> Subject: RE: [fdo-internals] Counting rows in managed
> IFeatureReader
> > >>
> > >>
> > >> Thanks for the tip, Robert. I need to use ISelect because of
> locking
> > >> (which
> > >> is not supported by ISelectAggregates).
> > >>
> > >> I'd suggest that generic IFeatureReader (neverthless of provider
> > >> involved)
> > >> expose two additional readonly properties:
> > >>
> > >> 1) Count - creates internal copy of a reader and traverses it,
> > returning
> > >> total number of feature instances found in it.
> > >>
> > >> 2) IsEmpty - boolean returning True if reader has no features
> > instances
> > >> in
> > >> it, False otherwise.
> > >>
> > >> Regards,
> > >> Maksim Sestic
> > >>
> > >>
> > >>
> > >> Robert Fortin wrote:
> > >>>
> > >>> Maksim,
> > >>>
> > >>> You can use the IselectAggregate command with the COUNT function
> to
> > >>> retrieve the number of rows for a given class.
> > >>> This will ensure that you use the optimize implementation of the
> > native
> > >>> data source to retrieve the number of row where it is available.
> > >>>
> > >>> Robert Fortin
> > >>>
> > >>> -----Original Message-----
> > >>> From: fdo-internals-bounces at lists.osgeo.org
> > >>> [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of
> Maksim
> > >>> Sestic
> > >>> Sent: Wednesday, April 16, 2008 6:11 AM
> > >>> To: fdo-internals at lists.osgeo.org
> > >>> Subject: [fdo-internals] Counting rows in managed IFeatureReader
> > >>>
> > >>>
> > >>> Hi all,
> > >>>
> > >>> Since IFeatureReader doesn't provide Count property for total
> > number of
> > >>> records in it, what's the most "proper" way to get it out from
> > managed
> > >>> IFeatureReader? I'm asking this because of necessity to
> explicitly
> > close
> > >>> a
> > >>> reder after it's returned from, say, ISelect command. How do I
> get
> > a
> > >>> safe
> > >>> copy of instantiated managed IFeatureReader object to count it's
> > >>> properties,
> > >>> without affecting the original one?
> > >>>
> > >>> It would be a no brainer if IFeatureReader was fully implemented
> in
> > >>> .NET,
> > >>> but managed bindings always make me wonder if unmanaged resources
> > get
> > >>> disposed in a proper way.
> > >>>
> > >>> Regards,
> > >>> Maksim Sestic
> > >>> --
> > >>> View this message in context:
> > >>> http://www.nabble.com/Counting-rows-in-managed-IFeatureReader-
> > tp16720338s18162p16720338.html
> > >>> Sent from the fdo-internals mailing list archive at Nabble.com.
> > >>>
> > >>> _______________________________________________
> > >>> fdo-internals mailing list
> > >>> fdo-internals at lists.osgeo.org
> > >>> http://lists.osgeo.org/mailman/listinfo/fdo-internals
> > >>> _______________________________________________
> > >>> fdo-internals mailing list
> > >>> fdo-internals at lists.osgeo.org
> > >>> http://lists.osgeo.org/mailman/listinfo/fdo-internals
> > >>>
> > >>>
> > >>
> > >> --
> > >> View this message in context:
> > >> http://www.nabble.com/Counting-rows-in-managed-IFeatureReader-
> > tp16720338s18162p16739953.html
> > >> Sent from the fdo-internals mailing list archive at Nabble.com.
> > >>
> > >> _______________________________________________
> > >> fdo-internals mailing list
> > >> fdo-internals at lists.osgeo.org
> > >> http://lists.osgeo.org/mailman/listinfo/fdo-internals
> > >> _______________________________________________
> > >> fdo-internals mailing list
> > >> fdo-internals at lists.osgeo.org
> > >> http://lists.osgeo.org/mailman/listinfo/fdo-internals
> > >>
> > >>
> > >
> > > --
> > > View this message in context:
> > > http://www.nabble.com/Counting-rows-in-managed-IFeatureReader-
> > tp16720338s18162p16823821.html
> > > Sent from the fdo-internals mailing list archive at Nabble.com.
> > >
> > > _______________________________________________
> > > fdo-internals mailing list
> > > fdo-internals at lists.osgeo.org
> > > http://lists.osgeo.org/mailman/listinfo/fdo-internals
> > > _______________________________________________
> > > fdo-internals mailing list
> > > fdo-internals at lists.osgeo.org
> > > http://lists.osgeo.org/mailman/listinfo/fdo-internals
> > >
> > >
> >
> > --
> > View this message in context: http://www.nabble.com/Counting-rows-in-
> > managed-IFeatureReader-tp16720338p16834596.html
> > Sent from the FDO Internals mailing list archive at Nabble.com.
> >
> > _______________________________________________
> > fdo-internals mailing list
> > fdo-internals at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/fdo-internals
> _______________________________________________
> fdo-internals mailing list
> fdo-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/fdo-internals
> _______________________________________________
> fdo-internals mailing list
> fdo-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/fdo-internals

More information about the fdo-internals mailing list