[fdo-internals] Counting rows in managed IFeatureReader

Robert Fortin robert.fortin at autodesk.com
Wed Apr 23 11:37:27 EDT 2008 

Traian,

Just want to make sure you are not suggesting 2 identical select to get the count and then process them.  This is a lot of back/forth and possibly memory consumption just to get the number of rows if you do it through a ReadNext() loop and a counter.

The 1st select should be a selectaggregate call using function Count() where it is available.  This will be optimized for most provider, will avoid a full table scan and will limit the number of trip to the server.
The 2nd is to process those rows as needed.

RF

-----Original Message-----
From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian Stanev
Sent: Wednesday, April 23, 2008 10:05 AM
To: fdo-internals at lists.osgeo.org
Subject: RE: [fdo-internals] Counting rows in managed IFeatureReader


Hi Maksim,

In general (and you are looking for a general solution) it is *not possible* to know how many results are returned by an FdoIFeatureReader in advance. The reason why it is *impossible* is because for some FDO providers, the select query is executed incrementally, as you are reading the reader. When you execute the Select command, it does not actually execute your query, if it involves a table scan. So it has no way of knowing how many things are returned.

Is it clear now?

To get the count the way you want, you will need to do two identical selects -- one where you just call ReadNext() to count how many items you get back and another, to do the processing on those items.


Traian


> -----Original Message-----
> From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-
> bounces at lists.osgeo.org] On Behalf Of Maksim Sestic
> Sent: Wednesday, April 23, 2008 9:21 AM
> To: fdo-internals at lists.osgeo.org
> Subject: RE: [fdo-internals] Counting rows in managed IFeatureReader
>
>
> Hi Orest,
>
> Actually, I was thinking of Count() avoiding duplicate selects by
> cloning
> once-instantiated internal reader object, saving a roadtrip to server
> (which
> is, naturally, an overkill). I'm already doing it on managed level, but
> I'm
> not sure how managed object clones relate to their unmanaged twins
> underneath (you know, managed bindings disposing patterns and other
> obscure
> stuff) :-) Why do you think that another instance (not reference) of a
> reader, a copy of it actually, would affect the original?
>
> I thought it would be nice to be able to inspect total count of
> features
> being fetched via ISelect, before processing actual reader contents.
> Another
> thing - ISelect is supported by every provider, and I need a generic
> solution. I'm using it heavily during both reader instance pre-
> processing
> and in-processing (i.e. progress bar, counters)...
>
> Regards,
> Maksim Sestic
>
>
>
> Orest Halustchak wrote:
> >
> > Hi Maksim,
> >
> > I think the point around performance issues with Count() on the
> feature
> > reader is not related to side effect if it isn't used but that if it
> is
> > used it may be much less efficient than using Count() with Select
> > Aggregates which may use a native Count() implementation. Selecting
> all
> > the records just to get a count and then selecting them all over
> again is
> > duplicated processing.
> >
> > Making an internal copy of a reader may be more of an issue than a
> simple
> > clone of that reader object. The reader is generated as a result of
> the
> > execution of a select. If you start reading through the results of
> the
> > reader, to be able to start reading again, in most cases it means the
> > provider needs to re-execute the select (exception may be providers
> that
> > support a scrollable reader, but not all have that sort of
> capability).
> > You can re-execute the select yourself and will get a separate
> reader.
> >
> > Could you explain your application scenario where you need this
> ability
> > and maybe we can think of other alternatives?
> >
> > Thanks,
> > Orest.
> >
> > -----Original Message-----
> > From: fdo-internals-bounces at lists.osgeo.org
> > [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Maksim
> Sestic
> > Sent: Tuesday, April 22, 2008 8:02 AM
> > To: fdo-internals at lists.osgeo.org
> > Subject: RE: [fdo-internals] Counting rows in managed IFeatureReader
> >
> >
> > Hi Barbara,
> >
> > I'm aware of it, but Count would not get calculated on IFeatureReader
> > instantiation (i.e. via ISelect.Execute). The property would act as a
> > function, when explicitly called (get). If called, it should create
> an
> > internal copy of underlying reader, traversing it to return total
> number
> > of
> > feature instances. If it's not calculated on IFeatureReader
> instantiation
> > then there're no performance side effects. A private member could be
> used
> > for storing once-calculated features count for faster consequent
> calls to
> > Count() function of the same instance of IFeatureReader.
> >
> > Now, why IsEmpty when there's a Count? Because IsEmpty traverses only
> the
> > first record of possibly many records, returning False if ReadNext
> doesn't
> > return Nothing. IsEmpty() works on the same premises as above
> described
> > Count() mechanism.
> >
> > So user can chain calls to get the results with less overhead:
> >
> > If Not reader.IsEmpty
> >    Dim cnt As Integer = reader.Count
> > ...
> > End If
> >
> > I think that it's better to have it implemented on unmanaged tier,
> since
> > doing it using managed bindings may sometimes lead to unexpected
> results.
> >
> > Robert's proposal is OK, but the downside is that SelectAggregates
> can't
> > lock the records being fetched.
> >
> > Regards,
> > Maksim Sestic
> >
> >
> >
> > Barbara Zoladek wrote:
> >>
> >> Maksim,
> >> Adding Count to IFeatureReader will have performance side effects.
> There
> >> is no way to find out the number of rows without either fetching all
> the
> >> rows or explicitly executing something like 'select count(*)' .
> >>
> >> Robert's suggestion looks better.
> >>
> >> Thanks,
> >> Barbara.
> >>
> >> -----Original Message-----
> >> From: fdo-internals-bounces at lists.osgeo.org
> >> [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Maksim
> Sestic
> >> Sent: Thursday, April 17, 2008 2:56 AM
> >> To: fdo-internals at lists.osgeo.org
> >> Subject: RE: [fdo-internals] Counting rows in managed IFeatureReader
> >>
> >>
> >> Thanks for the tip, Robert. I need to use ISelect because of locking
> >> (which
> >> is not supported by ISelectAggregates).
> >>
> >> I'd suggest that generic IFeatureReader (neverthless of provider
> >> involved)
> >> expose two additional readonly properties:
> >>
> >> 1) Count - creates internal copy of a reader and traverses it,
> returning
> >> total number of feature instances found in it.
> >>
> >> 2) IsEmpty - boolean returning True if reader has no features
> instances
> >> in
> >> it, False otherwise.
> >>
> >> Regards,
> >> Maksim Sestic
> >>
> >>
> >>
> >> Robert Fortin wrote:
> >>>
> >>> Maksim,
> >>>
> >>> You can use the IselectAggregate command with the COUNT function to
> >>> retrieve the number of rows for a given class.
> >>> This will ensure that you use the optimize implementation of the
> native
> >>> data source to retrieve the number of row where it is available.
> >>>
> >>> Robert Fortin
> >>>
> >>> -----Original Message-----
> >>> From: fdo-internals-bounces at lists.osgeo.org
> >>> [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Maksim
> >>> Sestic
> >>> Sent: Wednesday, April 16, 2008 6:11 AM
> >>> To: fdo-internals at lists.osgeo.org
> >>> Subject: [fdo-internals] Counting rows in managed IFeatureReader
> >>>
> >>>
> >>> Hi all,
> >>>
> >>> Since IFeatureReader doesn't provide Count property for total
> number of
> >>> records in it, what's the most "proper" way to get it out from
> managed
> >>> IFeatureReader? I'm asking this because of necessity to explicitly
> close
> >>> a
> >>> reder after it's returned from, say, ISelect command. How do I get
> a
> >>> safe
> >>> copy of instantiated managed IFeatureReader object to count it's
> >>> properties,
> >>> without affecting the original one?
> >>>
> >>> It would be a no brainer if IFeatureReader was fully implemented in
> >>> .NET,
> >>> but managed bindings always make me wonder if unmanaged resources
> get
> >>> disposed in a proper way.
> >>>
> >>> Regards,
> >>> Maksim Sestic
> >>> --
> >>> View this message in context:
> >>> http://www.nabble.com/Counting-rows-in-managed-IFeatureReader-
> tp16720338s18162p16720338.html
> >>> Sent from the fdo-internals mailing list archive at Nabble.com.
> >>>
> >>> _______________________________________________
> >>> fdo-internals mailing list
> >>> fdo-internals at lists.osgeo.org
> >>> http://lists.osgeo.org/mailman/listinfo/fdo-internals
> >>> _______________________________________________
> >>> fdo-internals mailing list
> >>> fdo-internals at lists.osgeo.org
> >>> http://lists.osgeo.org/mailman/listinfo/fdo-internals
> >>>
> >>>
> >>
> >> --
> >> View this message in context:
> >> http://www.nabble.com/Counting-rows-in-managed-IFeatureReader-
> tp16720338s18162p16739953.html
> >> Sent from the fdo-internals mailing list archive at Nabble.com.
> >>
> >> _______________________________________________
> >> fdo-internals mailing list
> >> fdo-internals at lists.osgeo.org
> >> http://lists.osgeo.org/mailman/listinfo/fdo-internals
> >> _______________________________________________
> >> fdo-internals mailing list
> >> fdo-internals at lists.osgeo.org
> >> http://lists.osgeo.org/mailman/listinfo/fdo-internals
> >>
> >>
> >
> > --
> > View this message in context:
> > http://www.nabble.com/Counting-rows-in-managed-IFeatureReader-
> tp16720338s18162p16823821.html
> > Sent from the fdo-internals mailing list archive at Nabble.com.
> >
> > _______________________________________________
> > fdo-internals mailing list
> > fdo-internals at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/fdo-internals
> > _______________________________________________
> > fdo-internals mailing list
> > fdo-internals at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/fdo-internals
> >
> >
>
> --
> View this message in context: http://www.nabble.com/Counting-rows-in-
> managed-IFeatureReader-tp16720338p16834596.html
> Sent from the FDO Internals mailing list archive at Nabble.com.
>
> _______________________________________________
> fdo-internals mailing list
> fdo-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/fdo-internals
_______________________________________________
fdo-internals mailing list
fdo-internals at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/fdo-internals

More information about the fdo-internals mailing list