[fdo-internals] New RFC 30 Posted

[Traian Stanev]

GetGeometry has two overloads. One of them returns a raw pointer:

virtual const FdoByte*      GetGeometry     (FdoString* propertyName, FdoInt32* len);

This is the API I always use in client code. Also, GetString() returns a raw pointer which is also only valid till the following ReadNext():

virtual FdoString*          GetString       (FdoString* propertyName);


However, we are digressing from the topic...


Traian


From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 6:04 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Still  even with SDF fdo clients are calling GetGeometry which returns FdoByteArray * which allocates new memory buffer and data is copied into new buffer.
Problem is not that data is copied once but that it is copied multiple times (not case with FdoByteArray ) but with other data types.

With preallocated shared data buffers, data returned by reader can be reused from same buffers without copying data even if reader is closed and destroyed.

Haris

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 11:15 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted


Yes, such corruption is a possible problem. Hoewever, if your scheme is based on cached (pre-created) PropertyValue objects, then it has the same problem – someone can hold on to the PropertyValue object while it changes from under them. If you want to fix that, you’d have to do some really ugly refcounting hacks.

Besides, there is precedent for allowing corruptible memory – one of the GetGeometry calls in FdoFeatureReader is only guaranteed to hold good data until the next ReadNext call, and callers must be aware of that. This allows providers like SDF and SQLite for example to return pointers directly to database memory, thus avoiding doing even a single copy of the data. This is also how the GetString() calls works.

Traian


From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 5:15 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

If that wrapper would return same data buffer  than we are into same problems as if we would send reader to another command.
Feature reader data buffer would be corrupted on next call and that wrapper in another command would give wrong result.

When I was talking about feature reader returning fdo data value, was primarily to avoid what we do now: switch(data type), we create new fdo data type , copy raw data to new fdo data value and pass that data to another command.
Intention was to reduce that switch and data copy code in every fdo application.

I think to improve  performance of reader, would be that reader will be able to fetch data in preallocated shared data buffers.

Haris

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 10:15 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted


Yes, a wrapper around either FdoFeatureReader or a PropertyValueCollection – same API for both. However, it would not necessarily copy the data from the FeatureReader – in the case where it wraps a FeatureReader it could simply do a passthrough, for example:

virtual FdoString* FdoFRFeature::GetString(FdoString* propName) { return m_reader->GetString(propName); }

Where m_reader is a private member variable pointing to the FeatureReader.

In the case where it wraps a PropertyValueCollection, it would be *roughly*:

virtual FdoString* FdoPVCFeature::GetString(FdoString* propName) { return ((FdoStringValue*)m_propValCol->FindItem(propName))->GetString(); }

The actual implementation is not the point though, as long as the outfacing API allows for implementing it efficiently. I don’t think an API which uses PropertyValue objects has the maximum potential for efficiency because you would be incurring at least two virtual calls to refcounting functions on top of already expensive virtual calls to get the value.

Traian


From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 4:19 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

I guess there is lot of discus on that topic.
You are proposing an wrapper over feature reader which would expose data of reader ?
Than that wrapper would need to read rows (or one row ) from reader and copy data to new data buffer before it can be used in another command?
If it would fetch data to new data buffers than it is same performance problem of copying data. Same performance as to copy data to property values as we do already eventually if we want to do something else with that data.
Data type conversions can be implemented on existing reader too.

What I see as way to improve performance is to be able to set data buffers for select command so returned reader would used those buffers. So this same data without copying can be reused.

Haris

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 9:34 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted


Think of it as a data row – a feature reader lets you iterate through many such data rows via ReadNext(). It is also very similar to OGRFeature for example (http://www.gdal.org/ogr/classOGRFeature.html).

So, for example say you want to transfer a feature from one connection to another – you would do a select from one connection, and that will return a FeatureReader. You can wrap that in the data row object, which has essentially the same API as the feature reader, except that it does not allow ReadNext(). Then you can pass that wrapper to an FdoInsert which will take the data row and insert it into the destination connection.

Internally you could also wrap a PropertyValue collection in such a data row object – no matter whether it is backed by a FeatureReader or a PropertyValue collection, it would offer the same API. Also, you can have the data row convert all its data into a PropertyValueCollection which you can use in case you don’t need performance, thus enabling interop with existing code, which uses such PropertyValue collections.

You can also for example define this object so that it can do automatic type conversion for you – say you call GetInt32 on an Int16 property – it can automatically do the cast for you. Which would provide equivalent functionality to RFC 30 by the way, or it can use whatever code RFC30 adds to FDO. ☺

Traian




From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 3:36 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

I am affraid I don’t understand now. That equivalent of reader would be returned by command (than it is new reader) or it would be created from returned current Reader ?

Haris

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 9:01 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted


Yes that’s a problem and that’s why I said “FdoFeatureReader (or equivalent)”. What I meant by “equivalent” was something that has the almost the same API as FdoIFeatureReader, without the ReadNext/Close functions. Essentially it would represent a single row of data, with an API that allows fast access in a way that is the same as reading features coming from a query.

Traian


From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 3:21 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Yes I understood that. I am afraid that it can be dangerous to use readers as input to another command.
One of thing would be that reader would not be closed before another command would be executed.
Not necessarily but I am worried that it could create problems with current readers/commands.

If we look further as you suggested to improve performance of readers, than it would be better to have them to returned shared pointers to data or to be able to set data buffers to be used by readers.
But even before that I would prefer to have index access to columns of readers as we speak of performance, we discussed it few times :)

Haris

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 8:41 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted


Yes, but creating PropertyValues would be a performance overhead. That’s why I’m suggesting the alternative approach of allowing commands to pull the feature’s data directly from a FeatureReader (or equivalent).

Traian

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 2:25 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted


Right now from Reader we can’t get PropertyValues only raw data.
To get Property value and property value collection which then could be passed to another command would be of great help and remove some of unnecessary  data conversions.

Haris

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian Stanev
Sent: Monday, November 24, 2008 7:39 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted



An alternative to this suggestion is to make the destination command accept an FdoIFeatureReader (or something equivalent that provides access to property values in a procedural way) as argument, rather than a collection of PropertyValues. I was thinking of this when I was implementing the SDF->SQLite converter, which requires two translations from PropertyValue to raw data – once when initializing the inputs of the insert command and a second time inside the destination provider when matching the input property values to the destination schema.

Traian



From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris Kurtagic
Sent: Monday, November 24, 2008 8:27 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Hi Brent,
not directly related to this RFC but still it is important issue regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to get value as basic type from reader and then convert to FdoDataValue etc...
I would like to be able to use data values and properties with values from readers directly and make those as input value for another FDO command.
Right now I need to check data type returned from property get that data as native convert back to fdo data value and put as input to FDO command.

Haris

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Brent Robinson
Sent: Friday, November 21, 2008 7:32 PM
To: FDO Internals Mail List
Subject: [fdo-internals] New RFC 30 Posted

Hi,

RFC 30 (http://trac.osgeo.org/fdo/wiki/FDORfc30) is now ready for review. The main purpose is add FDO API functions to encapsulate conversions between different FDO data types. Please review and let me know if you have any questions or comments.

If approved, this one will likely be implemented 2 versions after FDO 3.3.

Thanks,
Brent.