[fdo-internals] New RFC 30 Posted

Mon Nov 24 18:03:51 EST 2008

Still  even with SDF fdo clients are calling GetGeometry which returns
FdoByteArray * which allocates new memory buffer and data is copied into
new buffer.

Problem is not that data is copied once but that it is copied multiple
times (not case with FdoByteArray ) but with other data types.

With preallocated shared data buffers, data returned by reader can be
reused from same buffers without copying data even if reader is closed
and destroyed.

Haris

From: fdo-internals-bounces at lists.osgeo.org
[mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian
Stanev
Sent: Monday, November 24, 2008 11:15 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Yes, such corruption is a possible problem. Hoewever, if your scheme is
based on cached (pre-created) PropertyValue objects, then it has the
same problem - someone can hold on to the PropertyValue object while it
changes from under them. If you want to fix that, you'd have to do some
really ugly refcounting hacks. 

Besides, there is precedent for allowing corruptible memory - one of the
GetGeometry calls in FdoFeatureReader is only guaranteed to hold good
data until the next ReadNext call, and callers must be aware of that.
This allows providers like SDF and SQLite for example to return pointers
directly to database memory, thus avoiding doing even a single copy of
the data. This is also how the GetString() calls works.

Traian

From: fdo-internals-bounces at lists.osgeo.org
[mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris
Kurtagic
Sent: Monday, November 24, 2008 5:15 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

If that wrapper would return same data buffer  than we are into same
problems as if we would send reader to another command.

Feature reader data buffer would be corrupted on next call and that
wrapper in another command would give wrong result.

When I was talking about feature reader returning fdo data value, was
primarily to avoid what we do now: switch(data type), we create new fdo
data type , copy raw data to new fdo data value and pass that data to
another command.

Intention was to reduce that switch and data copy code in every fdo
application.

I think to improve  performance of reader, would be that reader will be
able to fetch data in preallocated shared data buffers.

Haris

From: fdo-internals-bounces at lists.osgeo.org
[mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian
Stanev
Sent: Monday, November 24, 2008 10:15 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Yes, a wrapper around either FdoFeatureReader or a
PropertyValueCollection - same API for both. However, it would not
necessarily copy the data from the FeatureReader - in the case where it
wraps a FeatureReader it could simply do a passthrough, for example:

virtual FdoString* FdoFRFeature::GetString(FdoString* propName) { return
m_reader->GetString(propName); }

Where m_reader is a private member variable pointing to the
FeatureReader. 

In the case where it wraps a PropertyValueCollection, it would be
*roughly*:

virtual FdoString* FdoPVCFeature::GetString(FdoString* propName) {
return ((FdoStringValue*)m_propValCol->FindItem(propName))->GetString();
}

The actual implementation is not the point though, as long as the
outfacing API allows for implementing it efficiently. I don't think an
API which uses PropertyValue objects has the maximum potential for
efficiency because you would be incurring at least two virtual calls to
refcounting functions on top of already expensive virtual calls to get
the value.

Traian

From: fdo-internals-bounces at lists.osgeo.org
[mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris
Kurtagic
Sent: Monday, November 24, 2008 4:19 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

I guess there is lot of discus on that topic.

You are proposing an wrapper over feature reader which would expose data
of reader ?

Than that wrapper would need to read rows (or one row ) from reader and
copy data to new data buffer before it can be used in another command?

If it would fetch data to new data buffers than it is same performance
problem of copying data. Same performance as to copy data to property
values as we do already eventually if we want to do something else with
that data.

Data type conversions can be implemented on existing reader too.

What I see as way to improve performance is to be able to set data
buffers for select command so returned reader would used those buffers.
So this same data without copying can be reused.

Haris

From: fdo-internals-bounces at lists.osgeo.org
[mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian
Stanev
Sent: Monday, November 24, 2008 9:34 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Think of it as a data row - a feature reader lets you iterate through
many such data rows via ReadNext(). It is also very similar to
OGRFeature for example (http://www.gdal.org/ogr/classOGRFeature.html).

So, for example say you want to transfer a feature from one connection
to another - you would do a select from one connection, and that will
return a FeatureReader. You can wrap that in the data row object, which
has essentially the same API as the feature reader, except that it does
not allow ReadNext(). Then you can pass that wrapper to an FdoInsert
which will take the data row and insert it into the destination
connection.

Internally you could also wrap a PropertyValue collection in such a data
row object - no matter whether it is backed by a FeatureReader or a
PropertyValue collection, it would offer the same API. Also, you can
have the data row convert all its data into a PropertyValueCollection
which you can use in case you don't need performance, thus enabling
interop with existing code, which uses such PropertyValue collections.

You can also for example define this object so that it can do automatic
type conversion for you - say you call GetInt32 on an Int16 property -
it can automatically do the cast for you. Which would provide equivalent
functionality to RFC 30 by the way, or it can use whatever code RFC30
adds to FDO. J

Traian

From: fdo-internals-bounces at lists.osgeo.org
[mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris
Kurtagic
Sent: Monday, November 24, 2008 3:36 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

I am affraid I don't understand now. That equivalent of reader would be
returned by command (than it is new reader) or it would be created from
returned current Reader ?

Haris

From: fdo-internals-bounces at lists.osgeo.org
[mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian
Stanev
Sent: Monday, November 24, 2008 9:01 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Yes that's a problem and that's why I said "FdoFeatureReader (or
equivalent)". What I meant by "equivalent" was something that has the
almost the same API as FdoIFeatureReader, without the ReadNext/Close
functions. Essentially it would represent a single row of data, with an
API that allows fast access in a way that is the same as reading
features coming from a query.

Traian

From: fdo-internals-bounces at lists.osgeo.org
[mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris
Kurtagic
Sent: Monday, November 24, 2008 3:21 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Yes I understood that. I am afraid that it can be dangerous to use
readers as input to another command.

One of thing would be that reader would not be closed before another
command would be executed.

Not necessarily but I am worried that it could create problems with
current readers/commands.

If we look further as you suggested to improve performance of readers,
than it would be better to have them to returned shared pointers to data
or to be able to set data buffers to be used by readers.

But even before that I would prefer to have index access to columns of
readers as we speak of performance, we discussed it few times :)

Haris

From: fdo-internals-bounces at lists.osgeo.org
[mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian
Stanev
Sent: Monday, November 24, 2008 8:41 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Yes, but creating PropertyValues would be a performance overhead. That's
why I'm suggesting the alternative approach of allowing commands to pull
the feature's data directly from a FeatureReader (or equivalent).

Traian

From: fdo-internals-bounces at lists.osgeo.org
[mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris
Kurtagic
Sent: Monday, November 24, 2008 2:25 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Right now from Reader we can't get PropertyValues only raw data.

To get Property value and property value collection which then could be
passed to another command would be of great help and remove some of
unnecessary  data conversions.

Haris

From: fdo-internals-bounces at lists.osgeo.org
[mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian
Stanev
Sent: Monday, November 24, 2008 7:39 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

An alternative to this suggestion is to make the destination command
accept an FdoIFeatureReader (or something equivalent that provides
access to property values in a procedural way) as argument, rather than
a collection of PropertyValues. I was thinking of this when I was
implementing the SDF->SQLite converter, which requires two translations
from PropertyValue to raw data - once when initializing the inputs of
the insert command and a second time inside the destination provider
when matching the input property values to the destination schema.

Traian

From: fdo-internals-bounces at lists.osgeo.org
[mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris
Kurtagic
Sent: Monday, November 24, 2008 8:27 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] New RFC 30 Posted

Hi Brent,

not directly related to this RFC but still it is important issue
regarding FDO data values.

I find a problem working with FDO Readers and commands is that I need to
get value as basic type from reader and then convert to FdoDataValue
etc...

I would like to be able to use data values and properties with values
from readers directly and make those as input value for another FDO
command.

Right now I need to check data type returned from property get that data
as native convert back to fdo data value and put as input to FDO
command.

Haris

From: fdo-internals-bounces at lists.osgeo.org
[mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Brent
Robinson
Sent: Friday, November 21, 2008 7:32 PM
To: FDO Internals Mail List
Subject: [fdo-internals] New RFC 30 Posted

Hi,

RFC 30 (http://trac.osgeo.org/fdo/wiki/FDORfc30) is now ready for
review. The main purpose is add FDO API functions to encapsulate
conversions between different FDO data types. Please review and let me
know if you have any questions or comments.

If approved, this one will likely be implemented 2 versions after FDO
3.3. 

Thanks,

Brent.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/fdo-internals/attachments/20081125/dca2d7b1/attachment-0001.html