[fdo-internals] RE: SDF feature reader

[Badreddine Karoui]

I think the reason is the fact that the SDF provider keeps an update buffer for every table. All readers are using the same cursor to avoid flushing the buffer everytime a query is executed. Writing and reading at the same time is a common use case.

Badreddine

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris Kurtagic
Sent: Wednesday, August 13, 2008 5:16 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] RE: SDF feature reader

if i interpret code correctly there was allready attempt to solve issue with  multiple readers with "Tag" in DbData and call to function RefreshData before every Get.. in feature reader.

I would like to ask is why only one sqlite cursor is used for one sqlite table for all readers. That looks like potential source of problem ?
Create new sqlite cursor for every reader ?

Haris

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Badreddine Karoui
Sent: Wednesday, August 13, 2008 10:47 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] RE: SDF feature reader

There is probably a way to fix this issue and only make a copy if 2 readers are used.

Badreddine

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian Stanev
Sent: Wednesday, August 13, 2008 4:39 PM
To: 'FDO Internals Mail List'
Subject: RE: [fdo-internals] RE: SDF feature reader


Yes, this is an intentional limitation, for the same reason (performance).

Traian


From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris Kurtagic
Sent: Wednesday, August 13, 2008 4:53 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] RE: SDF feature reader

Hm,  but problem is not thread related.

Having 2 feature readers for same SDF connection inside one thread will create invalid data.

Haris

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian Stanev
Sent: Wednesday, August 13, 2008 10:32 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] RE: SDF feature reader


This behavior is really a tradeoff between performance and full thread safety. SDF is near the performance end of the spectrum. I figure, if someone desires full
thread safety, they can use a real database.

Traian


From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Gavin Cramer
Sent: Wednesday, August 13, 2008 4:27 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] RE: SDF feature reader

GetGeometry (the one returning an FdoByte *) has documentation that says the following:

"The array's memory area is only guaranteed to be valid until a call to ReadNext() or Close(), or the disposal of this reader object."

So, the code is behaving according to the specification.  There should be a similar statement for GetString, as well as any other accessor that returns a pointer to non-reference-counted arrays, but I do not see one.  Anyway, an FDO client should expect the above behaviour from any FDO provider, even if it actually does some internal copying.

Regardless, it looks like the behaviour that Haris sees is a defect.  Different readers should not be sharing an internal cursor.

Gavin


From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Badreddine Karoui
Sent: Wednesday, August 13, 2008 4:14 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] RE: SDF feature reader

The free threaded provider would use FdoThreadCapability_MultiThreaded defined in https://svn.osgeo.org/fdo/trunk/Fdo/Unmanaged/Inc/Fdo/Connections/Capabilities/ThreadCapability.h

The SDF provider is returning FdoThreadCapability_PerConnectionThreaded.

I had a quick look and I can see that the SDFSimpleFeatureReader::GetGeometry(FdoString*, FdoInt32*) is returing the pointer from the internal buffer. This may have been done to avoid paying the penalty of copying the geometry. The second oveload, GetGeometry(FdoString*) is returing a copy and should not have the same issue.
We may have to be extra carefull in case we decide to fix this issue to make sure the reader performance is not impacted.

Badreddine

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris Kurtagic
Sent: Wednesday, August 13, 2008 3:12 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] RE: SDF feature reader

it looks that feature readers are sharing same sqlite cursor with one data buffer.   (BTW: Application is sending this readers to be used by another app so it is not able to create new buffers)

Hm, you have lost me in this threading namings :)

Same SDF connection can't be shared across threads, right ? ( I thought that as thread safe )
I would like to understand reasons for that behavior and if there are rooms for improvements.
Sqlite connection can be shared between threads.

Thank you,
Haris

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Badreddine Karoui
Sent: Wednesday, August 13, 2008 8:36 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] RE: SDF feature reader

It was not obvious that the readers were on the same thread. In that case it's a defect. Make sure to copy the geometry before ReadNext is called. It's possible the second reader(ReadNext) is invalidating the geometry.

The SDF provider is thread safe but not free threaded, it's per-connection threaded.

Badreddine

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris Kurtagic
Sent: Wednesday, August 13, 2008 2:39 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] RE: SDF feature reader

I was saying  for  two feature readers inside same thread will result in error.

Also, i would like to understand why SDF provider is not thread safe.

Haris




From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Badreddine Karoui
Sent: Wednesday, August 13, 2008 8:18 PM
To: FDO Internals Mail List
Subject: [fdo-internals] RE: SDF feature reader

The SDF provider is per connection threaded which means only one thread can access a reader at a time. You can have many threads using their own connections/readers at the same time.

Badreddine

From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris Kurtagic
Sent: Wednesday, August 13, 2008 2:29 PM
To: FDO Internals Mail List
Subject: [fdo-internals] SDF feature reader

Hi,
I have issue when when more than one feature reader is created against same SDF connection
It looks like same buffer ( created by sqlite cursor ) is shared betwean feature readers.
This will cause that data returned from feature reader with function GetGeometry will be corrupted when another reader is used.
Is this a bug or I am not looking into it correctly ?

I would like to be able to use multi thread access to sqlite database.
If I am right, SDF provider is not "thread safe" ? if so, what are main reasons for  it ?

Thank you,
Haris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/fdo-internals/attachments/20080813/6a8849cd/attachment-0001.html