[fdo-internals] Vote: FDO RFC 16 - FDO Provider for SQLite

Tue Apr 1 13:26:51 EDT 2008

I turn that question around to: What the new provider does to make that query fast that the SDF provider can't be made to do?

Badreddine

-----Original Message-----
From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian Stanev
Sent: Tuesday, April 01, 2008 1:21 PM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] Vote: FDO RFC 16 - FDO Provider for SQLite

What about performance -- would the new SDF perform faster than the current SDF for things like reading data from BBOX query?

Traian

> -----Original Message-----
> From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-
> bounces at lists.osgeo.org] On Behalf Of Badreddine Karoui
> Sent: Tuesday, April 01, 2008 1:03 PM
> To: FDO Internals Mail List
> Subject: RE: [fdo-internals] Vote: FDO RFC 16 - FDO Provider for SQLite
>
> The upgraded SDF provider can easily support the FdoSqlCommand and SQL
> passthrough. That would be a trivial command to implement.
> I think the main benefit of using SQLite compliant file would be to
> have other non-Fdo tools access to the file. At least that's the
> request I've seen many times. Something like having an ODBC or ADO.Net
> access to the SDF file.
>
> Badreddine
>
> -----Original Message-----
> From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-
> bounces at lists.osgeo.org] On Behalf Of Jason Birch
> Sent: Tuesday, April 01, 2008 12:44 PM
> To: FDO Internals Mail List
> Subject: RE: [fdo-internals] Vote: FDO RFC 16 - FDO Provider for SQLite
>
> Looks like my previous email got eaten by a misbehaving Exchange
> server... anyway:
>
> For me the most important part of this RFC is the creation of a format
> that is single file, open, and can be easily shared between open source
> applications.  This includes things like being able to take advantage
> of
> standard SQL queries, etc.
>
> The current SDF format is not documented, and it is my (possibly
> mistaken) understanding that it uses an FDO structure for metadata
> rather than the OGC SFSQL metadata tables.  Maybe I'm confusing
> provider
> with format here, but if the SDF provider is reading non-SDF files... ?
>
> If the SDF provider was re-purposed for this non-SDF format, would it
> still be able to handle native SQLite queries?  Would it allow
> SelectOrdering and SelectGrouping? (OK, that last one was gratuitous -
> but it's a pain point for me)
>
> Regardless, the important point for me here is that Traian is willing
> to
> write this open provider (already has) and it fills an important need.
> I don't want to see this opportunity go away (Traian loses interest,
> nobody else has resources to modify the SDF provider).
>
> Jason
>
> -----Original Message-----
> From: Badreddine Karoui
> Subject: RE: [fdo-internals] Vote: FDO RFC 16 - FDO Provider for SQLite
>
> Up until now, the SDF file format is the defacto FDO file format. I
> have
> 2 questions: 1) does Fdo need an another file format? And 2) if Fdo
> needs an other file format, then what this format brings new that the
> upgraded SDF will not?
>
> -----Original Message-----
> From: fdo-internals-bounces at lists.osgeo.org
> [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Traian
> Stanev
> Sent: Tuesday, April 01, 2008 12:29 PM
> To: FDO Internals Mail List
> Subject: RE: [fdo-internals] Vote: FDO RFC 16 - FDO Provider for SQLite
>
>
> SDF would end up implementing its own row encoder/decoder for SQLite.
> Why not use the one that SQLite itself has?
> SDF would end up using its own query engine. Why not use the one that
> SQLite itself has (SQL)?
> SDF has a persisted R-Tree.
>
> Those are design decisions that are different from I have in mind.
>
> Besides, I don't see what the argument is all about. We can have both
> the SDF provider and my provider support the same format.
>
>
>
> Traian
>
>
>
>
> > -----Original Message-----
> > From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-
> > bounces at lists.osgeo.org] On Behalf Of Badreddine Karoui
> > Sent: Tuesday, April 01, 2008 12:20 PM
> > To: FDO Internals Mail List
> > Subject: RE: [fdo-internals] Vote: FDO RFC 16 - FDO Provider for
> SQLite
> >
> > Only the existing binary encoder/decoder is maintained for backward
> > compatibity. A new binary encoder/decoder will be created to write
> > SQLite compliant data. I wouldn't call that 2 providers in one DLL.
> > The SDF provider is at best 3 years old and can hardly be called
> > "legacy". I'm sure any performance used in the new code can apply to
> > the SDF provider, wouldn't?
> >
> > Badreddine
> >
> > -----Original Message-----
> > From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-
> > bounces at lists.osgeo.org] On Behalf Of Traian Stanev
> > Sent: Tuesday, April 01, 2008 12:09 PM
> > To: FDO Internals Mail List
> > Subject: RE: [fdo-internals] Vote: FDO RFC 16 - FDO Provider for
> SQLite
> >
> >
> > It would be like having two providers in one DLL. There would be two
> > code paths for everything, and I don't think all that much code would
> > be shared between the two, apart from query execution.
> >
> > The SQLite provider (which I have already written), is right now
> > significantly faster than SDF in use cases that MapGuide cares about,
> > even with the virtual machine in the way. For BBOX spatial query on
> > several data sets I tried, it is at least 10 times faster than SDF. I
> > think that if the SDF codebase is used as a starting point, this
> would
> > not have been the case, due to all the legacy code that is in SDF.
> >
> >
> >
> > Traian
> >
> >
> >
> >
> > > -----Original Message-----
> > > From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-
> > > bounces at lists.osgeo.org] On Behalf Of Badreddine Karoui
> > > Sent: Tuesday, April 01, 2008 11:56 AM
> > > To: FDO Internals Mail List
> > > Subject: RE: [fdo-internals] Vote: FDO RFC 16 - FDO Provider for
> > SQLite
> > >
> > > Regardless of the terminology; these two providers persist their
> data
> > > to an SQLite file. The idea would be to upgrade the SDF provider to
> > > write SQLite complaint data blocks that can be consumed by other
> > SQLite
> > > tools/applications. At the same time the SDF provider maintains its
> > > backward compatibility to its current format. We get a seemless
> > > transition to the new format and stability since the bulk of the
> SDF
> > > provider code will be re-used. Also, since the SDF will continue to
> > > bypass the SQLite virtual machine, the performance will not be
> > degraded
> > > at least not by much.
> > >
> > > Badreddine
> > >
> > > -----Original Message-----
> > > From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-
> > > bounces at lists.osgeo.org] On Behalf Of Traian Stanev
> > > Sent: Tuesday, April 01, 2008 11:44 AM
> > > To: FDO Internals Mail List
> > > Subject: RE: [fdo-internals] Vote: FDO RFC 16 - FDO Provider for
> > SQLite
> > >
> > >
> > > The RFC does not propose "enhancements". It proposes a new format.
> > >
> > >
> > > Traian
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-
> internals-
> > > > bounces at lists.osgeo.org] On Behalf Of Greg Boone
> > > > Sent: Tuesday, April 01, 2008 11:32 AM
> > > > To: FDO Internals Mail List
> > > > Subject: RE: [fdo-internals] Vote: FDO RFC 16 - FDO Provider for
> > > SQLite
> > > >
> > > > Maybe this question comes a bit too late, but why can't the
> > > > enhancements proposed in the RFC be made in the existing SDF
> > provider
> > > > as opposed to building a whole new provider?
> > > >
> > > > Greg
> > > >
> > > > -----Original Message-----
> > > > From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-
> internals-
> > > > bounces at lists.osgeo.org] On Behalf Of Haris Kurtagic
> > > > Sent: Monday, March 31, 2008 5:29 PM
> > > > To: FDO Internals Mail List
> > > > Subject: RE: [fdo-internals] Vote: FDO RFC 16 - FDO Provider for
> > > SQLite
> > > >
> > > > +1
> > > > Haris
> > > >
> > > > -----Original Message-----
> > > > From: fdo-internals-bounces at lists.osgeo.org
> > > > [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Jason
> > > Birch
> > > > Sent: Monday, March 31, 2008 7:51 PM
> > > > To: FDO Internals Mail List
> > > > Subject: [fdo-internals] Vote: FDO RFC 16 - FDO Provider for
> SQLite
> > > >
> > > > Given the modifications to the RFC to address questions raised on
> > > this
> > > > list, and a lack of further discussion I hereby move to approve
> FDO
> > > RFC
> > > > 16 - FDO Provider for SQLite.
> > > >
> > > > https://trac.osgeo.org/fdo/wiki/FDORfc16
> > > >
> > > > +1 from me
> > > >
> > > > Jason
> > > >
> > > > -----Original Message-----
> > > > From: Traian Stanev
> > > > Subject: RE: [fdo-internals] FDO RFC 16 - FDO Provider for SQLite
> > > >
> > > > I think it has enough stuff in it to vote on.
> > > >
> > > > _______________________________________________
> > > > fdo-internals mailing list
> > > > fdo-internals at lists.osgeo.org
> > > > http://lists.osgeo.org/mailman/listinfo/fdo-internals
> > > >
> > > > No virus found in this incoming message.
> > > > Checked by AVG.
> > > > Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
> > > > 3/30/2008 12:32 PM
> > > >
> > > > _______________________________________________
> > > > fdo-internals mailing list
> > > > fdo-internals at lists.osgeo.org
> > > > http://lists.osgeo.org/mailman/listinfo/fdo-internals
> > > > _______________________________________________
> > > > fdo-internals mailing list
> > > > fdo-internals at lists.osgeo.org
> > > > http://lists.osgeo.org/mailman/listinfo/fdo-internals
> > > _______________________________________________
> > > fdo-internals mailing list
> > > fdo-internals at lists.osgeo.org
> > > http://lists.osgeo.org/mailman/listinfo/fdo-internals
> > > _______________________________________________
> > > fdo-internals mailing list
> > > fdo-internals at lists.osgeo.org
> > > http://lists.osgeo.org/mailman/listinfo/fdo-internals
> > _______________________________________________
> > fdo-internals mailing list
> > fdo-internals at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/fdo-internals
> > _______________________________________________
> > fdo-internals mailing list
> > fdo-internals at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/fdo-internals
> _______________________________________________
> fdo-internals mailing list
> fdo-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/fdo-internals
> _______________________________________________
> fdo-internals mailing list
> fdo-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/fdo-internals
> _______________________________________________
> fdo-internals mailing list
> fdo-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/fdo-internals
> _______________________________________________
> fdo-internals mailing list
> fdo-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/fdo-internals
_______________________________________________
fdo-internals mailing list
fdo-internals at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/fdo-internals