[gdal-dev] Adding Sync to CouchDB/Cloudant Driver

Norman Barker norman at cloudant.com
Wed Mar 4 08:24:42 PST 2015


On Wed, Mar 4, 2015 at 8:02 AM, Even Rouault <even.rouault at spatialys.com>
wrote:

> Norman,
>
> >
> > the aim is to sync data between masters as per -
> > http://dataprotocols.org/couchdb-replication/ An example would be using
> > GDAL in QGIS and going offline to edit data and then coming back online,
> > the user would need to synchronize the dataset. As replication is a core
> > part of CouchDB I would like to add it to the CouchDB OGR Driver.
>
> If I understand well, you would consider the local copy as a "master",
> even if
> it is implemented as a non-couchdb instance ?
>
> How would a user would decide that it enters offline mode : with a special
> connection string provided to the CouchDB driver ?
>

It wouldn't matter, it would only sync with an upstream server if there is
an available connection. However it could be useful
to override this (3G vs WiFi for example) so I will support a connection
string.

>
> >
> > Synchronizing at the database opening is a great idea, that makes sense.
>
> I've the vague feeling that offline editing and resync with server could be
> usefull for other drivers, but I'm not sure if it something that can be
> easily
> abstracted off the driver (i.e. if they aren't too many driver specific
> constraints to take into account during offline editing)
>

I think it would be useful for other drivers e.g. GeoPackage, however I
will put it in CouchDB with minimal API changes (if any) and
we can abstract it out later.

>
> >
> > LevelDB would be a new GDAL dependency for the CouchDB driver, the
> GeoJSON
> > will be persisted in leveldb so that subsequent connections to a couchdb
> > database would only retrieve any changes to an upstream CouchDB server.
> > LevelDB is under the BSD license. LevelDB is used in PouchDB so it is
> > relatively easy to follow that code. I would aim to abstract leveldb
> behind
> > a OGRCouchDBCache class so that this storage engine could be changed.
>
> It looks like offline editing would be almost like a driver, no ? You need
> to
> create/edit/delete features. Create/delete layers?
>
> As far as LevelDB, my remark would be that it will just make packagers'
> life a
> bit harder. Would that be a requirement for the CouchDB driver, or would
> that
> condition the offline editing feature only ?
> Are there killer features that you need that couldn't be provided with
> SQLite3, which is used for long by GDAL and a dependency satisfied into all
> builds I'm aware of ?
>

I will try with sqlite3, leveldb is a little easier, but I agree with your
build concerns.

>
> Even
>
> >
> > Norman
> >
> >
> >
> > On Wed, Mar 4, 2015 at 1:43 AM, Even Rouault <even.rouault at spatialys.com
> >
> >
> > wrote:
> > > Norman,
> > >
> > > I'm not familiar with what this "sync" is about, so please bear with
> the
> > > naive
> > > questions.
> > > Is it to synchronize all masters between them ?
> > > Why GetFeature() and/or not at the first GetNextFeature(), or at
> database
> > > opening ?
> > > What is the persistence you mention, and what is the connexion with the
> > > sync
> > > you mention ? Is it done at client (OGR) side ? Would leveldb be a new
> > > GDAL dependency ?
> > >
> > > Even
> > >
> > > > Hi,
> > > >
> > > > I am looking to add sync to the GDAL CouchDB driver. I would rather
> not
> > >
> > > add
> > >
> > > > additional API functions to do this but to wrap it up in the
> > > > usual GetFeature call. The first call to GetFeature would initiate a
> > >
> > > sync,
> > >
> > > > but thoughts on this are appreciated.
> > > >
> > > > Sync is a useful function that is a key differentiator with CouchDB
> > > > which is a multi-master database.
> > > >
> > > > There are a couple of ways of doing the sync, I could force a whole
> > > > database sync (the CouchDB model) or we could do a filtered sync
> based
> > > > on the bbox of the current local dataset. I would do the former in
> the
> > >
> > > initial
> > >
> > > > implementation.
> > > >
> > > > As for the persistence, I would like to use leveldb as a local store
> > >
> > > though
> > >
> > > > I could use sqlite. The two schemas are as follows;
> > >
> > >
> https://github.com/pouchdb/pouchdb/blob/master/lib/adapters/idb/idb.js#L5
> > > 6
> > >
> > >
> https://github.com/couchbaselabs/TouchDB-iOS/wiki/Object-Design-And-Schem
> > > a
> > >
> > > > I will be create a PR on this shortly, but I am interested in any
> early
> > > > feedback.
> > > >
> > > > Norman
> > >
> > > --
> > > Spatialys - Geospatial professional services
> > > http://www.spatialys.com
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20150304/7db3a3e2/attachment.html>


More information about the gdal-dev mailing list