[gdal-dev] Adding Sync to CouchDB/Cloudant Driver

[Even Rouault]

Norman,

> 
> the aim is to sync data between masters as per -
> http://dataprotocols.org/couchdb-replication/ An example would be using
> GDAL in QGIS and going offline to edit data and then coming back online,
> the user would need to synchronize the dataset. As replication is a core
> part of CouchDB I would like to add it to the CouchDB OGR Driver.

If I understand well, you would consider the local copy as a "master", even if 
it is implemented as a non-couchdb instance ?

How would a user would decide that it enters offline mode : with a special 
connection string provided to the CouchDB driver ?

> 
> Synchronizing at the database opening is a great idea, that makes sense.

I've the vague feeling that offline editing and resync with server could be 
usefull for other drivers, but I'm not sure if it something that can be easily 
abstracted off the driver (i.e. if they aren't too many driver specific 
constraints to take into account during offline editing)

> 
> LevelDB would be a new GDAL dependency for the CouchDB driver, the GeoJSON
> will be persisted in leveldb so that subsequent connections to a couchdb
> database would only retrieve any changes to an upstream CouchDB server.
> LevelDB is under the BSD license. LevelDB is used in PouchDB so it is
> relatively easy to follow that code. I would aim to abstract leveldb behind
> a OGRCouchDBCache class so that this storage engine could be changed.

It looks like offline editing would be almost like a driver, no ? You need to 
create/edit/delete features. Create/delete layers?

As far as LevelDB, my remark would be that it will just make packagers' life a 
bit harder. Would that be a requirement for the CouchDB driver, or would that 
condition the offline editing feature only ?
Are there killer features that you need that couldn't be provided with 
SQLite3, which is used for long by GDAL and a dependency satisfied into all 
builds I'm aware of ?

Even

> 
> Norman
> 
> 
> 
> On Wed, Mar 4, 2015 at 1:43 AM, Even Rouault <even.rouault at spatialys.com>
> 
> wrote:
> > Norman,
> > 
> > I'm not familiar with what this "sync" is about, so please bear with the
> > naive
> > questions.
> > Is it to synchronize all masters between them ?
> > Why GetFeature() and/or not at the first GetNextFeature(), or at database
> > opening ?
> > What is the persistence you mention, and what is the connexion with the
> > sync
> > you mention ? Is it done at client (OGR) side ? Would leveldb be a new
> > GDAL dependency ?
> > 
> > Even
> > 
> > > Hi,
> > > 
> > > I am looking to add sync to the GDAL CouchDB driver. I would rather not
> > 
> > add
> > 
> > > additional API functions to do this but to wrap it up in the
> > > usual GetFeature call. The first call to GetFeature would initiate a
> > 
> > sync,
> > 
> > > but thoughts on this are appreciated.
> > > 
> > > Sync is a useful function that is a key differentiator with CouchDB
> > > which is a multi-master database.
> > > 
> > > There are a couple of ways of doing the sync, I could force a whole
> > > database sync (the CouchDB model) or we could do a filtered sync based
> > > on the bbox of the current local dataset. I would do the former in the
> > 
> > initial
> > 
> > > implementation.
> > > 
> > > As for the persistence, I would like to use leveldb as a local store
> > 
> > though
> > 
> > > I could use sqlite. The two schemas are as follows;
> > 
> > https://github.com/pouchdb/pouchdb/blob/master/lib/adapters/idb/idb.js#L5
> > 6
> > 
> > https://github.com/couchbaselabs/TouchDB-iOS/wiki/Object-Design-And-Schem
> > a
> > 
> > > I will be create a PR on this shortly, but I am interested in any early
> > > feedback.
> > > 
> > > Norman
> > 
> > --
> > Spatialys - Geospatial professional services
> > http://www.spatialys.com

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com