[Qgis-developer] [Qgis-user] Caching remote postgis layers?

Tue Aug 6 02:56:13 PDT 2013

Hi Nathan,

I did not mean that SQLAlchemy can do this out of the box ;)

Cheers,
Angelos

On Tue, Aug 6, 2013 at 12:52 PM, Nathan Woodrow <madmanwoo at gmail.com> wrote:

> SQLAlchemy is a ORM it's not going to help with any kind of caching.
>
> There are plans for creating a in memory feature cache of sorts so that we
> don't have to fetch each feature everytime however there is a bit of
> work involved in making sure we update the view when the data
> source changes.
>
> - Nathan
>
>
> On Tue, Aug 6, 2013 at 7:33 PM, Angelos Tzotsos <gcpp.kalxas at gmail.com>wrote:
>
>> Hi all,
>>
>> I am *guessing* that Manifold is using the .NET Dataset type to cache
>> data, and that some kind of db manager takes care of the sync.
>>
>> SQLAlchemy could be used in a python plugin for the same purpose.
>>
>> Best,
>> Angelos
>>
>>
>>
>> On Tue, Aug 6, 2013 at 9:46 AM, Marco Hugentobler <
>> marco.hugentobler at sourcepole.ch> wrote:
>>
>>>  Hi Willem
>>>
>>> Another thing to explore is the offline editing plugin. It caches
>>> postgis layers to a local spatialite database and can even be used to
>>> synchronise edits between local spatialite db and remote postgis db.
>>>
>>>
>>> >Sourcepole have developed (for NIWA, New Zealand) a plugin that
>>> supports local caching of WMS & WFS data sources. This could probably be
>>> adapted to support other input >formats. Marco Hugentobler did the coding
>>> for this - he can perhaps comment?
>>>
>>> The plugin saves WMS as local rasters and WFS as local vectors. This is
>>> similar to save as local vector / raster file by hand. However the plugin
>>> is more convenient because it manages the local storage and the reference
>>> to the remote uri.  It has some other nice features like synchronizing /
>>> offlining in a graphical way. Btw., we hope to include this plugin (a C++
>>> one) into the main repository after 2.0 feature freeze is over.
>>>
>>> Regards,
>>> Marco
>>>
>>> On 06.08.2013 03:20, Brent Wood wrote:
>>>
>>>  Hi Willem,
>>>
>>>  Why can you not save such a layer or a selection from such a layer, as
>>> a local shapefile (or other supported format). is this not effectively a
>>> "local cache"? Delete/overwrite when you desire.
>>>
>>>  Where is the performance bottleneck? Postgis, network, QGIS?  Just
>>> creating a local cache will not necessarily speed things up, if your
>>> database is on a fast server with a fast network, it may even get slower.
>>>
>>>  Sourcepole have developed (for NIWA, New Zealand) a plugin that
>>> supports local caching of WMS & WFS data sources. This could probably be
>>> adapted to support other input formats. Marco Hugentobler did the coding
>>> for this - he can perhaps comment?
>>>
>>>  As Nathan says, caching a local snapshot of a dynamic, multi-user
>>> database will result in something that gets out of sybc with the source
>>> pretty quickly. Caching a local copy of a largely static database is
>>> relatively safe.
>>>
>>>  Cheers
>>>
>>>  Brent
>>>   ------------------------------
>>>  *From:* Nathan Woodrow <madmanwoo at gmail.com> <madmanwoo at gmail.com>
>>> *To:* Willem Buitendyk <willem at pcfish.ca> <willem at pcfish.ca>
>>> *Cc:* qgis-user <qgis-user at lists.osgeo.org> <qgis-user at lists.osgeo.org>;
>>> "qgis-developer at lists.osgeo.org" <qgis-developer at lists.osgeo.org>
>>> <qgis-developer at lists.osgeo.org> <qgis-developer at lists.osgeo.org>
>>> *Sent:* Tuesday, August 6, 2013 11:07 AM
>>> *Subject:* Re: [Qgis-user] Caching remote postgis layers?
>>>
>>>  Hey Willem,
>>>
>>>  I'm not aware of a feature like that.  The best place to ask this
>>> question to see if anyone is working, or planning to work, on it is the
>>> developer list.  I have copied it in so it show up there too.
>>>
>>>  Caching could be done by storing the returned results in a memory
>>> spatialite database. The tricky thing is knowing when to get any new
>>> results from the postgis database when things have changed on the database.
>>>  How does Manifold handling viewing an area, updating something using SQL
>>> on the database (not though Manifold) and then looking at that area again?
>>>
>>>  - Nathan
>>>
>>>
>>> On Tue, Aug 6, 2013 at 2:44 AM, Willem Buitendyk <willem at pcfish.ca>wrote:
>>>
>>> I've noticed looking at the roadmap that caching remote postgis data is
>>> not on the list.  I love qgis but a lot of my work involves connecting to a
>>> remote postgis database.  The speed of navigating a large dataset is, at
>>> the moment, rather unbearable.  I also use Manifold GIS and the difference
>>> in speed is shocking, presumably because Manifold is caching the dataset.
>>>  This seems like it should be a relatively easy thing to implement.  Is
>>> there interest on this board for this feature?  I was thinking at taking a
>>> stab at contributing to the source code but am wondering about methods to
>>> achieve data caching most efficiently.  Look forward to others thoughts?
>>>  Or maybe I missing something that is already in place?
>>>
>>> Willem
>>>
>>> _______________________________________________
>>> Qgis-user mailing list
>>> Qgis-user at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/qgis-user
>>>
>>>
>>>
>>> _______________________________________________
>>> Qgis-user mailing list
>>> Qgis-user at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/qgis-user
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Qgis-developer mailing listQgis-developer at lists.osgeo.orghttp://lists.osgeo.org/mailman/listinfo/qgis-developer
>>>
>>>
>>>
>>> --
>>> Dr. Marco Hugentobler
>>> Sourcepole -  Linux & Open Source Solutions
>>> Weberstrasse 5, CH-8004 Zürich, Switzerlandmarco.hugentobler at sourcepole.ch http://www.sourcepole.ch
>>> Technical Advisor QGIS Project Steering Committee
>>>
>>>
>>> _______________________________________________
>>> Qgis-developer mailing list
>>> Qgis-developer at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>>>
>>>
>>
>>
>> --
>> Angelos Tzotsos
>> Remote Sensing Laboratory
>> National Technical University of Athens
>> http://users.ntua.gr/tzotsos
>>
>> _______________________________________________
>> Qgis-developer mailing list
>> Qgis-developer at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>>
>

-- 
Angelos Tzotsos
Remote Sensing Laboratory
National Technical University of Athens
http://users.ntua.gr/tzotsos
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20130806/72d3da4e/attachment-0001.html>