[Qgis-developer] [Qgis-user] Caching remote postgis layers?

Tue Aug 6 02:52:29 PDT 2013

SQLAlchemy is a ORM it's not going to help with any kind of caching.

There are plans for creating a in memory feature cache of sorts so that we
don't have to fetch each feature everytime however there is a bit of
work involved in making sure we update the view when the data
source changes.

- Nathan

On Tue, Aug 6, 2013 at 7:33 PM, Angelos Tzotsos <gcpp.kalxas at gmail.com>wrote:

> Hi all,
>
> I am *guessing* that Manifold is using the .NET Dataset type to cache
> data, and that some kind of db manager takes care of the sync.
>
> SQLAlchemy could be used in a python plugin for the same purpose.
>
> Best,
> Angelos
>
>
>
> On Tue, Aug 6, 2013 at 9:46 AM, Marco Hugentobler <
> marco.hugentobler at sourcepole.ch> wrote:
>
>>  Hi Willem
>>
>> Another thing to explore is the offline editing plugin. It caches postgis
>> layers to a local spatialite database and can even be used to synchronise
>> edits between local spatialite db and remote postgis db.
>>
>>
>> >Sourcepole have developed (for NIWA, New Zealand) a plugin that supports
>> local caching of WMS & WFS data sources. This could probably be adapted to
>> support other input >formats. Marco Hugentobler did the coding for this -
>> he can perhaps comment?
>>
>> The plugin saves WMS as local rasters and WFS as local vectors. This is
>> similar to save as local vector / raster file by hand. However the plugin
>> is more convenient because it manages the local storage and the reference
>> to the remote uri.  It has some other nice features like synchronizing /
>> offlining in a graphical way. Btw., we hope to include this plugin (a C++
>> one) into the main repository after 2.0 feature freeze is over.
>>
>> Regards,
>> Marco
>>
>> On 06.08.2013 03:20, Brent Wood wrote:
>>
>>  Hi Willem,
>>
>>  Why can you not save such a layer or a selection from such a layer, as
>> a local shapefile (or other supported format). is this not effectively a
>> "local cache"? Delete/overwrite when you desire.
>>
>>  Where is the performance bottleneck? Postgis, network, QGIS?  Just
>> creating a local cache will not necessarily speed things up, if your
>> database is on a fast server with a fast network, it may even get slower.
>>
>>  Sourcepole have developed (for NIWA, New Zealand) a plugin that
>> supports local caching of WMS & WFS data sources. This could probably be
>> adapted to support other input formats. Marco Hugentobler did the coding
>> for this - he can perhaps comment?
>>
>>  As Nathan says, caching a local snapshot of a dynamic, multi-user
>> database will result in something that gets out of sybc with the source
>> pretty quickly. Caching a local copy of a largely static database is
>> relatively safe.
>>
>>  Cheers
>>
>>  Brent
>>   ------------------------------
>>  *From:* Nathan Woodrow <madmanwoo at gmail.com> <madmanwoo at gmail.com>
>> *To:* Willem Buitendyk <willem at pcfish.ca> <willem at pcfish.ca>
>> *Cc:* qgis-user <qgis-user at lists.osgeo.org> <qgis-user at lists.osgeo.org>;
>> "qgis-developer at lists.osgeo.org" <qgis-developer at lists.osgeo.org>
>> <qgis-developer at lists.osgeo.org> <qgis-developer at lists.osgeo.org>
>> *Sent:* Tuesday, August 6, 2013 11:07 AM
>> *Subject:* Re: [Qgis-user] Caching remote postgis layers?
>>
>>  Hey Willem,
>>
>>  I'm not aware of a feature like that.  The best place to ask this
>> question to see if anyone is working, or planning to work, on it is the
>> developer list.  I have copied it in so it show up there too.
>>
>>  Caching could be done by storing the returned results in a memory
>> spatialite database. The tricky thing is knowing when to get any new
>> results from the postgis database when things have changed on the database.
>>  How does Manifold handling viewing an area, updating something using SQL
>> on the database (not though Manifold) and then looking at that area again?
>>
>>  - Nathan
>>
>>
>> On Tue, Aug 6, 2013 at 2:44 AM, Willem Buitendyk <willem at pcfish.ca>wrote:
>>
>> I've noticed looking at the roadmap that caching remote postgis data is
>> not on the list.  I love qgis but a lot of my work involves connecting to a
>> remote postgis database.  The speed of navigating a large dataset is, at
>> the moment, rather unbearable.  I also use Manifold GIS and the difference
>> in speed is shocking, presumably because Manifold is caching the dataset.
>>  This seems like it should be a relatively easy thing to implement.  Is
>> there interest on this board for this feature?  I was thinking at taking a
>> stab at contributing to the source code but am wondering about methods to
>> achieve data caching most efficiently.  Look forward to others thoughts?
>>  Or maybe I missing something that is already in place?
>>
>> Willem
>>
>> _______________________________________________
>> Qgis-user mailing list
>> Qgis-user at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/qgis-user
>>
>>
>>
>> _______________________________________________
>> Qgis-user mailing list
>> Qgis-user at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/qgis-user
>>
>>
>>
>>
>> _______________________________________________
>> Qgis-developer mailing listQgis-developer at lists.osgeo.orghttp://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>>
>>
>> --
>> Dr. Marco Hugentobler
>> Sourcepole -  Linux & Open Source Solutions
>> Weberstrasse 5, CH-8004 Zürich, Switzerlandmarco.hugentobler at sourcepole.ch http://www.sourcepole.ch
>> Technical Advisor QGIS Project Steering Committee
>>
>>
>> _______________________________________________
>> Qgis-developer mailing list
>> Qgis-developer at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>>
>
>
> --
> Angelos Tzotsos
> Remote Sensing Laboratory
> National Technical University of Athens
> http://users.ntua.gr/tzotsos
>
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20130806/7c1b063b/attachment.html>