[Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Nyall Dawson]

On 7 Oct 2016 5:59 AM, "Even Rouault" <even.rouault at spatialys.com> wrote:
>
> Le samedi 17 septembre 2016 00:00:55, Jeff McKenna a écrit :
> > On 2016-09-16 3:53 PM, Even Rouault wrote:
> > > One of my hypothesis is some bad interaction with the "connection
pool".
> > > Could you (or anyone) open the attribute table before doing the
edits,and
> > > do the edits and commit them no more than 60 seconds afterwards ? I'm
> > > not sure but this might perhaps also require the shapefile to be big
> > > enough (several thousands of features).
> >
> > Here was my next test:
> >
> > - download OSM land polygons for the world, unsplit:
> > http://data.openstreetmapdata.com/land-polygons-complete-4326.zip
> > 424MB   568,635 features  <--- big test!
> >
> > - add shapefile to QGIS 2.16.2 view
> >
> > - open attribute table
> >
> > - start editing
> >
> > - after a while waiting for nodes to appear as QGIS works, message is
> > displayed of "qgis-bin.exe has stopped working"
>
> I've tried your file 2.16.3 in a Win7 VM with 4GB and trying to reproduce
your
> actions but didn't get a crash. I managed painfully to move a node of the
> coastline of western part of France. Note that I used a 64 bit build, and
I
> noticed that the memory consumption goes up to 3 GB when you enable the
Node
> tool (goes down to 300 MB once you finish all editing actions), so if you
use a
> 32 bit build, a crash is not surprising.
> All the actions are super slow, particularly making a node active and
moving
> it. There are clearly issues, with algorithms, data structures and memory
use,
> when dealing with huge polygons.
> I also noticed that there's apparently always a core busy until you finish
> editing.

The issue here is that the node tool adds a new QGraphicsItem to the canvas
for every node - so in a complex polygon you can end up with 100k's of
items in the scene. Then these items are repainted for every mouse move
across the canvas.... Ouch!!

One good solution here would be to enable Qt's inbuilt caching for the map
canvas QGraphicsItems. I experimented with this with good results, but also
a bunch of bugs I didn't have a chance to fix at the time... (Could not
find anyway to force qt to invalidate the cache for every item in the scene
when the map scale/extent changed)

Nyall

>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20161007/77a659e9/attachment.html>