[Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Jeff McKenna]

On 2016-09-16 3:53 PM, Even Rouault wrote:
>
> One of my hypothesis is some bad interaction with the "connection pool".
> Could you (or anyone) open the attribute table before doing the edits,and do
> the edits and commit them no more than 60 seconds afterwards ? I'm not sure
> but this might perhaps also require the shapefile to be big enough (several
> thousands of features).
>
>


Here was my next test:

- download OSM land polygons for the world, unsplit: 
http://data.openstreetmapdata.com/land-polygons-complete-4326.zip 
424MB   568,635 features  <--- big test!

- add shapefile to QGIS 2.16.2 view

- open attribute table

- start editing

- after a while waiting for nodes to appear as QGIS works, message is 
displayed of "qgis-bin.exe has stopped working"

With QGIS 2.14.0:

- exact same issue when starting editing of that layer

With QGIS 1.8.0:

- able to click start editing and no crash as nodes are displayed for world

- able to use "Split Features" and save edits

- seems to write changes to .dbf .shp .shx (to be honest, changes don't 
look like a valid feature split, hard to tell, but it seems a few more 
feature records were added)

You likely aren't happy that I tried with such large of a file, but, it 
was honestly the first big file that I found.

====
other test for fun:

- checked layer summary through ogrinfo
- indexed the .shp with the shptree utility (generates 22MB .qix index 
file, wow)
- created a small MapServer mapfile with only that layer, for world extents
- generated map image through MapServer, recording layer draw speed (cool!)
- tried to compare that speed, to QGIS 2.16.2 loading that same 
shapefile (doing both a same time)

Fun results!  Any guess how MapServer did?  :)

Honestly I didn't plan to check that, and that is not why I tried to 
help here, but when I saw QGIS having a hard time with this large file, 
on a 16MB RAM new machine, I thought 'hmm i wonder how MapServer would 
handle this huge file?'   For one of the previous WMS Benchmarking 
exercises at an old FOSS4G event I tested with a 1GB shapefile, and we 
had to fix the indexing for MapServer...so it is good to try to push 
limits once in a while.

sorry if i offend anyone with these tests, was just having some fun 
pushing MapServer as well.

Let me know if you need more tests.

-jeff


-- 
Jeff McKenna
MapServer Consulting and Training Services
http://www.gatewaygeomatics.com/