[Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Randal Hale]

I just ran a test

Ubuntu Linux
QGIS 2.16.2
Shapefile with 925,000+ polygons

I opened the attribute table which took a bit.
Started editing.
Split 5 polygons.
Saved.
Stopped editing.
Removed the layer.
Added it back - it didn't corrupt.

I'm wondering if I should have done something more deliberate like 
delete or add more polygons.

Randy




On 09/16/2016 06:00 PM, Jeff McKenna wrote:
> On 2016-09-16 3:53 PM, Even Rouault wrote:
>>
>> One of my hypothesis is some bad interaction with the "connection pool".
>> Could you (or anyone) open the attribute table before doing the 
>> edits,and do
>> the edits and commit them no more than 60 seconds afterwards ? I'm 
>> not sure
>> but this might perhaps also require the shapefile to be big enough 
>> (several
>> thousands of features).
>>
>>
>
>
> Here was my next test:
>
> - download OSM land polygons for the world, unsplit: 
> http://data.openstreetmapdata.com/land-polygons-complete-4326.zip 
> 424MB   568,635 features  <--- big test!
>
> - add shapefile to QGIS 2.16.2 view
>
> - open attribute table
>
> - start editing
>
> - after a while waiting for nodes to appear as QGIS works, message is 
> displayed of "qgis-bin.exe has stopped working"
>
> With QGIS 2.14.0:
>
> - exact same issue when starting editing of that layer
>
> With QGIS 1.8.0:
>
> - able to click start editing and no crash as nodes are displayed for 
> world
>
> - able to use "Split Features" and save edits
>
> - seems to write changes to .dbf .shp .shx (to be honest, changes 
> don't look like a valid feature split, hard to tell, but it seems a 
> few more feature records were added)
>
> You likely aren't happy that I tried with such large of a file, but, 
> it was honestly the first big file that I found.
>
> ====
> other test for fun:
>
> - checked layer summary through ogrinfo
> - indexed the .shp with the shptree utility (generates 22MB .qix index 
> file, wow)
> - created a small MapServer mapfile with only that layer, for world 
> extents
> - generated map image through MapServer, recording layer draw speed 
> (cool!)
> - tried to compare that speed, to QGIS 2.16.2 loading that same 
> shapefile (doing both a same time)
>
> Fun results!  Any guess how MapServer did?  :)
>
> Honestly I didn't plan to check that, and that is not why I tried to 
> help here, but when I saw QGIS having a hard time with this large 
> file, on a 16MB RAM new machine, I thought 'hmm i wonder how MapServer 
> would handle this huge file?'   For one of the previous WMS 
> Benchmarking exercises at an old FOSS4G event I tested with a 1GB 
> shapefile, and we had to fix the indexing for MapServer...so it is 
> good to try to push limits once in a while.
>
> sorry if i offend anyone with these tests, was just having some fun 
> pushing MapServer as well.
>
> Let me know if you need more tests.
>
> -jeff
>
>

-- 
------------------------------------------------
Randal Hale
North River Geographic Systems, Inc
http://www.northrivergeographic.com
423.653.3611 rjhale at northrivergeographic.com
twitter:rjhale     http://about.me/rjhale
http://www.northrivergeographic.com/introduction-to-quantum-gis
https://www.facebook.com/NRGSInc