[Qgis-developer] Warning - shapefile corruption on 2.16/master and how to recover

[Jeff McKenna]

On 2016-09-16 7:00 PM, Jeff McKenna wrote:
> On 2016-09-16 3:53 PM, Even Rouault wrote:
>>
>> One of my hypothesis is some bad interaction with the "connection pool".
>> Could you (or anyone) open the attribute table before doing the
>> edits,and do
>> the edits and commit them no more than 60 seconds afterwards ? I'm not
>> sure
>> but this might perhaps also require the shapefile to be big enough
>> (several
>> thousands of features).
>>
>>
>
>
> Here was my next test:
>
> - download OSM land polygons for the world, unsplit:
> http://data.openstreetmapdata.com/land-polygons-complete-4326.zip
> 424MB   568,635 features  <--- big test!
>
> - add shapefile to QGIS 2.16.2 view
>
> - open attribute table
>
> - start editing
>
> - after a while waiting for nodes to appear as QGIS works, message is
> displayed of "qgis-bin.exe has stopped working"
>
> With QGIS 2.14.0:
>
> - exact same issue when starting editing of that layer
>
> With QGIS 1.8.0:
>
> - able to click start editing and no crash as nodes are displayed for world
>
> - able to use "Split Features" and save edits
>
> - seems to write changes to .dbf .shp .shx (to be honest, changes don't
> look like a valid feature split, hard to tell, but it seems a few more
> feature records were added)
>
> You likely aren't happy that I tried with such large of a file, but, it
> was honestly the first big file that I found.
>
> ====
> other test for fun:
>
> - checked layer summary through ogrinfo
> - indexed the .shp with the shptree utility (generates 22MB .qix index
> file, wow)
> - created a small MapServer mapfile with only that layer, for world extents
> - generated map image through MapServer, recording layer draw speed (cool!)
> - tried to compare that speed, to QGIS 2.16.2 loading that same
> shapefile (doing both a same time)
>
> Fun results!  Any guess how MapServer did?  :)
>
> Honestly I didn't plan to check that, and that is not why I tried to
> help here, but when I saw QGIS having a hard time with this large file,
> on a 16MB RAM new machine, I thought 'hmm i wonder how MapServer would
> handle this huge file?'   For one of the previous WMS Benchmarking
> exercises at an old FOSS4G event I tested with a 1GB shapefile, and we
> had to fix the indexing for MapServer...so it is good to try to push
> limits once in a while.
>
> sorry if i offend anyone with these tests, was just having some fun
> pushing MapServer as well.
>
> Let me know if you need more tests.
>
> -jeff
>
>

it would be fun, yet make no sense to technical people, to compare 
MapServer WFS output or WFS-Transaction requests, versus other tools 
like QGIS editing.  But that is apples vs oranges and could never be 
agreed upon by development teams (or any sane person).

Anyway, I was just having some fun pushing MapServer.

I hope those tests are somewhat useful!

-jeff