[gdal-dev] shapefile enhancements

[Jan Heckman]

Hi,
It appears I have to do some homework on ogr's shapefile functions as it
stands now.

8GB: If interoperability is more of a priority than capacity, that's a
valid consideration. I've not really needed anything > 4GB so far.

Delete:
By delete I mean leaving the information in the file but (shapefile) taking
it out of the index chain (.shx), and .dbf, marking the record with an
asterisk in its firs byte.
As far as arcgis, I did a delete in this way and tried to load it. When I
do not reduce the record count in the dbf header, arcgis will not load it;
when I do reduce the record count in the header, arcgis will load the
shapefile but the attributes will not match the shapes. As a cross-check,
you can open the .dbf in open office or excel: the delete will be
recognized.

My guess is that arcgis maps the shaperecords to the physical records of
the dbf only.

To allow use of the shapefile in arcgis,  I have to compact the .dbf. The
shape will then be handled correctly.

A recipe to try this out:
create a new empty point shapefile, load it in arcgis. Using arccatalog to
create the shapefile, it will have a single ID integer attribute. That's
the starting point.
Create 3 points and give them ID's 1 - 3.
Now to 'delete' the second record using a diskeditor:
Copy the shapefile. Open the .shx. The .shx has a header and records
consisting of offset-length pairs. A pair takes 8 bytes. Change the 2nd
offset to be identical to the last (00000040 -> 0000004E). Diminish the
filelength indicator in the header (offset 0x18) by 4 (0000003E to
0000003A). Copy the file, except the last 8 bytes to the new .shx file.
DBF: open in editor, change the first byte of the second record (at offset
0x48) to an asterisk. The recordcount in the header is at offset 4 (little
endian).

Load in arcgis, will fail.
Change  the dbf-recordcount to 2, load and open the attribute table. The
file will load. The ID's shown will be 1 and 2. The latter should have been
3.
Conclusion: shape deletes are possible, .dbf not. Compacting the .dbf will,
of course, repair things.

Best regards,
Jan



On Mon, Apr 28, 2014 at 11:48 PM, Even Rouault <even.rouault at mines-paris.org
> wrote:

> Le lundi 28 avril 2014 23:39:34, Jan Heckman a écrit :
> > Hi,
> >
> > I've used and maintained an offshoot shapefile library (thanks, Frank),
> and
> > added bits of functionality when needed. There are two that I think might
> > be of wider interest.
> >
> > 1) possibility to have shapefiles up to 8 GB (instead of 4 GB in ogr or 2
> > GB elsewhere);
>
> I'm +/- 0 on this, since it might cause interoperabily issues. Actually
> I've
> added a creation option in GDAL 1.11 to avoid crossing the 2GB barrier.
>
> > 2) implementing deletes.
> > ad 1) this requires little more than replacing fseek() by _fseeki64() on
> a
> > windows box and using the fact that size indicators are about words
> rather
> > than bytes.
> > The use of this feature is limited to library use; afaik gis programs (at
> > least arcgis) will not be able to use such an extended shapefile.
> > Next, portability might be an issue.
> > ad 2) a quick delete without closing and rewriting the .shp and /or the
> > .dbf file can obviously be useful. Shapefile as well as dBase III files
> > offer handles to do this.
>
> Not sure to understand what you call by delete here. Is it auto-compaction
> of
> files when deleting a record ? something like DeleteFeature() + REPACK in
> the
> shapefile driver ?
>
> > Issues:
> > Arcgis wil not recognize .dbf with records marked for deletion (by
> starting
> > a record with an asterix instead of a space character);
> > the (ogr) dbase driver will equally disregard deleted records.
> > Result: at the end of the session using the shapefile library, the
> > shapefile can be handled without a major file-rewrite, but the .dbf needs
> > to have the deleted records removed 'physically'. Still, this can be done
> > relatively quickly when closing the file.
>
> Hum, shapelib already supports deleting dbf record and the ogr shapefile
> driver
> does honour that. So, you are saying that ArcGIS doesn't handle that
> properly
> ?
>
> > Portability might again be an issue.
> >
> > So,does anyone need these features, and is it useful to start a
> discussion
> > how to implement either of these enhancements?
> >
> > regards,
> > Jan
>
> --
> Geospatial professional services
> http://even.rouault.free.fr/services.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20140429/a3672db4/attachment-0001.html>