[gdal-dev] Large shapefile issues

[Even Rouault]

Le mardi 26 novembre 2013 00:46:18, Stephen Woodbridge a écrit :
> Unless something has changed, I have never been able to work with dbf
> file over 2GB using shapelib.

With shapelib or OGR ? With shapelib, you need to define SAOffset to be a 64bit 
integer, which OGR does.
The shapefile driver regression tests include a test to read and update a 
record over 2GB.

> 
> -Steve W
> 
> On 11/25/2013 5:52 PM, Even Rouault wrote:
> > Le lundi 25 novembre 2013 11:42:23, CARMAN, Darren a écrit :
> >> Hi List
> >> 
> >> 
> >> 
> >> I notice on the OGR formats page for ESRI Shapefile the following is
> >> mentioned:
> >> 
> >> ================
> >> 
> >> Size Issues
> >> 
> >> Geometry: The Shapefile format explicitly uses 32bit offsets and so
> >> cannot go over 8GB (it actually uses 32bit offsets to 16bit words).
> >> Hence, it is is not recommended to use a file size over 4GB.
> >> 
> >> Attributes: The dbf format does not have any offsets in it, so it can be
> >> arbitrarily large.
> >> 
> >> ================
> >> 
> >> 
> >> 
> >> Yet on the ESRI website:
> >> 
> >> ================
> >> 
> >> Geometry limitations
> >> 
> >> There is a 2 GB size limit for any shapefile component file, which
> >> translates to a maximum of roughly 70 million point features. The actual
> >> number of line or polygon features you can store in a shapefile depends
> >> on the number of vertices in each line or polygon (a vertex is
> >> equivalent to a point).
> >> 
> >> ================
> >> 
> >> 
> >> 
> >> I assume the OGR web page is wrong, or has a different meaning outside
> >> of ESRI S/W use.
> > 
> > Darren,
> > 
> > Yes, as underlined by Chaitanya the actual limit depends on the software
> > implementation. Actually the limit in OGR was 4 GB for the .SHP, and
> > AFAICS unlimited for DBF.
> > 
> > I've added in http://trac.osgeo.org/gdal/changeset/26657 a 2GB_LIMIT=YES
> > layer creation option (and SHAPE_2GB_LIMIT configuration option) that
> > will enforce the 2GB limit. And in
> > http://trac.osgeo.org/gdal/changeset/26658 a change so that when the
> > layer is reached the file is properly closed with the valid information.
> > 
> > Spliting files over several files would properly need to be done outside
> > of this, in a script, by restarting from the source layer at the index
> > next to the one that was last written in the shapefile.
> > 
> > Best regards,
> > 
> > Even
> 
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
Geospatial professional services
http://even.rouault.free.fr/services.html