[gdal-dev] Large shapefile issues

[Even Rouault]

Le lundi 25 novembre 2013 11:42:23, CARMAN, Darren a écrit :
> Hi List
> 
> 
> 
> I notice on the OGR formats page for ESRI Shapefile the following is
> mentioned:
> 
> ================
> 
> Size Issues
> 
> Geometry: The Shapefile format explicitly uses 32bit offsets and so
> cannot go over 8GB (it actually uses 32bit offsets to 16bit words).
> Hence, it is is not recommended to use a file size over 4GB.
> 
> Attributes: The dbf format does not have any offsets in it, so it can be
> arbitrarily large.
> 
> ================
> 
> 
> 
> Yet on the ESRI website:
> 
> ================
> 
> Geometry limitations
> 
> There is a 2 GB size limit for any shapefile component file, which
> translates to a maximum of roughly 70 million point features. The actual
> number of line or polygon features you can store in a shapefile depends
> on the number of vertices in each line or polygon (a vertex is
> equivalent to a point).
> 
> ================
> 
> 
> 
> I assume the OGR web page is wrong, or has a different meaning outside
> of ESRI S/W use.

Darren,

Yes, as underlined by Chaitanya the actual limit depends on the software 
implementation. Actually the limit in OGR was 4 GB for the .SHP, and AFAICS 
unlimited for DBF.

I've added in http://trac.osgeo.org/gdal/changeset/26657 a 2GB_LIMIT=YES layer 
creation option (and SHAPE_2GB_LIMIT configuration option) that will enforce 
the 2GB limit. And in http://trac.osgeo.org/gdal/changeset/26658 a change so 
that when the layer is reached the file is properly closed with the valid 
information.

Spliting files over several files would properly need to be done outside of 
this, in a script, by restarting from the source layer at the index next to 
the one that was last written in the shapefile.

Best regards,

Even
-- 
Geospatial professional services
http://even.rouault.free.fr/services.html