[gdal-dev] Large shapefile issues

[Stephen Woodbridge]

Unless something has changed, I have never been able to work with dbf 
file over 2GB using shapelib.

-Steve W

On 11/25/2013 5:52 PM, Even Rouault wrote:
> Le lundi 25 novembre 2013 11:42:23, CARMAN, Darren a écrit :
>> Hi List
>>
>>
>>
>> I notice on the OGR formats page for ESRI Shapefile the following is
>> mentioned:
>>
>> ================
>>
>> Size Issues
>>
>> Geometry: The Shapefile format explicitly uses 32bit offsets and so
>> cannot go over 8GB (it actually uses 32bit offsets to 16bit words).
>> Hence, it is is not recommended to use a file size over 4GB.
>>
>> Attributes: The dbf format does not have any offsets in it, so it can be
>> arbitrarily large.
>>
>> ================
>>
>>
>>
>> Yet on the ESRI website:
>>
>> ================
>>
>> Geometry limitations
>>
>> There is a 2 GB size limit for any shapefile component file, which
>> translates to a maximum of roughly 70 million point features. The actual
>> number of line or polygon features you can store in a shapefile depends
>> on the number of vertices in each line or polygon (a vertex is
>> equivalent to a point).
>>
>> ================
>>
>>
>>
>> I assume the OGR web page is wrong, or has a different meaning outside
>> of ESRI S/W use.
>
> Darren,
>
> Yes, as underlined by Chaitanya the actual limit depends on the software
> implementation. Actually the limit in OGR was 4 GB for the .SHP, and AFAICS
> unlimited for DBF.
>
> I've added in http://trac.osgeo.org/gdal/changeset/26657 a 2GB_LIMIT=YES layer
> creation option (and SHAPE_2GB_LIMIT configuration option) that will enforce
> the 2GB limit. And in http://trac.osgeo.org/gdal/changeset/26658 a change so
> that when the layer is reached the file is properly closed with the valid
> information.
>
> Spliting files over several files would properly need to be done outside of
> this, in a script, by restarting from the source layer at the index next to
> the one that was last written in the shapefile.
>
> Best regards,
>
> Even
>