[gdal-dev] Large shapefile issues

[Chaitanya kumar CH]

Darren,

These limitations are a result of the shapefile reader/writer
implementation. While the specification permits up to 8GB, the
implementation might not use it fully.

According to the specification a feature offset is stored as a 32-bit
integer to a 16-bit word. So, that's (2^32)*16 bits or (2^32)*2 bytes. To
fully utilize this, we need to use unsigned 32-bit integers and check for
overflows.
Originally, the shapelib library used by GDAL had the same limit as ESRI's.

If you are reaching the file size limits, I suggest you look into some
other file format to store your data. SQLite/SpatiaLite is a good option.

A hacky way to split the features across multiple shapefiles is to use the
-where option in ogr2ogr. You can filter based on the FID values.



On Mon, Nov 25, 2013 at 4:12 PM, CARMAN, Darren <
Darren.Carman at astrium.eads.net> wrote:

> Hi List
>
>
>
> I notice on the OGR formats page for ESRI Shapefile the following is
> mentioned:
>
> ================
>
> Size Issues
>
> Geometry: The Shapefile format explicitly uses 32bit offsets and so cannot
> go over 8GB (it actually uses 32bit offsets to 16bit words). Hence, it is
> is not recommended to use a file size over 4GB.
>
> Attributes: The dbf format does not have any offsets in it, so it can be
> arbitrarily large.
>
> ================
>
>
>
> Yet on the ESRI website:
>
> ================
>
> Geometry limitations
>
> There is a 2 GB size limit for any shapefile component file, which
> translates to a maximum of roughly 70 million point features. The actual
> number of line or polygon features you can store in a shapefile depends on
> the number of vertices in each line or polygon (a vertex is equivalent to a
> point).
>
> ================
>
>
>
> I assume the OGR web page is wrong, or has a different meaning outside of
> ESRI S/W use.
>
>
>
> I notice that the PostGIS utility pgsql2shp stops processing with an error
> once the dbf file goes over 2GB. Is there any way to get ogr2ogr to do this?
>
>
>
> Ideally, something I can’t find a lot of information about online, is
> there a way to get ogr2ogr to start writing to a new shapefile at a certain
> processing point (number of objects or file size)?
>
>
>
> Alternatively, assuming the OGR website text is correct for files it
> creates, is there any utility that could be used to split the created
> shapefile files into ones with a size less than 2GB?
>
>
>
> Thanks in advance
>
> Darren
>
>
>
>
>
>
>
> *Darren Carman*
> Senior Software Engineer
>
> GEO-Information Services
>
>
>
> *Astrium Services*
>
> Tel +44 (0)1252 362138
>
> http://www.astrium-geo.com <http://www.infoterra.co.uk>
>
>
>
>
>
> Infoterra Ltd. Is part of the Astrium GEO-Information Services Division
> and a wholly owned subsidiary of Astrium, Europe's leading space systems
> and services specialist.
>
> Disclaimer. The information contained in this e-mail and its attachments
> are confidential and intended only for the use of the named addressee(s).
> If you are not the intended addressee, please do not read, copy, use or
> disclose this message or its attachments. If you have received this message
> in error, please notify the sender immediately and delete or destroy all
> copies of this message and attachments in all media. Any views or opinions
> expressed are solely those of the author and do not necessarily represent
> those of Infoterra Ltd and shall not form part of any binding agreement.
>
> Infoterra Limited a company registered in England under number 2359955
> and having its registered office at Europa House, Southwood Crescent,
> Farnborough, GU14 0NL. VAT number GB 476 0468 27.
>
> P Before printing, think about the environment
>
>
>
>
>
>
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>



-- 
Best regards,
Chaitanya kumar CH.

+91-9494447584
17.2416N 80.1426E
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20131125/92e84b86/attachment-0001.html>