[mapserver-users] Is there still a 2GB size limit on shapefiles?

[Stephen Woodbridge]

I think it would be interesting to see how/if ESRI responded to a change 
in shapelib that allow it to detect standard files or large files and 
allow application built on it to work with both. If enough FOSS 
application supported that then ESRI might be forced to deal with it or 
have a lot of clients with files that they can not read.

 From my usage, I almost always have a problem with the DBF getting too 
large mostly because of a lot of attribute columns. A lot of my data 
processing script have an option to split the output into numbered parts 
like:

file-01.*
file-02.*
etc

and then I use a tileindex to reassemble the parts in mapserver or ogr. 
If would be nice if we didn't have to worry about this and could just 
make large files. Personally, shapefiles are a convenient and fast 
format to work with and only on rare occasions do I have to deal with an 
ESRI client. Since I have a work around for them, that is not a problem.

-Steve W

On 5/9/2013 1:35 PM, Frank Warmerdam wrote:
> Folks,
>
> The changes I made were mostly focused on supporting large DBF files
> which don't have the 32bit offsets issue.
>
> The offsets for shapefiles are actually to 16bit words, so in theory if
> the offsets were treated as unsigned 32bit values they could address up
> to 8GB .shp files.  I don't recall however, if I ever got that working.
>   Even if it was done it is not clear to me these files would work
> smoothly with other applications.
>
> I'd been hoping someone else might have authoritative comment in this
> thread. :-)
>
> Best regards,
> Frank
>
>
> On Wed, May 8, 2013 at 9:49 PM, Stephen Woodbridge
> <woodbri at swoodbridge.com <mailto:woodbri at swoodbridge.com>> wrote:
>
>     In general the shapefile spec is owned by ERSI and unless they
>     change the spec you will have a problem. Last I knew, there are
>     pointers/offsets inside the shapefile that are only 32bits wide per
>     the spec. it is because of these that you are limited to 2GB.
>
>     At some point I thought Frank W or someone was working on a version
>     of shapelib that could support greater than 2GB, but only programs
>     that used that version would be able to read/write these larger
>     files. I have not heard anything on that front in a while so I don't
>     know if that was just talk, or code ever materialized.
>
>     -Steve W
>
>
>     On 5/9/2013 12:32 AM, Rahkonen Jukka wrote:
>
>         Hi,
>
>         I am not sure either if reading >2GB shapefiles works with the
>         native
>         shp reader on all platforms, but as the ticket says, you should be
>         able to use OGR connection in the mapfile even with your current
>         version. If you use OGR you can also split the shapefile into pieces
>         and read them through the ogr tileindex system which may be faster
>         sometimes.
>
>         -Jukka Rahkonen-
>
>         __________________________________ Andrea Peri wrote:
>
>             If it don't work you should evalute the option to use
>             spatialite.
>             It work well with dataset greater than 2GB . I use it with
>             mapserver 6.3dev and gdal 1.10.
>
>
>
>
>         2013/5/9 Roger André <randre at gmail.com
>         <mailto:randre at gmail.com><mailto:randr__e at gmail.com
>         <mailto:randre at gmail.com>>>
>         Pretty sure I just hit
>         this<http://trac.osgeo.org/__mapserver/ticket/3514
>         <http://trac.osgeo.org/mapserver/ticket/3514>> problem in my 6.0.3
>         local stack.  I haven't tested it in the current release, but does
>         anyone know for sure that a shapefile with any component larger than
>         2GB works in 6.2?
>
>         Roger
>
>         _________________________________________________ mapserver-users
>         mailing list
>         mapserver-users at lists.osgeo.__org
>         <mailto:mapserver-users at lists.osgeo.org><mailto:mapserver-users at __lists.osgeo.org
>         <mailto:mapserver-users at lists.osgeo.org>>
>
>
>     http://lists.osgeo.org/__mailman/listinfo/mapserver-__users
>     <http://lists.osgeo.org/mailman/listinfo/mapserver-users>
>
>
>
>
>
>         -- ----------------- Andrea Peri . . . . . . . . . qwerty àèìòù
>         ----------------- _________________________________________________
>         mapserver-users mailing list mapserver-users at lists.osgeo.__org
>         <mailto:mapserver-users at lists.osgeo.org>
>         http://lists.osgeo.org/__mailman/listinfo/mapserver-__users
>         <http://lists.osgeo.org/mailman/listinfo/mapserver-users>
>
>
>     _________________________________________________
>     mapserver-users mailing list
>     mapserver-users at lists.osgeo.__org
>     <mailto:mapserver-users at lists.osgeo.org>
>     http://lists.osgeo.org/__mailman/listinfo/mapserver-__users
>     <http://lists.osgeo.org/mailman/listinfo/mapserver-users>
>
>
>
>
> --
> ---------------------------------------+--------------------------------------
> I set the clouds in motion - turn up   | Frank Warmerdam,
> warmerdam at pobox.com <mailto:warmerdam at pobox.com>
> light and sound - activate the windows | http://pobox.com/~warmerdam
> <http://pobox.com/%7Ewarmerdam>
> and watch the world go round - Rush    | Geospatial Software Developer