[gdal-dev] Re: Fastest vector format for combining shapefiles

[Rahkonen Jukka]

Jukka Rahkonen writes:

> 
> Hi,
> 
> I am combining some GIS data where each layer is divided to around
thousand
> separare shapefiles by mapsheets. Now I would like to store all the
35000
> shapefiles to something that is more easy to handle. At first pushing
each layer
> to own Spatialite database feeled perfect, but I have problems with
one layer
> which has rather lot of data. Appending shapefiles one by one to
Spatialite
> database gets too slow after the database file has reached a size of
around 6
> gigabytes. Up till 3-4 gigabyte file size appending data to Spatialite
is pretty
> fast and because it is a database I guess I will use that for small
layers.  But
> what might be the fastest vector format that ogr supports to collect
the big
> layer (thousand shapefiles with total size of about 10 gigabytes)
together?  I
> would prefer some file based format because data goes to long-time
storage, but
> I can use Oracle or PostGIS in between if it is faster to do the
conversion in
> two steps.  What is recommended? Shapefiles, MapInfo tab, Oracle,
PostGIS or
> something else?

I can tell now that shapefile format is not suitable at all. The shp
part can 
obviously not go over 2 GB limit because after that ogr2ogr throws 
these error messages:
ERROR 1: Error in psSHP->sHooks.FSeek() or fwrite() writing object to
.shp file.  

Dbf seems not to have such size limit because it grew up till 32
gigabytes.
I will try MapInfo tab before I believe that it is just best to keep the
1000
shapefiles or upload them all to PostGIS.

-Jukka-