[gdal-dev] Fastest vector format for combining shapefiles

[Vincent Schut]

Matt Wilkie wrote:
>>
>> I am combining some GIS data where each layer is divided to around 
>> thousand
>> separare shapefiles by mapsheets. 
> We are in the same boat. For the moment I'm exploring the approach of 
> aggregrating map sheets, creating super tiles, until we approach the 
> 2gb shapefile limit. I've not found a good method for predicting when 
> we'll hit that limit though so it's error prone with a lot of repeats.
>
> Looking forward, the as-yet unreleased file geodatabase API/spec would 
> be a solution. ESRI has said they will do so, but it's been several 
> years since they first announced it and when it is finally is released 
> there is no guarantee it will be under license terms open source 
> projects can use. This isn't to say it will be unusable either, we 
> just don't know.
>
> I've been reading about Binary XML, which has an open source library 
> published by Cuberwerx, http://www.cubewerx.com/bxml. They say it is a 
> drop in replacement for XML. Perhaps binary gml would be a good format 
> for >2gb data packages? How much work would it be to add this to 
> gdal/ogr? I invite developers to submit price estimates to me. I don't 
> have an active project assigned to this or a budget, but I can't get 
> one without an idea of what to ask for either.
Folks,

I'm by no way a vector expert, but for this kind of stuff I'd go the 
postgis way, at least for intermediate storage. You won't have any 2GB 
limit on vector db size, have reasonable fast rw access to the data, and 
will be able to do spatial operations on the db directly. Once you've go 
all your data in the db the way you want it, you can export to any ogr 
supported vector format.
For a platform independent way of distributing vector files that are 
 >2GB, I'd check out spatialite; however, I don't know in what shape 
ogr's support for real spatialite db's is, so you'd have to do some 
research here. Note that, once 32bit windows OS come in sight, this 
might put a barrier at 2GB no matter what vector file format you use, 
though I'm not sure about that either. I try not to use windows if 
possible :-)

just my 2 cents...
Vincent.
>
> best regards,
>
> matt wilkie
> --------------------------------------------
> Geomatics Analyst
> Information Management and Technology
> Yukon Department of Environment
> 10 Burns Road * Whitehorse, Yukon * Y1A 4Y9
> 867-667-8133 Tel * 867-393-7003 Fax
> http://environmentyukon.gov.yk.ca/geomatics/
> --------------------------------------------
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev