[OSGeo-Discuss] idea for an OSGeo project -- a new, open data format

[Stephen Woodbridge]

P Kishor wrote:
> So, I am thinking, Shapefile is the de facto data standard for GIS
> data. That it is open (albeit not Free) along with the deep and wide
> presence of ESRI's products from the beginning of the epoch, it has
> been widely adopted. Existence of shapelib, various language bindings,
> and ready use by products such as MapServer has continued to cement
> Shapefile as the format to use. All this is in spite of Shapefile's
> inherent drawbacks, particularly in the area of attribute data
> management.
> 
> What if we came up with a new and improved data format -- call it
> "Open Shapefile" (extension .osh) -- that would be completely Free,
> single-file based (instead of the multiple .shp, .dbf, .shx, etc.),
> and based on SQLite, giving the .osh format complete relational data
> handling capabilities. We would require a new version of Shapelib,
> improved language bindings, make it the default and preferred format
> for MapServer, and provide seamless and painless import of regular
> .shp data into .osh for native rendering. Its adoption would be quick
> in the open source community. The non-opensource community would
> either not give a rat's behind for it, but it wouldn't affect them...
> they would still work with their preferred .shp until they learned
> better. By having a completely open and Free single-file based, built
> on SQLite, fully relational dbms capable spatial data format, it would
> be positioned for continued improvement and development.
> 
> Is this too crazy?
> 
Well, yes and no.

I think this idea of an open free format has some merit.
I'm not sure the your proposed solution has had enough analysis and 
design or requirements backing it to have merit, but it is a stake in 
the ground.

I think you have to first look at the plus and minus of the existing 
formats. What are your design requirements? One size fits all will not 
fit anyone well.

For me shapefiles work well because they are fast! This is my number one 
requirement and I can live with most other issues. For North America, I 
have in the ballpark of 27M street segments, and 20-30GB of shapefile 
data with qix indexes and tileindexes. I have to serve 5M map draws a 
week (~440k/day) in under 1.5sec per draw. So fine if you say continue 
to use shapefiles, but not so much if you tell most people that because 
you get very little adoption.

There needs to be REAL value in a new data format. Having a new format 
for the sake of being able to say it is free, has zero value in my book 
if there are not other strong compelling reasons for people to move to 
it. If people do not move to it, it will not become a defacto standard 
or a standard of any use. Data vendors will not support it for data 
distribution, etc. It would be better to invest everyone's time 
elsewhere if there is not real concrete value to be had by this new format.

My 2 cents,
   -Steve W.