[OSGeo-Discuss] idea for an OSGeo project -- a new, open data format

[blammo]

All,

My two cents worth . . . .

Interesting thread, and timely too.  I had a discussion  yesterday  
about using SDF as a neutral format.  I'm kinda wondering though what  
the end use something like this is intended for.   Is it to store a  
dataset by a user of for serving the data via the web or both.     
I've come to like the SHP file format especially because it is basic  
and is abstracted as much as it is, makes building systems much  
easier in the long run.  I also understand that it doesn't quite come  
up to snuff when used as a end user storage medium, very cumbersome  
and hard to keep track of all the pieces.  I did work with the FME  
FFS file type with some good results too, but I don't know if that is  
an open format or not.

The FDO stuff is a bit new for me to develop a production service,  
although, I'm using it already (transparently) with a AutoCAD/Oracle  
update mechanism via  AutoCAD Map.

A single file format would certainly help with User publishing issue  
that we're starting to run into, where individuals need to publish  
small local area (Project related) datasets and keep them up to date  
over a relatively short time, like about a yaer for example, then  
they become part of the larger system.

I've also been working with 3D visualization for the last couple of  
years, and keep coming back to the best data model to use for the  
transport layer, it's currently settled out in our office with a 2D  
database, and 3D attributes, but this doesn't seem very extendable  
into the future.   A composite data format might have room in it some  
of my 3D data as well, even as a derived dataset would be an  
acceptable option at this point.

One last piece to this would be styling, it would be real handy to  
apply the styling in the composite file as well, both 2d and 3d.  I  
mean as long as your going to build something . . .  Although, I  
think I may have come close to describing the DWG format here in the  
end  :c)

I've tended to use the formats that work the best for the task for  
the last few years, and SHP files seem to still be the best for Web  
distribution of the data,with a good, highly indexed, server side,  
spatial SQL database being next in line.

I would like to hear about intended uses and business problems solved  
by using a composite, single file, approach to the data storage.

bobb


On Nov 13, 2007, at 9:42 AM, Landon Blake wrote:

> Puneet,
>
> You wrote: "Is this too crazy?"
>
> I don't think this idea is crazy at all. In fact, I think it is a very
> good idea. I do have a couple of comments, which you can read below:
>
> You wrote: "What if we came up with a new and improved data format --
> call it
> "Open Shapefile" (extension .osh)..."
>
> I think we would have to completely avoid the term "Shapefile" as  
> it is
> probably an ESRI trademark.
>
> You wrote: " that would be completely Free,
> single-file based (instead of the multiple .shp, .dbf, .shx, etc..."
>
> Is there a problem with a multiple file format? I have tinkered with
> some different binary file formats, and it seems that separating some
> information in a spatial data set (like indexes, for example) makes it
> easier to create programs that parse and write the format.
>
> You wrote: " ...and based on SQLite, giving the .osh format complete
> relational data
> handling capabilities."
>
> I would prefer tightly integrating any software package, even if it  
> was
> FOSS, into a new data format. SQLite is written in the C programming
> language, as an example, and that doesn't do me a lot of good as a  
> Java
> programmer. Tightly integrating a Java library or program wouldn't do
> much good for a programmer using C. That is the real beauty of an open
> file format: If it is designed properly your programming language
> doesn't matter!
>
> I did a little brainstorming for a binary file format that could  
> replace
> Shapefiles. Its called BOFF, or Binary Open Feature Format. I talked
> over small bits of the design for BOFF with Frank Wammerdam, who
> expressed some small interest in it at the time.
>
> Perhaps it would offer some ideas?
>
> I'd be very interesting in offering Java support for a shapefile
> replacement endorsed by the OSGeo.
>
> The Sunburned Surveyor
>
> -----Original Message-----
> From: discuss-bounces at lists.osgeo.org
> [mailto:discuss-bounces at lists.osgeo.org] On Behalf Of P Kishor
> Sent: Tuesday, November 13, 2007 6:53 AM
> To: OSGeo Discussions
> Subject: [OSGeo-Discuss] idea for an OSGeo project -- a new, open data
> format
>
> So, I am thinking, Shapefile is the de facto data standard for GIS
> data. That it is open (albeit not Free) along with the deep and wide
> presence of ESRI's products from the beginning of the epoch, it has
> been widely adopted. Existence of shapelib, various language bindings,
> and ready use by products such as MapServer has continued to cement
> Shapefile as the format to use. All this is in spite of Shapefile's
> inherent drawbacks, particularly in the area of attribute data
> management.
>
> What if we came up with a new and improved data format -- call it
> "Open Shapefile" (extension .osh) -- that would be completely Free,
> single-file based (instead of the multiple .shp, .dbf, .shx, etc.),
> and based on SQLite, giving the .osh format complete relational data
> handling capabilities. We would require a new version of Shapelib,
> improved language bindings, make it the default and preferred format
> for MapServer, and provide seamless and painless import of regular
> shp data into .osh for native rendering. Its adoption would be quick
> in the open source community. The non-opensource community would
> either not give a rat's behind for it, but it wouldn't affect them...
> they would still work with their preferred .shp until they learned
> better. By having a completely open and Free single-file based, built
> on SQLite, fully relational dbms capable spatial data format, it would
> be positioned for continued improvement and development.
>
> Is this too crazy?
>
> -- 
> Puneet Kishor
> _______________________________________________
> Discuss mailing list
> Discuss at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/discuss
>
>
> Warning:
> Information provided via electronic media is not guaranteed against  
> defects including translation and transmission errors. If the  
> reader is not the intended recipient, you are hereby notified that  
> any dissemination, distribution or copying of this communication is  
> strictly prohibited. If you have received this information in  
> error, please notify the sender immediately.
> _______________________________________________
> Discuss mailing list
> Discuss at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/discuss
>