[OSGeo-Discuss] Re: idea for an OSGeo project -- a new, open data format

[P Kishor]

disclaimer: I speak from a position of complete ignorance about SDF,
so everything I might say could be wrong.

On 11/14/07, Robert Bray <rbray at robertbray.net> wrote:
> A little clarification on SDF, since it comes up once or twice in this
> thread. Jason's earlier descriptions of it's capabilities are pretty good.
> It supports multiple Feature Classes / Tables per file, strongly typed
> properties, multiple geometry properties per class, with seperate R-Trees
> for each geometry property. All of this is stored in a single file that is
> heavily optimized for spatial reads. The SDF FDO Provider suppports a
> multiple reader / single writer model. The geometries themselves include
> simple features + circular arcs in 2D, 2D with Z, 2D with M, or 2D with Z &
> M. The R-Trees are currently 2D.
>

SDF does sound the cat's meow.

> Is it an open format?  ABSOLUTELY (we just never wrote a spec, but I am
> willing to get it done)
>

What is the licensing of this format? What is the plan for licensing
of this format? Is the format going to be put out in public domain
ever? This is an important discussion, but it probably doesn't belong
in this thread. I bring it up here just to make sure it is not
forgotten.

> Another little known fact is that in the process of creating SDF we
> (Autodesk) literally wrote the code three times. The first time we built SDF
> on BerkleyDB, a great open source project but it has some fairly significant
> license fees for using it in a proprietary product. The second time we wrote
> it on SQLite, however the performance penalaty of the Relational layer was
> significant (read orders of magnitude). The third time we chose to strip
> away the SQLite relational layer and built directly on the SQLite Backend
> components (B-Tree, Pager, and OS Interface). In the end the third
> implementation actually turned out to be faster than BDB and is the one we
> use today.
>

By not having the relational layer we lose a lot in terms of attribute
data handling, or is that still somehow preserved in SDF?

Keep in mind, SQLite has gone through major revisions, some of them
bringing huge speed bumps in the process. That said, different folks
have different evaluation critieria -- I believe it was Steve W who
said that pretty much all he cared about was speed. For him it seems
SDF might work very well. For me, I don't mind sacrificing some speed
for ease and flexibility of related data storage, querying, and
retrieval. How can we achieve both?

> All this said, I'd really like to understand everyones requirements for this
> new format. If SDF fits thats great, if not thats ok too. We are always
> happy to contribute what we can to the community.

Thanks Bob. As you can see, we are still articulating the
requirements, and while we might be accused of requirement wanking,
design by committee might be better than no design at all.

I find two problems with Shapefiles -- one, that it is not in public
domain (I am not even sure of what licensing there is on it), and
while ESRI is not likely to pull a Unisys on us, it just is
philosophically better to free if possible. The second, more major
problem is Shapefile's antiquated data technology. DBF is a royal pain
in the derierre with its 10 char column name, no relational tables, no
indexing of data constraints. The geometry part of Shapefiles seems to
be pretty good and adequate; it is the data part that is the problem.

>
> Bob
>
> ----- Original Message -----
> From: "Michael P. Gerlek" <mpg at lizardtech.com>
> To: <punkish at eidesis.org>; "OSGeo Discussions" <discuss at lists.osgeo.org>;
> <bitner at gyttja.org>
> Sent: Tuesday, November 13, 2007 12:23 PM
> Subject: RE: [OSGeo-Discuss] Re: idea for an OSGeo project -- a new,open
> data format
>
>
> >> > Regarding the suggestion that MapServer takes on this new format as
> > the
> >> > primary format:  I think this is way beyond the scope of what OSGeo
> > should
> >> > be doing.
> >
> > I agree with bitnerd.  If the MapServer team thinks this is a valuable
> > and worthwhile format, they will adopt it at some point.  It would not
> > be unreasonable for them to step back and see how thing progresses
> > before deciding to spend their valuable ergs on it.  The burden is on
> > the "OpenShape" people to show the idea is worthwhile and meritorious.
> >
> > (My two cents on the "standards" question: OSGeo is not a standards
> > organization, but / however / on the other hand / nonetheless one of the
> > reasons OSGeo exists is to foster such collaborations.  If some people
> > want to try and develop something new like this, I'm all in favor of
> > OSGeo offering mailing list and wiki space to help out.  Declaring this
> > to be a "standard" effort, however, is probably premature in any case --
> > more useful at this point to see the idea sketched out further, see
> > who's interested, etc.)
> >
> > -mpg
> > _______________________________________________
> > Discuss mailing list
> > Discuss at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/discuss
> >
>
>
> _______________________________________________
> Discuss mailing list
> Discuss at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/discuss
>


-- 
Puneet Kishor
http://punkish.eidesis.org/
Nelson Institute for Environmental Studies
http://www.nelson.wisc.edu/
Open Source Geospatial Foundation (OSGeo)
http://www.osgeo.org/
Summer 2007 S&T Policy Fellow, The National Academies
http://www.nas.edu/