[OSGeo-Discuss] Software Standards - A Start

[Frank Warmerdam]

Landon Blake wrote:
> There were a few recent threads about software standards. Some of these 
> threads dealt specifically with the role of OSGeo in software standards 
> development and the OGC. I didn’t what this conversation to die out, 
> because I think it is very important to the survival of open source 
> software, and a topic that is important to me personally.
> 
>  
> 
> I know we may not be ready for a mailing list on this topic just yet, 
> but I did throw up a page on the OSGeo wiki. You can find it here:
> 
> http://wiki.osgeo.org/index.php/Software_Standards#Suggested_Standards

Landon,

Your suite of questions on that page could each have a book written
about them!

I think one page worth breaking off might be a list of significant
standards, and for each listing OSGeo (and other FOSS4G) packages that
implement them as well as a pointer off to resources about the standard.

I'm not good at conversing-via-wiki-editing so I'm just going to try
and provide my one line answer to a few of the questions.

5) Do We Really Need Software Standards?

I'm not sure I'd refer to them as "Software Standards".  They are
interface standards -- they define the interface between different
software packages.  Generally our standards of interest are file formats
or web protocols.

I think standards are very important to improving developer and user
effectiveness, though I suppose we could hobble along in a world of
software silos with hackey custom importers from each others proprietary
data formats.

6) What Role Do Software Standards Have In The Development Of Open Source 
Geospatial Software?

I think interface standards make it more practical for folks to assemble
software stacks including open source software components because there
are well defined standardized interfaces.  In particular I think the
open source world is relatively weak at producing an integrated "platform"
that does everything.

Producing a complete GIS platform often involves dozens of software
developers working together in one place over decade type time scales.
Here I am thinking of products like ArcGIS, with integrated components
for doing allmost everything you might think of.

With all due respect, it is hard for open source developers to accomplish
this sort of thing.  So it is very important that we can break our tasks
down into more componentized approaches with well defined interfaces.  So
we can have one team working on a web mapping engine.  Others on
web mapping client interfaces, others on low level geometry engines, etc.

Standards not only provide us with interfaces to use between open source
components, but they also provide interface points at which we can hope
to fit into a mixed stack of proprietary and open source software in
use at client sites.

Note that standards aren't absolutely critical for this, but it substantially
evens the playing field.

7) Should You Have To Pay For A Copy Of A Software Standard?

I think having software standards freely available substantially
encourages adoption and is in keeping with the open source view
of enablement.  I detest the fact that ISO's funding model is based
on charging for standards, and I think this reduces interest in ISO.
I applaud OGC for making their finalized standards freely available.

8) Should The Development Of A Software Standard Be Open? How Open?

The more open, the more likely folks will get involved which can result
in improving the standards before they are released.

Openness does tend to slow down standards development, as the opinions of
more people have to be considered, and integrated.

I think a wide variety of levels of open-ness *can* work.

At one end, a single organization can develop a standard and publish it
encouraging wide use.  For example Shapefiles are an example of this working
fairly well.

A next step is a standard developed by a smallish and unofficial group
representing several different parties in an adhoc manner.  GeoTIFF is quite
a successful example of this I think.  Several vendors got together on a
mailing list, lead by two principals, and produced the most widely
supported geospatial raster format.  The involvement of several vendors
really helped getting it launched (and the open source libgeotiff
reference implementation was also valuable!)

I think the geo-rss and WMS Tile Caching standards also fall into this
category.

The next step up is the formal standards organization like ISO or OGC.
Here formal procedures are in place in an attempt to prevent any one
member from taking over and unduely influencing the standards.  This is
the world where one of the greatest issues is vendors trying to work
together without getting screwed by the others.  The formal structures
are setup to provide a "safe and level meeting ground".  But the
formality of such organization slows down standards development.

I'm personally comfortable with some standards being developed at different
points on the "formality/openness" scale.  All levels have benefits and
drawbacks.

9) Do Software Standards Have To Be Approved By A Standards Body To Be Valid?

I think the answer is "no", *but* having a standard approved and/or
managed by a formal standards body provides a comfort level to many
organizations.  Implementing organizations are often more comfortable
with standards not under the direct control of a rival.  Customer
organizations often find it easier to require implementation of particular
standards in RFPs if it has a formal approval.

 From my perspective, one great weakness of the GeoTIFF standard is that
there is no mechanism to get clarification on ambiguities, nor is there
a process to come out with new revisions.  The original defacto working
group has dispersed to some extent and now there is no obvious way to
approve a new version.  Formality of standards bodies helps address
this sort of problem.

10) When Does A Widely Used File Format Become A De Facto Software Standard?

Umm, when it is widely used?

11) What The Advantages Of Text Over Binary For Open File Formats? Is XML The 
Best Solution?

I think question is in a different class from the others.

XML is clearly the solution.  If it doesn't seem right for you, then fix
your problem so it fits XML.

Ok, that wasn't really a serious response.  XML gives a well structured
way of expressing complex data structures and has well defined parsing
rules that are well understood and widely implemented. It is also (a bit)
human readable.  So when compactness and efficiency of access is not
paramount I think it is reasonable to express data in XML profiles.

Binary formats, in my opinion, are often more efficient for processing
(saving an atof() for every number in a big dataset is not insignificant!)
More importantly it is very hard to use XML in an indexed fashion.  For
instance, to read the 477th feature in a GML dataset, you basically have
to parse through to that point in the file.  But with a binary format
like Shapefiles it is easy to have indexes pointing to particular features.

You could, in theory, do this with XML to, but likely only by severely
constraining the XML generality and it would require keeping seperate
index files in-sync with the XML.

So, for large datasets that I want fast access to (working formats
for instance) I would generally prefer a binary format.  In specific
cases this might not be appropriate though.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org