[GRASS-dev] GRASS GIS errata proposal

[Maris Nartiss]

Rationale.
GRASS GIS is widely used in scientific, private and government sector.
Scientific theories, environmental decisions and actions depend on the
outcome of spatial analysis performed with GRASS GIS. Any errors in
analytical modules might lead to erroneous conclusions and actions
based on them. Currently GRASS GIS project has no clearly outlined
procedure how to warn GRASS GIS users of found issues in analytical
modules that could lead to good looking but erroneous analysis
results. Bug reporting system (Trac) contains different kind of issues
and is hard to follow for non-developers; other issues are fixed
without opening a bug report in Trac.

Proposal.
GRASS GIS could publish an errata document listing found issues in
official releases. PSC would make a final decision if issue is serious
enough to announce an erratum. Only issues causing meaningful but
incorrect results should get an erratum name/number and announcement.
An RFC should be adopted to clearly outline procedure and criteria of
issuing a GRASS GIS erratum. Following steps should send out a clear
message that GRASS project is taking quality issues seriously and
should be considered a choice #1 for high quality spatial analysis
(aka world domination). Target audience would be scientists basing
their research on GRASS GIS, GIS analysts.

Final remarks.
As final decision is made by PSC, it will increase PSC work load.
There must be nominated a responsible person for GRASS GIS erratum
text preparation and publishing.
Probably it could be a nice idea to set up low traffic mailing list
for receiving GRASS GIS errata issues. Should be discussed if existing
announcement mailing list should be reused.
An option of adding a GRASS GIS version entry to ALL GRASS map
metadata should be discussed. It would allow to know exact version
used to generate or making of last edits of particular map.

Will wait for discussion about proposal in general, corrections for
RFC draft (my English is not a top one :( ), other potential changes
to be done to increase GRASS GIS trustworthiness and quality.

GRASS GIS erratum RFC draft
Any bug reporter or developer can nominate a bug for escalating to
GRASS GIS erratum issue. Nomination is done by adding a notice to bug
at bug tracker or discussing directly at developer mailing list. Any
nomination is discussed in developer mailing list to gather necessary
information on it's scope, impact, causes and solutions. GRASS PSC
evaluates nomination based on information provided by bug report,
discussion in developer mailing list and/or other sources. PSC makes a
final decision if nominated bug matches criteria of issuing GRASS GIS
erratum. Appointed person (WHO?) prepares errata text publishes it.

Criteria for bug escalating into erratum. Bug must be present in a
official GRASS GIS release. Bug must cause generation of incorrect
analysis results that are not so easy to notice. Module crashes or
bugs causing easy to identify incorrect results should not be given an
erratum. Examples of possible erratum worth bugs are single cell shift
of raster result, not enough randomness of expected random module
output, loss of output precision due to incorrect floating point
handling etc.

GRASS GIS erratum text should contain:
it's number;
date of issue;
name(s) of affected module(s);
information about affected release(s);
a short description of problem;
steps resulting in incorrect output (i.e. specific input parameter combination);
current state of problem;
references to bug report (Trac bug number), developer mailing list thread;
any other information relevant to erratum.

GRASS GIS errata might receive updates, if it's found to be necessary
(i.e. notice of fixing issue, issue scope update etc.).

GRASS GIS errata is added to file ERRATUM in all affected branches and
any upcoming release branch (trunk). It is published on GRASS GIS web
site (Where?). Erratum text and any of it's updates are sent out to ??
mailing list. If erratum has related entry in the bug database, a
notice of issuing erratum is added to the bug entry.


WBR etc.
Maris.