[GRASS5] 0 != no data

[Justin Hickey]

Hello all

First of all, thank you to all who gave comments and suggested
improvements. They were greatly appreciated.

Roger Bivand wrote:
> 
> Justin and friends,
> 
> Would this modified version answer Moritz' question? I agree with him 
> that we might risk getting a limited response because a regular 4.*
> user mightn't grasp the issue. I haven't got a 4.3 installed, but as
> far as I remember, there aren't any NULL values in the 4,* series are
> there? Can someone please check?

If I recall correctly, GRASS 4.x does not have NULL values.

> I've also added a third, compromise, choice - remove
> 0 as NULL from 5.1, but retain it in 5.0: the motivation is to lower 
> the threshold for 4.* users migrating to 5.*. I shortened the draft
> by taking out the script details - they are valuable, but could be
> included as a link to a note on the website.

OK thanks for your suggestions.

Below is Roger's post with Helena's suggestion (thanks Helena) added in
as well as a couple of my own. If there are no more comments for this
version, I'll send it to the user's list tomorrow.

Summary of my changes:

Changed 

    NULL is implemented as a separate mask of bits representing 
    "data"/"no data",

to

    NULL is implemented as a separate mask of bits representing "no 
    data",

since I think saying "NULL represents "data" " is confusing.

Added

    Please note that incidents caused by such confusion have been 
    reported.

to the same paragraph the above comes from to indicate that this issue
is causing problems for people and it is not just a wish of the
developers to make this change.

Added

    Please see details of the proposed script at the following site

    http://www.geog.uni-hannover.de/grass/nullConvert.html

to indicate where users could find details of the conversion script.

Added 

    Note that GRASS 5.0 can convert any value to NULL and vice versa.

to include Markus's earlier comment.

------------------------------------------------------------
Subject: Representation of "no data": we need your opinion
 
Hello all users
 
Could we please ask you to review the issue of the representation of "no
data" described below and give us your views on the alternatives we
propose? This affects users of GRASS 4.* considering moving to GRASS
5.*, and users of GRASS 5.0 beta who depend on using legacy data. This
is your chance to influence a major decision which may impact your use
of GRASS!

Recently on the developers' mailing list, there has been a discussion
concerning the current representation of "no data" in raster map layers.
When working with categories/classes, it was convenient to use integer
zero (0) to express "no data", because the classes each had non-zero
numbers - like land-use classes. This is the position in GRASS up to
4.*. GRASS 5.0 beta integrated experimental work done over many years by
people needing both floating-point raster values, and needing to use
numeric zero as a number.

This meant that representing "no data" raised a conflict of interest:
people with legacy data didn't want their "no data" (0) to "come back to
life", while the numerical people really needed zero, as in zero slope,
to have a meaning other than "no data". The term introduced to cover "no
data" is "NULL", which can include both "no data available", and other
situations where no numeric representation exists, like log of a
negative number, or dividing by zero.

There were then basically 2 groups of users in the mid 1990's:

1. Those who worked mostly with classes and had a large number of
   files with 0 set as "no data"

2. Those who started to use raster maps as representation of
   continuous fields and 0 was a data value

The majority of the users were in group 1 and thus, the decision was
made to keep the feature that 0 could be interpreted as "no data".

In the discussion on the developers' list, some developers have
indicated that, since GRASS 5.0 implements the concept of NULL (which is
represented by machine dependent constants), it is confusing to also
have the feature that 0 can be interpreted as NULL, especially to new
users of GRASS. NULL is implemented as a separate mask of bits
representing "no data", in principle for each raster map layer, but not
all modules generate them automatically, meaning that users not knowing
the history get mixed up. Please note that incidents caused by such
confusion have been reported.

In order to eliminate this confusion, a proposal was made that 0 should
never be considered as "no data" in GRASS 5.0. Also, in order to
facilitate those users who wish to use their GRASS 4.x files which use 0
as "no data" with the new GRASS 5.0, we would provide a script to assist
them in converting the 0 values of their data to appropriate NULL values
in GRASS 5.0. Note that GRASS 5.0 can convert any value to NULL and vice
versa. Please see details of the proposed script at the following site

http://www.geog.uni-hannover.de/grass/nullConvert.html 

What we would like you to do is provide us with your opinion on whether
we should keep the feature of 0 being interpreted as "no data". To make
things a little easier, we would like you to reply to this post stating
which of the following options you wish to support:

    a) Full legacy interoperability: keep the feature of 0 being
       optionally interpreted as "no data"

    b) Transition from GRASS 5.0 onwards: keep the feature of 0 being
       optionally interpreted as "no data" in GRASS 5.0 only, 
       eliminating it in subsequent releases

    c) Transition now: eliminate the feature from GRASS 5.0, but 
       provide a script to convert GRASS 4.x files to GRASS 5.0 format
       for those who wish 0 to be NULL

Thank you for your time in this matter.

------------------------------------------------------------

-- 
Sincerely,

Jazzman (a.k.a. Justin Hickey)  e-mail: jhickey at hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand
==================================================================
People who think they know everything are very irritating to those
of us who do.  ---Anonymous

Jazz and Trek Rule!!!
==================================================================