[GRASS5] 0 != no data

[Justin Hickey]

Hello all

Markus Neteler wrote:
> 
> On Tue, Jun 12, 2001 at 07:17:07AM -0500, Helena wrote:
> > Roger explained the issue correctly - let me just add a little bit 
> > more. When NULL was being implemented there were 2 groups of users:
> > 1. Those who worked mostly with classes and had large number of 
> >    files with 0 set as NULL - that was the standard when I came to
> >    CERL.
> > 2. Those who started to use raster maps as representation of
> >    continuous fields and 0 was a data value. Both lack of FP
> >    support and NULL was a big problem.
> >
> > It was clear that one of those 2 groups will have to update their 
> > databases when NULL is introduced and because the first one was
> > overhelming majority, we ended up with what we have now.
> > As I said, I don't like it because I am the user in the no. 2
> > group, but I strongly suggest to send a message to the users group
> > to see what the reaction would be.

Yes, it was my intention to question the users list before doing
anything.

> > There are people who have 10 years of GRASS4.* data accumulated in 
> > their databases and my experience is that even those old data are
> > quite often used. The removal of the NULL file may require a lot of
> > work and it would be good to evaluate how does it compare with
> > other priorities.

The actual major changes would be done after 5.0 is released. I only
suggest users switch now when they expect a major change.

> > The issue with LZW compression was different because there weren't
> > very many people who had lots of floating point data because the FP
> > GRASS was never officially released. So it was relatively easy and
> > fair to ask them to update.
> >
> > So Markus, let us know what do you think and if you believe that it 
> > is worth considering such a change please write a message to the
> > users list to see what the reaction would be,
> 
> If you ask me: I agree with Justin that the additional NULL file is 
> somewhat confusing and obviously not necessary any more.
> 
> In my opinion major changes should be done for 5.0 and not for 5.1 as 
> there are many 4.x users still waiting for the stable release. When
> published, they will switch over to 5.0. To have much extra work
> again in 5.1 to convert raster data will not be very satisfying.
> 
> At time I am not clear about the percentage of GRASS users working
> already with 5.x. Can't we implement a sort of autodetection which
> checkes the data at GRASS startup and going to modify the NULL stuff
> if required? A sort of automated raster data upgrade (similar to
> r.lzw2z, but fully automated).

No, we cannot automate the conversion. The deciding factor in whether to
convert a given file or not is user preference (do they want 0 to be
interpreted as NULL). If we automatically convert 4.x files so that 0 is
turned to NULL, then users who want 0 = 0 will have to convert back
again. However, I believe I can write a script that, given a list of
databases, can go through a user's data and perform the conversion. I
haven't tested it, but I have an idea of how to do it. This of course
would depend on whether users have maintained their data properly. That
is, all directories under a database are locations and all directories
under a location are mapsets. 

> Even if it delays the 5.0 again (luckily pre1 is out), the raster 
> part should be stable for 5.0 as we have a new vector management for
> 5.1. GRASS 5.0 is announced to provide a new raster engine, so it
> should be completed.
> 
> Concerning the 4.x existing data: I have already put a 4.x and a 5.x
> version online of SPEARFISH and friends. It was a simple run of
> r.support (-r now). I didn't dig too deeply into the NULL problems
> yet, but it doesn't look unfeasible to me as it should be mostly a
> library issue.
> 
> Generally we should minimize the *number* of transitions due to the 
> large number of datasets around. I would prefer a transition more
> time consuming for the individual conversion rather than invoking new
> transitions when upgrading to 5.1 (if I come from 4.x). Multiply the
> number of transitions with number of datasets which doesn't equals
> user's happiness...
> 
> Shall I start a poll on number of 4.x/5.x users? There are some free 
> poll sites around.

Yes, start a poll but not on the number of grass 4 and 5 users. The
question should be related to whether we keep the feature or not. See
below for more on this.
 
> We should't carry GRASS history around for all versions (backward
> compatibility). I actually don't see any reason to fall back to 4.x,
> hope that most 5.0.0pre1 users agree.
> Think of the Windows/DOS story... :-)

Since I will be posting this issue to the user list, I would like to
suggest the wording for the post here and have all concerned developers
change, or add to it, to make sure I don't leave anything out. Helena, I
hope you don't mind me borrowing part of your post.

======================== Start of post ===============================

Subject: We need your opinion

Hello all users

Recently on the developers mailing list, there has been a discussion
concerning the current implementation of NULL. When the NULL concept was
first introduced and implemented there were basically 2 groups of users:

1. Those who worked mostly with classes and had a large number of 
   files with 0 set as NULL

2. Those who started to use raster maps as representation of
   continuous fields and 0 was a data value

The majority of the users were in group 1 and thus, the decision was
made to keep the feature that 0 could be interpreted as NULL.

In the discussion on the developers list, some developers have indicated
that since GRASS 5.0 has the concept of NULL (which is represented by
non-zero values), it is confusing to also have the feature that 0 can be
interpreted as NULL, especially to new users of GRASS. This feature has
also caused some problems for some users who had greyscale aerial
photographs stored as raster maps in GRASS 4.x, and then transferred
them to GRASS 5.0. The problem was that the color black (value 0) was
being interpreted as NULL causing them to believe that their data had
been corrupted somehow.

In order to eliminate this confusion, a proposal was put forth on the
list that 0 should never be considered as NULL in GRASS 5.0. Also, in
order to facilitate those users who wish to use their GRASS 4.x files
with the new GRASS 5.0, we would provide a script to assist them in
converting the 0 values of their data to appropriate NULL values in
GRASS 5.0.

This script has not been written yet but the proposed usage would be
that a user would run the script and provide a list of GRASS 4.x
databases which contain raster files in which 0 should be interpreted as
NULL. The script will automatically go through all locations and mapsets
in each database and perform the conversion for all raster files. The
only requirement for this script would be that the GRASS 4.x data would
have to be maintained in proper GRASS databases. That is, all
directories under a database will be locations, and all directories
under a location will be mapsets. If a user has files that he/she does
not want converted then either the files can be moved to a different
database, or the user can provide a specific location, or mapset, or
raster file to the script.

What we would like you to do is provide us with your opinion on whether
we should keep the feature of 0 being interpreted as NULL. To make
things a little easier, we have set up a poll at

<URL>

for you to submit your opinion. We would greatly appreciate it if you
could take a few moments to cast a vote.

Thank you for your time in this matter.

========================= End of post ================================

So, please feel free to edit the above message in any way. For the poll,
perhaps we could say something like.

The feature of 0 being interpreted as NULL should be

    a) kept in GRASS 5.0
    b) eliminated from GRASS 5.0, but a script will be provided to
       convert GRASS 4.x files to GRASS 5.0 format for those who wish
       0 to be NULL

Again, please feel free to edit this statement.

Thanks for listening.

-- 
Sincerely,

Jazzman (a.k.a. Justin Hickey)  e-mail: jhickey at hpcc.nectec.or.th
High Performance Computing Center
National Electronics and Computer Technology Center (NECTEC)
Bangkok, Thailand
==================================================================
People who think they know everything are very irritating to those
of us who do.  ---Anonymous

Jazz and Trek Rule!!!
==================================================================