[GRASS5] 0 != no data

[Markus Neteler]

Hi all,

On Wed, Jun 13, 2001 at 03:55:24PM +0700, Justin Hickey wrote:
> Hello all
> 
> Markus Neteler wrote:
> > 
> > On Tue, Jun 12, 2001 at 07:17:07AM -0500, Helena wrote:
> > > Roger explained the issue correctly - let me just add a little bit 
> > > more. When NULL was being implemented there were 2 groups of users:
> > > 1. Those who worked mostly with classes and had large number of 
> > >    files with 0 set as NULL - that was the standard when I came to
> > >    CERL.
> > > 2. Those who started to use raster maps as representation of
> > >    continuous fields and 0 was a data value. Both lack of FP
> > >    support and NULL was a big problem.
> > >
> > > It was clear that one of those 2 groups will have to update their 
> > > databases when NULL is introduced and because the first one was
> > > overhelming majority, we ended up with what we have now.
> > > As I said, I don't like it because I am the user in the no. 2
> > > group, but I strongly suggest to send a message to the users group
> > > to see what the reaction would be.
> 
> Yes, it was my intention to question the users list before doing
> anything.
> 
> > > There are people who have 10 years of GRASS4.* data accumulated in 
> > > their databases and my experience is that even those old data are
> > > quite often used. The removal of the NULL file may require a lot of
> > > work and it would be good to evaluate how does it compare with
> > > other priorities.
> 
> The actual major changes would be done after 5.0 is released. I only
> suggest users switch now when they expect a major change.
> 
> > > The issue with LZW compression was different because there weren't
> > > very many people who had lots of floating point data because the FP
> > > GRASS was never officially released. So it was relatively easy and
> > > fair to ask them to update.
> > >
> > > So Markus, let us know what do you think and if you believe that it 
> > > is worth considering such a change please write a message to the
> > > users list to see what the reaction would be,
> > 
> > If you ask me: I agree with Justin that the additional NULL file is 
> > somewhat confusing and obviously not necessary any more.
> > 
> > In my opinion major changes should be done for 5.0 and not for 5.1 as 
> > there are many 4.x users still waiting for the stable release. When
> > published, they will switch over to 5.0. To have much extra work
> > again in 5.1 to convert raster data will not be very satisfying.
> > 
> > At time I am not clear about the percentage of GRASS users working
> > already with 5.x. Can't we implement a sort of autodetection which
> > checkes the data at GRASS startup and going to modify the NULL stuff
> > if required? A sort of automated raster data upgrade (similar to
> > r.lzw2z, but fully automated).
> 
> No, we cannot automate the conversion. The deciding factor in whether to
> convert a given file or not is user preference (do they want 0 to be
> interpreted as NULL). If we automatically convert 4.x files so that 0 is
> turned to NULL, then users who want 0 = 0 will have to convert back
> again. However, I believe I can write a script that, given a list of
> databases, can go through a user's data and perform the conversion. I
> haven't tested it, but I have an idea of how to do it. This of course
> would depend on whether users have maintained their data properly. That
> is, all directories under a database are locations and all directories
> under a location are mapsets. 
> 
> > Even if it delays the 5.0 again (luckily pre1 is out), the raster 
> > part should be stable for 5.0 as we have a new vector management for
> > 5.1. GRASS 5.0 is announced to provide a new raster engine, so it
> > should be completed.
> > 
> > Concerning the 4.x existing data: I have already put a 4.x and a 5.x
> > version online of SPEARFISH and friends. It was a simple run of
> > r.support (-r now). I didn't dig too deeply into the NULL problems
> > yet, but it doesn't look unfeasible to me as it should be mostly a
> > library issue.
> > 
> > Generally we should minimize the *number* of transitions due to the 
> > large number of datasets around. I would prefer a transition more
> > time consuming for the individual conversion rather than invoking new
> > transitions when upgrading to 5.1 (if I come from 4.x). Multiply the
> > number of transitions with number of datasets which doesn't equals
> > user's happiness...
> > 
> > Shall I start a poll on number of 4.x/5.x users? There are some free 
> > poll sites around.
> 
> Yes, start a poll but not on the number of grass 4 and 5 users. The
> question should be related to whether we keep the feature or not. See
> below for more on this.
>  
> > We should't carry GRASS history around for all versions (backward
> > compatibility). I actually don't see any reason to fall back to 4.x,
> > hope that most 5.0.0pre1 users agree.
> > Think of the Windows/DOS story... :-)
> 
> Since I will be posting this issue to the user list, I would like to
> suggest the wording for the post here and have all concerned developers
> change, or add to it, to make sure I don't leave anything out. Helena, I
> hope you don't mind me borrowing part of your post.
> 
> ======================== Start of post ===============================
> 
> Subject: We need your opinion
> 
> Hello all users
> 
> Recently on the developers mailing list, there has been a discussion
> concerning the current implementation of NULL. When the NULL concept was
> first introduced and implemented there were basically 2 groups of users:
> 
> 1. Those who worked mostly with classes and had a large number of 
>    files with 0 set as NULL
> 
> 2. Those who started to use raster maps as representation of
>    continuous fields and 0 was a data value
> 
> The majority of the users were in group 1 and thus, the decision was
> made to keep the feature that 0 could be interpreted as NULL.
> 
> In the discussion on the developers list, some developers have indicated
> that since GRASS 5.0 has the concept of NULL (which is represented by
> non-zero values), it is confusing to also have the feature that 0 can be
> interpreted as NULL, especially to new users of GRASS. This feature has
> also caused some problems for some users who had greyscale aerial
> photographs stored as raster maps in GRASS 4.x, and then transferred
> them to GRASS 5.0. The problem was that the color black (value 0) was
> being interpreted as NULL causing them to believe that their data had
> been corrupted somehow.
> 
> In order to eliminate this confusion, a proposal was put forth on the
> list that 0 should never be considered as NULL in GRASS 5.0. Also, in
> order to facilitate those users who wish to use their GRASS 4.x files
> with the new GRASS 5.0, we would provide a script to assist them in
> converting the 0 values of their data to appropriate NULL values in
> GRASS 5.0.
> 
> This script has not been written yet but the proposed usage would be
> that a user would run the script and provide a list of GRASS 4.x
> databases which contain raster files in which 0 should be interpreted as
> NULL. The script will automatically go through all locations and mapsets
> in each database and perform the conversion for all raster files. The
> only requirement for this script would be that the GRASS 4.x data would
> have to be maintained in proper GRASS databases. That is, all
> directories under a database will be locations, and all directories
> under a location will be mapsets. If a user has files that he/she does
> not want converted then either the files can be moved to a different
> database, or the user can provide a specific location, or mapset, or
> raster file to the script.
> 
> What we would like you to do is provide us with your opinion on whether
> we should keep the feature of 0 being interpreted as NULL. To make
> things a little easier, we have set up a poll at
> 
> <URL>
> 
> for you to submit your opinion. We would greatly appreciate it if you
> could take a few moments to cast a vote.
> 
> Thank you for your time in this matter.
> 
> ========================= End of post ================================
> 
> So, please feel free to edit the above message in any way. For the poll,
> perhaps we could say something like.
> 
> The feature of 0 being interpreted as NULL should be
> 
>     a) kept in GRASS 5.0
>     b) eliminated from GRASS 5.0, but a script will be provided to
>        convert GRASS 4.x files to GRASS 5.0 format for those who wish
>        0 to be NULL
> 
> Again, please feel free to edit this statement.
> 

thanks for your nice proposal. A few remarks:

- Perhaps (even for the new users), an example may be added to demonstrate
  the 0 to be NULL thing 
- it should be pointed out, that GRASS 5 will keep the feature to store
  NULLs (for those not carefully ready), perhaps in a bottom summary
- it should be pointed out, that GRASS 5 will keep the feature to convert
  individual values to NULL and back
- maybe in a bottom summary it should be explained, that, if eliminating the
  feature of 0 to be NULL, that both groups (1) and (2) will stay happy,
  only that a conversion step is required (or not?).

BTW: Does anyone know a free poll server without adult images or annoying
ads? I don't want that they catch all the users' email adresses and sell
them anywhere.

Regards

 Markus