[GRASS5] 0 != no data

Tue Jun 12 12:36:36 EDT 2001

On Tue, Jun 12, 2001 at 07:17:07AM -0500, Helena wrote:
> Roger explained the issue correctly - let me just add a little bit more.
> When NULL was being implemented there were 2 groups of users:
> 1. Those who worked mostly with classes and had large number of files
>    with 0 set as NULL - that was the standard when I came to CERL.
> 2. Those who started to use raster maps as representation of continuous
>   fields and 0 was a data value. Both lack of FP support and NULL was a big
>   problem.
> 
> It was clear that one of those 2 groups will have to update their databases
> when NULL is introduced and because the first one was overhelming majority,
> we ended up with what we have now.
> As I said, I don't like it because I am the user in the no. 2 group, but I
> strongly suggest to send a message to the users group to see what the
> reaction would be.

> There are people who have 10 years of GRASS4.* data accumulated in their
> databases and my experience is that even those old data are quite often
> used. The removal of the NULL file may require a lot of work and it would
> be good to evaluate how does it compare with other priorities.
> 
> The issue with LZW compression was different because there weren't very many
> people who had lots of floating point data because the FP GRASS was never
> officially released. So it was relatively easy and fair to ask them to
> update.
> 
> So Markus, let us know what do you think and if you believe that it is worth
> considering such a change please write a message to the users list to see
> what the reaction would be,

If you ask me: I agree with Justin that the additional NULL file is somewhat
confusing and obviously not necessary any more.

In my opinion major changes should be done for 5.0 and not for 5.1 as there
are many 4.x users still waiting for the stable release. When published,
they will switch over to 5.0. To have much extra work again in 5.1 to
convert raster data will not be very satisfying.

At time I am not clear about the percentage of GRASS users working already
with 5.x. Can't we implement a sort of autodetection which checkes the data
at GRASS startup and going to modify the NULL stuff if required? A sort of
automated raster data upgrade (similar to r.lzw2z, but fully automated).

Even if it delays the 5.0 again (luckily pre1 is out), the raster part
should be stable for 5.0 as we have a new vector management for 5.1. GRASS
5.0 is announced to provide a new raster engine, so it should be completed.

Concerning the 4.x existing data: I have already put a 4.x and a 5.x version
online of SPEARFISH and friends. It was a simple run of r.support (-r now).
I didn't dig too deeply into the NULL problems yet, but it doesn't look
unfeasible to me as it should be mostly a library issue.

Generally we should minimize the *number* of transitions due to the large
number of datasets around. I would prefer a transition more time consuming
for the individual conversion rather than invoking new transitions when
upgrading to 5.1 (if I come from 4.x). Multiply the number of transitions
with number of datasets which doesn't equals user's happiness...

Shall I start a poll on number of 4.x/5.x users? There are some free poll
sites around.

We should't carry GRASS history around for all versions (backward
compatibility). I actually don't see any reason to fall back to 4.x,
hope that most 5.0.0pre1 users agree.
Think of the Windows/DOS story... :-)

Just my 0.02 Euro,

 Markus

> Roger Bivand wrote:
> 
> > Hi Justin
> >
> > On Tue, 12 Jun 2001, Justin Hickey wrote:
> >
> > > Roger Bivand wrote:
> > > >
> > > > The chosen NULL
> > > > solution was dictated by the fact that all pre-NULL programs (many
> > > > really for categorical rather than numeric layers) used 0 as 0 and
> > > > NULL (see traces in r.slope.aspect adding 1 to zero slopes etc. and
> > > > now commented out. Moving straight to the representation you describe
> > > > was thought to raise too large transition problems for the many
> > > > legacy users, so their needs had to be incorporated. I don't know how
> > > > the user base splits now between 4.* and 5.* releases, but guess that
> > > > 4.* still has a loyal following. Maybe first we should get them over
> > > > to 5.0 using the existing transitional mechanisms, and only next time
> > > > round migrate them away from 0 as NULL (a bit like lzw2z)?
> > >
> > > I tend to disagree on this one. In your proposal, the users will have to
> > > go through two transitions: one for using new floating point and NULL
> > > code, and then a second one for removing the 0 = NULL feature. To me,
> > > making a single transition is better in this case. Also, I don't think
> > > the transition problem in this case will be large. The 4.x users are
> > > used to 0 being NULL. All we do is inform them that they need to run
> > > r.mapcalc (or we can provide a script for them) to update their data to
> > > GRASS 5 if they want to convert their 0 values to NULL values, and that
> > > from now on 0 is a valid value. They would probably expect this anyway
> > > since they know the data format has changed.
> > >
> > > In order to implement this, we simply hide the fact that a null file is
> > > required. As far as I can tell, most import programs generate the null
> > > file. The only case we need to worry about is when users copy 4.x data
> > > to 5 databases. In this case, we tell them to run our script to upgrade
> > > and we run r.mapcalc and generate the null file for them.
> > >
> > > The only remaining case I can think of is if a user wants 0 to represent
> > > NULL for newly created maps. In this caes I think we should say "Sorry,
> > > but Grass 5 requires 0 to be a valid data value". Although backward
> > > compatibility is good in most cases, sometimes it is bad, and I believe
> > > that this is one of those cases. I think it is pretty safe to say that
> > > most users (if not all) consider "0 as an invalid data value" a bug.
> > > Afterall, that's the reason the NULL value concept was introduced, to
> > > fix this bug.
> > >
> > > So now we get users accustomed to upgrading their data with our upgrade
> > > script, and when we eliminate the null file for 5.1, there is no
> > > transition for them. I think they would be happier this way because they
> > > are expecting a big change in their operation methods this time round.
> > > But they probably will not expect a big change in their operations for
> > > 5.1.
> > >
> > That seems well judged. My explanation was really about the history -
> > essentially the same points that Helena made earlier in this thread. I
> > still feel that 5.0 should be as open as possible to people using 4.*, but
> > agree that from 5.1 the solutions chosen should be as clean and
> > maintainable as possible. While the pipe-sockets transition doesn't affect
> > legacy data, NULL might, so your script solution would work well. I know
> > that the legacy NW Leicestershire data set is 4.* style - I suppose
> > updating the demonstration data sets would help too?
> >
> > Roger
> >
> > --
> > Roger Bivand
> > Economic Geography Section, Department of Economics, Norwegian School of
> > Economics and Business Administration, Breiviksveien 40, N-5045 Bergen,
> > Norway. voice: +47 55 95 93 55; fax +47 55 95 93 93
> > e-mail: Roger.Bivand at nhh.no
> > and: Department of Geography and Regional Development, University of
> > Gdansk, al. Mar. J. Pilsudskiego 46, PL-81 378 Gdynia, Poland.
> >
> > _______________________________________________
> > grass5 mailing list
> > grass5 at geog.uni-hannover.de
> > http://www.geog.uni-hannover.de/mailman/listinfo/grass5
> 
> _______________________________________________
> grass5 mailing list
> grass5 at geog.uni-hannover.de
> http://www.geog.uni-hannover.de/mailman/listinfo/grass5

-- 
Markus Neteler *  University of Hannover
Institute of Physical Geography and Landscape Ecology
Schneiderberg 50 * D-30167 Hannover * Germany
Tel: ++49-(0)511-762-4494  Fax: -3984