[GRASS-dev] make on SMP [was: GRASS 6.3.0 release preparation]

[Glynn Clements]

Maciej Sieczka wrote:

> Glynn Clements wrote:
> > Maciej Sieczka wrote:
> > 
> >>>> sorry but I'm not an make guru and thus see it simple: if I run comand
> >>>> "make -j4" it should get along with it or fail with "err blah parallel
> >>>> buld not supported". Current CVS version does neither of it.
> >>> I hope we don't give up so early. Probably just a few Makefiles
> >>> need to be fixed (basically, get the target order right therein).
> >>>> As multicore CPU's are getting more popular, more and more users will be
> >>>> trying to run "make -j X" and thus report failure as bug [1].
> >>> Not always. As written yesterday, "make -j3" worked for me on my RHEL5
> >>> box. Now testing with Mandriva.
> >>>> If I can help with testing, drop me instructions. I have Gentoo ~x86
> >>>> system (gcc version 4.2.1; GNU Make 3.81).
> >>> Please help testing. Just run it and see if and where it fails.
> >>> Then check the problematic Makefile(s) for correct order:
> >>> http://grass.gdf-hannover.de/wiki/Development#GRASS_Makefile_writing
> >> As for testing: I get the following errors with make -j3 on
> >> Intel Core 2 6600 running 2.6.15-29-amd64-xeon (Ubuntu Dapper):
> >>
> >> --
> >> Errors in:
> >> /home/maciek/src/straight/grass63/lib/gpde
> >> /home/maciek/src/straight/grass63/imagery/i.vpoints
> >> /home/maciek/src/straight/grass63/misc/m.cogo
> >> /home/maciek/src/straight/grass63/raster/r.gwflow
> >> /home/maciek/src/straight/grass63/raster/r.to.rast3elev
> >> /home/maciek/src/straight/grass63/raster3d/r3.gwflow
> >> /home/maciek/src/straight/grass63/vector/v.sample
> > 
> > What errors, exactly?
> 
> That's all it says:
> 
> Started compilation: Thu Oct 18 00:07:37 CEST 2007
> --
> Errors in:
> /home/maciek/src/straight/grass63/lib/gpde
> /home/maciek/src/straight/grass63/misc/m.cogo
> /home/maciek/src/straight/grass63/raster/r.gwflow
> /home/maciek/src/straight/grass63/raster3d/r3.gwflow
> --
> In case of errors please change into the directory with
> error and run 'make'. If you get multiple errors, you need
> to deal with them in the order they appear in the error log.
> If you get an error building a library, you will also get
> errors from anything which uses the library.
> --
> Finished compilation: Thu Oct 18 00:11:12 CEST 2007
> make: *** [default] Error 1
> 
> If you'd like to see full make and configure output they are
> here [1].

The relevant make output is necessary.

lib/gpde:
	N_arrays.c:19:25: error: grass/N_pde.h: No such file or directory

I've just figured out what's going on; it's the way that phony targets
are being used. E.g. lib/gpde/Makefile has:

	default: headers lib
	
	headers:
		for file in ./N_*.h ; do $(INSTALL_DATA) $$file $(GISBASE)/include/grass/ ; done
	
	lib: headers

"headers" is a prerequisite of "lib", and so will be built before it. 
But "lib" is defined in Lib.make as:

	lib: $(GRASS_LIBRARY_TYPE)

$(GRASS_LIBRARY_TYPE) is either "shlib" or "stlib". Assuming "shlib",
that's defined in Shlib.make as:

	shlib: $(SHLIB)

When there are multiple rules for a given target, all of the
dependencies are merged, so in effect we have:

	lib: headers $(SHLIB)

IOW, the headers target and the actual shared library are built in
parallel.

Using phony targets for real files (as opposed to e.g. "clean") sucks.

What we really want is a dependency like:

	$(SHLIB): headers

But it isn't clear how to abstract out the library type. Note that:

	$(GRASS_LIBRARY_TYPE): headers

won't work, as that just gives e.g.:

	shlib: headers
	
	shlib: $(SHLIB)

which is equivalent to:

	shlib: headers $(SHLIB)

which will still result in headers and $(SHLIB) being built in
parallel.

Note to self (and anyone else who it concerns): a dependency such as
"foo: bar" does *not* make "foo" into an alias for "bar". Sure, making
foo will make bar, but if foo has other prerequisites, they will be
made in parallel with bar, not before it.

Ugh. I don't immediately know how to solve this, but I do at least
know what's going on now. I'm fairly sure that that this is the cause
of many of the outstanding problems with parallel builds.

-- 
Glynn Clements <glynn at gclements.plus.com>