[osgeo4w-dev] Tech Review and Experiments

[Frank Warmerdam]

Folks,

Yesterday I spent most of the day looking into approaches to OSGeo4W.   I
started out with a review of NSIS, especially with regard to supporting
fine grained packages with dependencies and with installing packages from the
network.  I came to the following conclusions.

1) I couldn't find any obvious way to express detailed dependencies between
components though others tell me there is a way.  In any event, I'm concerned
it will be hard to express complex dependencies (like python-mapscript depends
on having python and mapserver packages).

2) There are mechanisms to download files from the internet and if this was
a zip file for instance, it could be unpacked as a form of installing.  But
this seems to lose the benefits of NSIS tracking the file list for uninstall
purposes.

3) I can't see any obvious way to run an NSIS based installer to add additional
packages to an existing install or to uninstall individual components.

Based on the above, and the degree of frustration I've had in past doing
anything moderately complicated with NSIS I just couldn't see how to go
forward with it.  So, at that point I started casting around again looking
for a many-package-net-installer.

I reviewed the option of producing MSI files, possibly using something like
the WiX project.  This has the benefit of being very automatically installable
in a way that system administrators for large windows installations would like.
But I couldn't figure out how to handle this as a network installation if we
were to use even moderately fine grained packaging.

My understanding is that a single MSI file can have components which the
user can individually select.  But all the components must be embedded
in the MSI file.  If we took the approach that all of OSGeo4W was one MSI
file it would be (eventually) huge.

If we took the alternate approach that we produce many MSI files, one
per project or component, then we could "front end" things with an installer
build using something like NSIS to select the packages, and then fetch
and do a UI-less install of all the MSI files.  The biggest downside of
this approach in my mind is that (as far as I know) each MSI file ends
up being a package in the "Add or Remove Programs" area and I'm not sure
that removing it from there would truely understand the dependencies.  At
the very least, I think we would need to take a more coarse grained approach
if we wanted to do that.

I looked around for other open source net installer packages for a while
and really found nothing else that seemed any closer to what I wanted.

I did come to the conclusion that what I really want is what I have on
Linux with package managers.  Basically, I have a graphical front end that
makes it easy to find all available packages, and install them from the net.
The packages understand their depencencies, and installing a package pulls
in anything it depends on.  That's what I want for OSGeo4W.

Looking at it that way, an obvious candidate technology is the setup program
used for Cygwin.  It is package oriented and net-install based with good
understanding of dependencies.  *But* I didn't want to depend on the cygwin
"unix emulation" view of the world.  I want to be able to work with "real"
win32 based applications.

Pursuing this I looked at whether the Cygwin "setup" could be adapted to our
needs.  I fetched the source, modified it to talk about "OSGeo4W", and created
a setup.ini file from scratch myself and a few packages (zlib, libjpeg,
and libtiff).  This was successful.  What I like about this approach:

1) Net install works naturally.  Setup knows how to show progress though
a full install well, taking into account the network install aspect.

2) A reasonable GUI for selecting packages from a tree list.

3) The GUI can be rebranded for OSGeo and modified as needed.

4) I can very easily see how this approach will allow us to manage many
packages independently, and them become part of a coherent install.

Things I don't like about the Cygwin Setup approach:

1) If we use fine grained packaging, the number of packages may be
somewhat overwhelming for folks (as I find is the case with Cygwin).
I think the obvious approach to this using a "meta package" category as
our primary "easy install" interface.

2) The Cygwin user interface feels a bit old fashioned and clunky.
At the very least I think I would see if some steps (such as mirror
selection) can be streamlined out.

3) There doesn't seem to be an out of the box approach to doing batch
installs.  You basically need to do it through the gui interface.  I'd
be inclined to add something for this (like "setup.exe -install gdal-1.5.0"
doing a batch install of the named package and it's dependencies).

4) There is no built-in scripting language for "install actions".  For
Cygwin you can run a bash or .bat script for post-install actions,
but since we won't want a bash dependency we are basically down to
running .bat files.  It seems like some complex post-install actions
that can't be handled via .bat files would either need to be custom
.exe files or perhaps C code built into the setup program (as is
done for the desktop icon generation).   We had previously
contemplated always installing a scripting language (such as PHP)
as part of a base install.  This would be a plausible solution.

So, my question to the group is what technological approach do we want to
take.  If there are folks advocating an NSIS approach, can you prepare a
prototype that demonstrates net installation and good dependency understanding?

A related question is whether we want to do "fine grained" packages like you
find in Cygwin, and the various linux packaging systems.  For instance,
this might mean two packages for libjpeg - one for the runtime DLL, and one
for the developer files.  Or do we want to use a more coarse grained approach.
For instance we might just bundle stuff like libjpeg, and libtiff into a
single GDAL package, and just treat developer stuff as part of the standard
packages.

Benefits of course grained:
  o Less fudzing around for users doing installs, trying to find what they
    want out of a long list of packages.
  o Less packages to maintain.

Benefits of fine grained:
  o Easier to upgrade individual components.  For instance, upgrading libjpeg
    doesn't require having to be able to build the whole GDAL project.
  o Reduces the amount of unnecessary stuff getting installed.  Select just
    what you need.
  o Keeps things factored out more naturally so we don't fall into the
    situation of having to add a depencency on GDAL (for instance) when
    another package only really needs libjpeg.

I personally am leaning somewhat towards fine grained, but a large part of
that is the influence of the linux (and cygwin) packaging communities who
have already gone that way.  I'm not sure I understand all the implications
of this decision, but I feel more comfortable approach it in a well explored
and successfully used manner.

I will continue a bit further with my "cygwin setup" experiment, in an effort
to at least get GDAL and it's dependencies packaged as a demonstration and then
upload it for folks to see.  After that I'll hold off till others have a chance
to make a case for other approaches (and hopefully demonstrate them!).

I would note that the actual low level binary packages produced for Cygwin can
be fairly easily repackaged for other purposes.  Basically for each package
I upload a .tar.bz file and a text file expressing it's dependencies on other
packages, version info and such.  It should be relatively easy to construct
MSI files, or NSIS packages from the same binaries using some scripting.  So,
if we can decide on fine-grained vs. coarse-grained we can start working on
packages and work out the detailed of the actual setup technology over a bit
more time - or at least without undue fear of wasting all our work so far.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org