[Live-demo] More ideas on collaborative development

Thu Aug 27 00:26:10 EDT 2009

Hamish, this is excellent advice.

I hope we can incorporate the vast majority of it into our build process.
The challenge is to see how much we can get in place before FOSS4G and 
in particular, before the press release going out tomorrow (Fri 28), 
asking project owners to write scripts for the GISVM, and pointing them at:
http://wiki.osgeo.org/wiki/GISVM_Build

Feel free to update scripts and wiki with what you think is appropriate.

Hamish wrote:
> Cameron wrote:
>   
>> I've been thinking hard about recent comments
>>     
>
> As mentioned in my last post, I've been trying to hold back on this until after the current image is complete. Folks seem well on their way and I fear to disrupt that with shifting requirements. I think the important thing that could happen now to improve things for the future is to keep a careful notes about everything which was installed, any needed modifications to system config files, and a layout of what data was installed and where.
>
>
> the basic breakdown of lists that need to be made goes like:
>
> - Packages to install directly from the repositories
> - Packages which need to be made locally
> - 3rd part data to be copied onto the disk (be it in /home or elsewhere)
> - scripts that need to run to fetch &/or build either of the above two
> - Patches and files to apply over the top of a generic OS install
> - Files external to the live image but on the disk (MS Windows installers;
>    autorun.inf -> osgeo.ico + open Readme.html)
>
>
> apologies in advance about the rambling that follows/
>
>
> For a package listing the simplest way is to just dump the list to a file:
>
> dpkg --get-selections > installed_packages.txt
>
> later if you set up a new system you can flag all those packages to be automatically added using the opposite command:
>
> dpkg --set-selections < installed_packages.txt
>
> followed by "apt-get install" (?) or similar to actually fetch and install them.
>
> if you like you can cull down that list to only include the GIS packages we want to add, and that way it won't tie itself so much to a specific OS version as package names do change with time. Along those lines if you just ask for "gdal-bin" it will automatically fetch the version it needs, whereas if you ask for "libgdal1.5.0" you'll have to modify the list every time a new version comes out.
>
>   
>> I notice that Massimo is part way here, having written the
>> installation steps into the wiki at:
>> http://wiki.osgeo.org/wiki/GFOSS.it_Live_DVD_Install
>>     
>
> I would suggest we maintain somewhere in SVN a common list of packages, and then all pull from that as needed. e.g. fetch with:
>
> # checkout
> svn export 
> http://svn.osgeo.org/osgeo/livedvd/live-helper/trunk/config/chroot_local-packageslists/DebianGIS
>
>
>
>   
>> 1. Write install scripts for each application, and put these scripts
>> into subversion:
>>     
>
> On a wiki page we should make a list of applications that need special attention, and those that just need to have their package name "on the list".
>
> currently the list of packages was based on the DebianGIS list, and is as follows:
> http://trac.osgeo.org/osgeo/browser/livedvd/live-helper/trunk/config/chroot_local-packageslists/
>
>
>   
>> I suggest we create bash install scripts for each application we
>> want on the LiveDVD. These scripts will contain all the steps
>> necessary to install a project. Ie, we would have one script for
>> geoserver, one for mapserver, one for qgis, etc.
>>     
>
> [note: for the record I will point to the relevant live-helper
> destination dir for each of the following; don't take that as me being pushy about using that method, it's just a working example of what it can look like & to remind myself what they are :). for more see
> http://trac.osgeo.org/osgeo/browser/livedvd/live-helper/trunk/README.txt ]
>
>
> - For anything that already has an official Ubuntu or Debian package all you need is the name.
> http://trac.osgeo.org/osgeo/browser/livedvd/live-helper/trunk/config/chroot_local-packageslists/
>
>
> - For anything that is not in official Ubuntu/Debian but the project provides .deb packages (eg Qgis) all you need to do is add the name of the repository to the master /etc/sources.list file. Then just put the package name on the master "to be installed" list and hopefully it figures out the rest on its own.
>
>
> - For anything that does not have a useful prepackaged version available (either not the version we want to use or not for the version of the Distribution we choose) BUT does have .deb packages for other versions, in the source code you will typically find a debian/ directory containing the build-script instructions. Often it is simply a case of "svn checkout" followed by "debuild binary" and after much blinking of lights and whiring of motors a nice package.deb pops out the other end, custom built for your platform. This is easily written into a shell script.
> These .debs to be automatically installed go into:
> http://trac.osgeo.org/osgeo/browser/livedvd/live-helper/trunk/config/chroot_local-packages/
>
>
> - Anything files you want to go into $HOME can be collected for bulk insertion. In the following it assumes the username is "gis" so /home/gis/ is the home dir.
> http://trac.osgeo.org/osgeo/browser/livedvd/live-helper/trunk/config/chroot_local-includes/home/gis/
>
>
> - A class of scripts to run at build time (e.g. wget http://gis.org/data/geodata.tar.gz)
> I think also package download + compiling scripts would be included at this stage. Not sure if they would happen first then drop their .debs into the "custom .deb" folder, or if you'd have to run run 'dpkg -i' manually after creating them. 
> http://trac.osgeo.org/osgeo/browser/livedvd/live-helper/trunk/config/chroot_local-hooks
>
>
> - Patches to be applied to system (/etc config) files: for small changes keep them as "diff -u oldfile newfile" and put them into e.g. $LIVEDIR/config/chroot_local-patches/  OR  put a whole copy of the file into (e.g.) $LIVEDIR/config/chroot_local-includes/etc/configfile  anything in there will automatically overwrite anything previously installed.
>
>
> - Another set of scripts would fetch/install stuff to go on the final DVD which do not have much to do with the live boot system. this would be for example where you put the D:\osgeo4w\ installer + local copy of the osgeo4w repository. These go here:
> http://trac.osgeo.org/osgeo/browser/livedvd/live-helper/trunk/config/binary_local-includes/osgeo4w/
> IIUC any wget script to fetch those files would go into $LIVEDIR/config/binary_local-hooks/
>
>
> I'm not sure if that's exactly correct, but hopefully it gives you an idea how the different build stages and breakdown can be subdivided for a build-script based ISO. In so, it may be a nice logical model for keeping a manifest of a non-build script based Image.
>
>
>   
>> * Community engagement: It would be straight forward for projects
>> to write their own installation scripts.
>>     
>
> hmmm.. how to keep the barrier to entry low? Document the hell out of it I guess.
>
>   
>> * I expect these bash scripts would eventually become the basis of a
>> .deb file?
>>     
>
> It is very easy to make a meta-package (aka task) which simply depends on a number of packages you want installed. See the existing education-geography task from debian-edu listed at http://wiki.debian.org/DebianGis/PackageList
> All you do is e.g. "apt-get install education-geography" and all the packages listed there get pulled in.
>
> As for a .deb to build a live image, I am not quite sure of a way to make work, but FWIW the instructions here are not so onerous:
> http://trac.osgeo.org/osgeo/browser/livedvd/live-helper/trunk/README.txt
> (at least to anyone who would otherwise be comfortable dealing with raw .debs)
>
>
>   
>> 2. Xubuntu:
>> I suggest that we use Xubuntu as a basis for the GISVM instead of Ubuntu,
>> as it is specifically designed to run on a low powered environment, but
>> still runs all the Ubuntu packages.
>>     
>
> I have no experience with Xubuntu, but if it meets the dual goals of "user oriented" and "lightweight with room to spare", like it sounds that it does, then sure, great.
>
>
>   
>> 3. Wiki pages:
>> There are a few too many LiveDVD wiki pages, which makes it hard for a
>> newbie to find out what is current, and what is not.
>> http://wiki.osgeo.org/wiki/Category:Live-demo
>>     
>
> Perhaps part of the problem is that we are treating the Index page like a table of contents? Given a more hierarchical front/home page it could mostly solve the issue. 
>
> The Category:Live-demo list is really intended as a catch-all for any page related to the project; I don't think it is so bad to have lots of pages, as long as you don't swamp the use with cruft or equally promote obsolete pages with modern ones.
>
>   
>> I suggest that we consolidate down to a few pages, removing content from
>> some pages and just providing a pointer to more recent pages.
>>     
>
> I hate to throw away good info, and am not comfortable removing stuff which I am not intimately familiar with. A good start would be for original authors to take some responsibility here. And otherwise if you think something is obsolete remove main-page links to it & add a big '''Note''' at the top of that page.
>
>   
>> 4. Central VM, which we rsync from:
>> As mentioned earlier, I think that we should set up a central virtual
>> machine, probably on OSGeo hardware, which we can all rsync from in
>> order to collaboratively update the same virtual machine.
>>     
>
> I'll respond to that in another email, so as not to swamp cc:Frank with the rest of this stuff.
>
>
> well, this email is long enough so I'll end it here and respond to more recent posts in another.
>
>
> regards,
> Hamish
>
>
>
>       
>
>   

-- 
Cameron Shorter
Geospatial Systems Architect
Tel: +61 (0)2 8570 5050
Mob: +61 (0)419 142 254

Think Globally, Fix Locally
Geospatial Solutions enhanced with Open Standards and Open Source
http://www.lisasoft.com