[SAC] Backups

[Howard Butler]

I would like to start a thread where folks who are primarily  
responsible for some piece of our infrastructure describe the current  
state of the backup situation of that component.  It would be  
instructive to find out where we are with things, and it would help  
illuminate work items that we need to complete.  I think the following  
components need to be described, and I'll start with subversion:

- subversion
- Trac
- wiki
- drupal
- buildbot
- download.osgeo.org
- upload.osgeo.org
- apache configuration
- LDAP
- mailing lists
...

Subversion
---------------------

Subversion's primary backup is an offsite, svnsync server at http://svnmirror.osgeo.org 
.  As new repositories are added to osgeo1, new mirrors are also  
manually created by Howard at svnmirror.osgeo.org.  svnmirror is  
housed on a server at TelaScience.  Where possible, the GUID of the  
repositories that we manage on osgeo1 are the same on the mirror.   
Some repositories that are not housed at osgeo1 are also in the  
mirror, such as TileCache, FeatureServer, and OpenLayers, as a  
courtesy to these projects.

An incremental dump of subversion is also available for some  
repositories at http://svn.osgeo.org/dumps, and this directory is wget  
--mirror'd to a machine at TelaScience.  This approach was done before  
the svnsync stuff was in wide use.  It is not known if recovery from  
these incremental dumps is practically possible, but it is  
theoretically feasible.  The standard svn mirror makes the issue moot,  
however.

If a catastrophic failure were to happen today, how screwed would we be?
------------------------------------------------------------------------------------------------------------

The mirror does a sync at 19 minutes past the hour, every hour.  Worst  
case, our data would be one hour old, and individuals would also have  
their local copies of various trees that they could use to commit  
missing revisions that slipped away in that hour.

A major component of subversion is also the Apache configuration and  
the hook scripts.  At this time, it is not known if/how the hook  
scripts are backed up.  The Apache configuration is backed up into a  
subversion repository that is locally hosted on osgeo1, but that repo  
is not mirrored onto the svnmirror.osgeo.org mirror because of its  
sensitive nature.

Action Items
............................

- Determine how the hook scripts are backed up and ensure that they  
and their configuration for each repository is preserved.
- Determine how to better offsite the Apache configuration

Recovery procedure from a catastrophic event
-------------------------------------------------------------

- Notify Howard to log into svnmirror.osgeo.org and create dumps of  
all repositories
- Import those dumps into the new home
- Re-enable Apache configuration from backup
- Re-enable hook scripts from backup