[GeoNode-devel] GeoNode Backup and Restore: first draft

Thu Feb 25 07:22:58 PST 2016

Dear all,

please find here

*https://github.com/GeoNode/geonode/tree/backup_restore/scripts/backup-restore
<https://github.com/GeoNode/geonode/tree/backup_restore/scripts/backup-restore>*

a first implementation of the backup and restore scripts using JSON
fixtures.

The scripts allow to automatically store into a ZIP archive all the GeoNode
Objects as JSON files along with the static resources.

A restore command takes the zip file as input and restores everything. At
the end you will have an exact copy of the old GeoNode replicated into the
new instance independently from the datastore.

*NOTICE 1:* I'm still working on the GeoServer side. GeoServer will be
enriched with a plugin allowing an administrator to store on a zip file the
whole data and configuration and having the ability to restore it at any
time.

*NOTICE 2: *In order to make things working you'll need to apply also the
"patch" attached to the email to the GeoNode source code *OR* clone the
*backup_restore *repository

How the GeoNode scripts work:

1. Backup (on source instance)
----------------------------------------------

Just go to the geonode/scripts/backup-restore folder and run the command

    *python backup.py*

The command will output few lines on the console

geonode at geonode:~/geonode/scripts/backup-restore$ python backup.py
Dumping 'people' into 'people.json'.
Dumping 'account' into 'accounts.json'.
Dumping 'avatar.avatar' into 'avatars.json'.
Dumping 'base.license' into 'licenses.json'.
Dumping 'base.topiccategory' into 'topiccategories.json'.
Dumping 'base.region' into 'regions.json'.
Dumping 'base.resourcebase' into 'resourcebases.json'.
Dumping 'base.contactrole' into 'contactroles.json'.
Dumping 'base.link' into 'links.json'.
Dumping 'base.restrictioncodetype' into 'restrictioncodetypes.json'.
Dumping 'base.spatialrepresentationtype' into
'spatialrepresentationtypes.json'.
Dumping 'guardian.userobjectpermission' into 'useropermissions.json'.
Dumping 'guardian.groupobjectpermission' into 'groupopermissions.json'.
Dumping 'layers.uploadsession' into 'uploadsessions.json'.
Dumping 'layers.layer' into 'layers.json'.
Dumping 'layers.attribute' into 'attributes.json'.
Dumping 'maps.map' into 'maps.json'.
Dumping 'maps.maplayer' into 'maplayers.json'.
Dumping 'documents.document' into 'documents.json'.
Dumping 'taggit' into 'tags.json'.
Saved Media Files from '/home/geonode/geonode/geonode/uploaded'.
Saved Static Root from '/home/geonode/geonode/geonode/static_root'.
Saved Static Files from '/home/geonode/geonode/geonode/static'.
Saved Template Files from '/home/geonode/geonode/geonode/templates'.
Saved Locale Files from '/home/geonode/geonode/geonode/locale'.
Backup Finished. Archive generated 'backup/*2016-02-25_180915.zip*'.
None
geonode at geonode:~/geonode/scripts/backup-restore$

and will store a timestamped ZIP file into the "backup" subfolder.

Notice that the scripts gets the locations and settings automatically from
the "geonode.settings" file.

2. Restore (on destination instance)
----------------------------------------------------

*WARNING: **This procedure will completely purge and overwrite your
destination GeoNode instance. BE CAREFUL.*

Go to the geonode/scripts/backup-restore folder and edit the
*settings.ini* file,
providing the connection parameter of the *TARGET DB*.

[targetdb]
dbname = geonode_rst
host   = localhost
port   = 5432
user   = geonode
passwd = geonode

This is necessary in case you have a PostgreSQL Backend. The Restore
procedures tries to apply some patches to the DB in order to make things
working.

If you don't have a PostgreSQL backend, just ignore this passage.

Now run the Restore command providing the ZIP file to restore

   *python restore.py backup/2016-02-25_180915.zip*

The command will ask you confirmation to proceed.
By inserting *"y"* it will wipe out the target instance DB and folders and
restore the old ones.

geonode at geonode:~/geonode/scripts/backup-restore$ python restore.py
backup/2016-02-25_180915.zip
WARNING: The restore will overwrite all your GeoNode data and files. Are
you sure you want to proceed? [n]|y: y
Creating tables ...
Installing custom SQL ...
Installing indexes ...
Deserializing restore/2016-02-25_180915/people.json
Installed 3 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/accounts.json
Installed 5 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/avatars.json
WARNING: No valid fixture data found for 'avatars'.
Deserializing restore/2016-02-25_180915/licenses.json
Installed 7 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/topiccategories.json
Installed 19 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/regions.json
Installed 259 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/resourcebases.json
Installed 2 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/contactroles.json
Installed 4 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/links.json
Installed 27 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/restrictioncodetypes.json
Installed 8 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/spatialrepresentationtypes.json
Installed 6 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/useropermissions.json
Installed 18 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/groupopermissions.json
Installed 4 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/uploadsessions.json
Installed 3 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/layers.json
Installed 1 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/attributes.json
Installed 3 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/maps.json
Installed 1 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/maplayers.json
Installed 5 object(s) from 1 fixture(s)
Deserializing restore/2016-02-25_180915/documents.json
WARNING: No valid fixture data found for 'documents'.
Deserializing restore/2016-02-25_180915/tags.json
WARNING: No valid fixture data found for 'tags'.
Media Files Restored into '/home/geonode/geonode/geonode/uploaded'.
Static Root Restored into '/home/geonode/geonode/geonode/static_root'.
Static Files Restored into '/home/geonode/geonode/geonode/static'.
Template Files Restored into '/home/geonode/geonode/geonode/templates'.
Locale Files Restored into '/home/geonode/geonode/geonode/locale'.
Restore finished. Please find restored files and dumps into:
'restore/2016-02-25_180915'.
geonode at geonode:~/geonode/scripts/backup-restore$

That's all for now.
I'm moving forward with the Backup&Restore plugin for GeoServer and more
improvements on the GeoNode backup/restore scripts.

Notice that having JSON fixtures it should be possible with the next
improvements to:

1. Filter out only some features to restore. As an instance you may want to
restore just a subset of the Layers and or Maps

2. Remap the Layers (or part of them) on another instance

For the moment if you want/need to do some experiments with this, just do a
full backup and manually edit the layers.json file inside the ZIP archive.

Best Regards,
Alessio Fabiani.

==
GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.
==

Ing. Alessio Fabiani
@alfa7691
Founder/Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:     +39 0584 1660272
mob:   +39 331 6233686

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

---------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/geonode-devel/attachments/20160225/20086ff2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: backup_restore.patch
Type: application/octet-stream
Size: 1959 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/geonode-devel/attachments/20160225/20086ff2/attachment-0001.obj>