[GeoNode-devel] [GeoNode-users] GeoNode Backup and Restore: first draft

Fri Feb 26 05:16:38 PST 2016

Just a last note,
the main scope of this work is to add to GeoNode a full backup and restore
functionality, migration one would be an extension.

Best Regards,
Alessio Fabiani.

==
GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.
==

Ing. Alessio Fabiani
@alfa7691
Founder/Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:     +39 0584 1660272
mob:   +39 331 6233686

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

---------------------------------------------------------------------

On Fri, Feb 26, 2016 at 2:15 PM, Alessio Fabiani <
alessio.fabiani at geo-solutions.it> wrote:

> Yep,
> currently the basic idea would be to add code capable of patching the JSON
> fixtures directly.
>
> My intention is to start adding features to the management commands in
> order to deserialize the JSON fixtures, filter them and/or serialize back
> for other versions of GeoNode.
>
> Potentially there could be specific fixtures serializers for any GeoNode
> release.
>
>
> Best Regards,
> Alessio Fabiani.
>
> ==
> GeoServer Professional Services from the experts!
> Visit http://goo.gl/it488V for more information.
> ==
>
> Ing. Alessio Fabiani
> @alfa7691
> Founder/Technical Lead
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> Italy
> phone: +39 0584 962313
> fax:     +39 0584 1660272
> mob:   +39 331 6233686
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> -------------------------------------------------------
>
> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
>
>
> The information in this message and/or attachments, is intended solely for
> the attention and use of the named addressee(s) and may be confidential or
> proprietary in nature or covered by the provisions of privacy act
> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
> Code).Any use not in accord with its purpose, any disclosure, reproduction,
> copying, distribution, or either dissemination, either whole or partial, is
> strictly forbidden except previous formal approval of the named
> addressee(s). If you are not the intended recipient, please contact
> immediately the sender by telephone, fax or e-mail and delete the
> information in this message that has been received in error. The sender
> does not give any warranty or accept liability as the content, accuracy or
> completeness of sent messages and accepts no responsibility  for changes
> made after they were sent or for other risks which arise as a result of
> e-mail transmission, viruses, etc.
>
> ---------------------------------------------------------------------
>
> On Fri, Feb 26, 2016 at 2:08 PM, Ariel Nunez <ingenieroariel at gmail.com>
> wrote:
>
>> Alesio, thanks for this work, looks very exciting.
>>
>> Very excited to see the GeoServer plugin side afterwards.
>>
>> One question, what's the strategy for migrating between different
>> versions of GeoNode? In particular between GeoNode forks and between future
>> (and db incompatible) GeoNode versions.
>>
>> One of the main uses for these scripts would be to help users backup
>> their GeoNode before attempting an upgrade, either between major releases
>> or between heavily modified dev versions.
>>
>> -a
>>
>> On Fri, Feb 26, 2016 at 7:07 AM, Alessio Fabiani <
>> alessio.fabiani at geo-solutions.it> wrote:
>>
>>> Dear all,
>>> as a follow up on this you can find here
>>>
>>>
>>> https://github.com/GeoNode/geonode/commit/cf1c7e7c7a90239f5171a6f83608d4d40360c262
>>>
>>> The GeoNode backup and restore utilities converted into GeoNode Base
>>> Management Commands.
>>>
>>> They can be invoked now as:
>>>
>>>
>>> *python manage.py backup --backup-dir=/tmp/*
>>>
>>> *python manage.py restore --backup-file=/tmp/2016-02-26_054445
>>> <054445>.zip*
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Best Regards,
>>> Alessio Fabiani.
>>>
>>> ==
>>> GeoServer Professional Services from the experts!
>>> Visit http://goo.gl/it488V for more information.
>>> ==
>>>
>>> Ing. Alessio Fabiani
>>> @alfa7691
>>> Founder/Technical Lead
>>>
>>> GeoSolutions S.A.S.
>>> Via di Montramito 3/A
>>> 55054  Massarosa (LU)
>>> Italy
>>> phone: +39 0584 962313
>>> fax:     +39 0584 1660272
>>> mob:   +39 331 6233686
>>>
>>> http://www.geo-solutions.it
>>> http://twitter.com/geosolutions_it
>>>
>>> -------------------------------------------------------
>>>
>>> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>>>
>>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>>> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
>>> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
>>> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
>>> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
>>> darcene notizia via e-mail e di procedere alla distruzione del messaggio
>>> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>>> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
>>> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
>>> principi dettati dal D.Lgs. 196/2003.
>>>
>>>
>>>
>>> The information in this message and/or attachments, is intended solely
>>> for the attention and use of the named addressee(s) and may be confidential
>>> or proprietary in nature or covered by the provisions of privacy act
>>> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
>>> Code).Any use not in accord with its purpose, any disclosure, reproduction,
>>> copying, distribution, or either dissemination, either whole or partial, is
>>> strictly forbidden except previous formal approval of the named
>>> addressee(s). If you are not the intended recipient, please contact
>>> immediately the sender by telephone, fax or e-mail and delete the
>>> information in this message that has been received in error. The sender
>>> does not give any warranty or accept liability as the content, accuracy or
>>> completeness of sent messages and accepts no responsibility  for changes
>>> made after they were sent or for other risks which arise as a result of
>>> e-mail transmission, viruses, etc.
>>>
>>> ---------------------------------------------------------------------
>>>
>>> On Thu, Feb 25, 2016 at 4:22 PM, Alessio Fabiani <
>>> alessio.fabiani at geo-solutions.it> wrote:
>>>
>>>> Dear all,
>>>>
>>>> please find here
>>>>
>>>> *https://github.com/GeoNode/geonode/tree/backup_restore/scripts/backup-restore
>>>> <https://github.com/GeoNode/geonode/tree/backup_restore/scripts/backup-restore>*
>>>>
>>>> a first implementation of the backup and restore scripts using JSON
>>>> fixtures.
>>>>
>>>> The scripts allow to automatically store into a ZIP archive all the
>>>> GeoNode Objects as JSON files along with the static resources.
>>>>
>>>> A restore command takes the zip file as input and restores everything.
>>>> At the end you will have an exact copy of the old GeoNode replicated into
>>>> the new instance independently from the datastore.
>>>>
>>>> *NOTICE 1:* I'm still working on the GeoServer side. GeoServer will be
>>>> enriched with a plugin allowing an administrator to store on a zip file the
>>>> whole data and configuration and having the ability to restore it at any
>>>> time.
>>>>
>>>> *NOTICE 2: *In order to make things working you'll need to apply also
>>>> the "patch" attached to the email to the GeoNode source code *OR* clone
>>>> the *backup_restore *repository
>>>>
>>>> How the GeoNode scripts work:
>>>>
>>>> 1. Backup (on source instance)
>>>> ----------------------------------------------
>>>>
>>>> Just go to the geonode/scripts/backup-restore folder and run the command
>>>>
>>>>     *python backup.py*
>>>>
>>>> The command will output few lines on the console
>>>>
>>>> geonode at geonode:~/geonode/scripts/backup-restore$ python backup.py
>>>> Dumping 'people' into 'people.json'.
>>>> Dumping 'account' into 'accounts.json'.
>>>> Dumping 'avatar.avatar' into 'avatars.json'.
>>>> Dumping 'base.license' into 'licenses.json'.
>>>> Dumping 'base.topiccategory' into 'topiccategories.json'.
>>>> Dumping 'base.region' into 'regions.json'.
>>>> Dumping 'base.resourcebase' into 'resourcebases.json'.
>>>> Dumping 'base.contactrole' into 'contactroles.json'.
>>>> Dumping 'base.link' into 'links.json'.
>>>> Dumping 'base.restrictioncodetype' into 'restrictioncodetypes.json'.
>>>> Dumping 'base.spatialrepresentationtype' into
>>>> 'spatialrepresentationtypes.json'.
>>>> Dumping 'guardian.userobjectpermission' into 'useropermissions.json'.
>>>> Dumping 'guardian.groupobjectpermission' into 'groupopermissions.json'.
>>>> Dumping 'layers.uploadsession' into 'uploadsessions.json'.
>>>> Dumping 'layers.layer' into 'layers.json'.
>>>> Dumping 'layers.attribute' into 'attributes.json'.
>>>> Dumping 'maps.map' into 'maps.json'.
>>>> Dumping 'maps.maplayer' into 'maplayers.json'.
>>>> Dumping 'documents.document' into 'documents.json'.
>>>> Dumping 'taggit' into 'tags.json'.
>>>> Saved Media Files from '/home/geonode/geonode/geonode/uploaded'.
>>>> Saved Static Root from '/home/geonode/geonode/geonode/static_root'.
>>>> Saved Static Files from '/home/geonode/geonode/geonode/static'.
>>>> Saved Template Files from '/home/geonode/geonode/geonode/templates'.
>>>> Saved Locale Files from '/home/geonode/geonode/geonode/locale'.
>>>> Backup Finished. Archive generated 'backup/*2016-02-25_180915.zip*'.
>>>> None
>>>> geonode at geonode:~/geonode/scripts/backup-restore$
>>>>
>>>>
>>>> and will store a timestamped ZIP file into the "backup" subfolder.
>>>>
>>>> Notice that the scripts gets the locations and settings automatically
>>>> from the "geonode.settings" file.
>>>>
>>>> 2. Restore (on destination instance)
>>>> ----------------------------------------------------
>>>>
>>>> *WARNING: **This procedure will completely purge and overwrite your
>>>> destination GeoNode instance. BE CAREFUL.*
>>>>
>>>> Go to the geonode/scripts/backup-restore folder and edit the
>>>> *settings.ini* file, providing the connection parameter of the *TARGET
>>>> DB*.
>>>>
>>>> [targetdb]
>>>> dbname = geonode_rst
>>>> host   = localhost
>>>> port   = 5432
>>>> user   = geonode
>>>> passwd = geonode
>>>>
>>>>
>>>> This is necessary in case you have a PostgreSQL Backend. The Restore
>>>> procedures tries to apply some patches to the DB in order to make things
>>>> working.
>>>>
>>>> If you don't have a PostgreSQL backend, just ignore this passage.
>>>>
>>>> Now run the Restore command providing the ZIP file to restore
>>>>
>>>>    *python restore.py backup/2016-02-25_180915.zip*
>>>>
>>>> The command will ask you confirmation to proceed.
>>>> By inserting *"y"* it will wipe out the target instance DB and folders
>>>> and restore the old ones.
>>>>
>>>> geonode at geonode:~/geonode/scripts/backup-restore$ python restore.py
>>>> backup/2016-02-25_180915.zip
>>>> WARNING: The restore will overwrite all your GeoNode data and files.
>>>> Are you sure you want to proceed? [n]|y: y
>>>> Creating tables ...
>>>> Installing custom SQL ...
>>>> Installing indexes ...
>>>> Deserializing restore/2016-02-25_180915/people.json
>>>> Installed 3 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/accounts.json
>>>> Installed 5 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/avatars.json
>>>> WARNING: No valid fixture data found for 'avatars'.
>>>> Deserializing restore/2016-02-25_180915/licenses.json
>>>> Installed 7 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/topiccategories.json
>>>> Installed 19 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/regions.json
>>>> Installed 259 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/resourcebases.json
>>>> Installed 2 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/contactroles.json
>>>> Installed 4 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/links.json
>>>> Installed 27 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/restrictioncodetypes.json
>>>> Installed 8 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/spatialrepresentationtypes.json
>>>> Installed 6 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/useropermissions.json
>>>> Installed 18 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/groupopermissions.json
>>>> Installed 4 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/uploadsessions.json
>>>> Installed 3 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/layers.json
>>>> Installed 1 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/attributes.json
>>>> Installed 3 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/maps.json
>>>> Installed 1 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/maplayers.json
>>>> Installed 5 object(s) from 1 fixture(s)
>>>> Deserializing restore/2016-02-25_180915/documents.json
>>>> WARNING: No valid fixture data found for 'documents'.
>>>> Deserializing restore/2016-02-25_180915/tags.json
>>>> WARNING: No valid fixture data found for 'tags'.
>>>> Media Files Restored into '/home/geonode/geonode/geonode/uploaded'.
>>>> Static Root Restored into '/home/geonode/geonode/geonode/static_root'.
>>>> Static Files Restored into '/home/geonode/geonode/geonode/static'.
>>>> Template Files Restored into '/home/geonode/geonode/geonode/templates'.
>>>> Locale Files Restored into '/home/geonode/geonode/geonode/locale'.
>>>> Restore finished. Please find restored files and dumps into:
>>>> 'restore/2016-02-25_180915'.
>>>> geonode at geonode:~/geonode/scripts/backup-restore$
>>>>
>>>>
>>>>
>>>> That's all for now.
>>>> I'm moving forward with the Backup&Restore plugin for GeoServer and
>>>> more improvements on the GeoNode backup/restore scripts.
>>>>
>>>> Notice that having JSON fixtures it should be possible with the next
>>>> improvements to:
>>>>
>>>> 1. Filter out only some features to restore. As an instance you may
>>>> want to restore just a subset of the Layers and or Maps
>>>>
>>>> 2. Remap the Layers (or part of them) on another instance
>>>>
>>>> For the moment if you want/need to do some experiments with this, just
>>>> do a full backup and manually edit the layers.json file inside the ZIP
>>>> archive.
>>>>
>>>>
>>>> Best Regards,
>>>> Alessio Fabiani.
>>>>
>>>> ==
>>>> GeoServer Professional Services from the experts!
>>>> Visit http://goo.gl/it488V for more information.
>>>> ==
>>>>
>>>> Ing. Alessio Fabiani
>>>> @alfa7691
>>>> Founder/Technical Lead
>>>>
>>>> GeoSolutions S.A.S.
>>>> Via di Montramito 3/A
>>>> 55054  Massarosa (LU)
>>>> Italy
>>>> phone: +39 0584 962313
>>>> fax:     +39 0584 1660272
>>>> mob:   +39 331 6233686
>>>>
>>>> http://www.geo-solutions.it
>>>> http://twitter.com/geosolutions_it
>>>>
>>>> -------------------------------------------------------
>>>>
>>>> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>>>>
>>>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>>>> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
>>>> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
>>>> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
>>>> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
>>>> darcene notizia via e-mail e di procedere alla distruzione del messaggio
>>>> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>>>> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
>>>> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
>>>> principi dettati dal D.Lgs. 196/2003.
>>>>
>>>>
>>>>
>>>> The information in this message and/or attachments, is intended solely
>>>> for the attention and use of the named addressee(s) and may be confidential
>>>> or proprietary in nature or covered by the provisions of privacy act
>>>> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
>>>> Code).Any use not in accord with its purpose, any disclosure, reproduction,
>>>> copying, distribution, or either dissemination, either whole or partial, is
>>>> strictly forbidden except previous formal approval of the named
>>>> addressee(s). If you are not the intended recipient, please contact
>>>> immediately the sender by telephone, fax or e-mail and delete the
>>>> information in this message that has been received in error. The sender
>>>> does not give any warranty or accept liability as the content, accuracy or
>>>> completeness of sent messages and accepts no responsibility  for changes
>>>> made after they were sent or for other risks which arise as a result of
>>>> e-mail transmission, viruses, etc.
>>>>
>>>> ---------------------------------------------------------------------
>>>>
>>>
>>>
>>> _______________________________________________
>>> geonode-users mailing list
>>> geonode-users at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/geonode-users
>>>
>>>
>>
>> _______________________________________________
>> geonode-users mailing list
>> geonode-users at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/geonode-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/geonode-devel/attachments/20160226/aba0cfee/attachment-0001.html>