[GeoNode-devel] GeoNode Backup and Restore ... any clue on DJango serialization?

Alessio Fabiani alessio.fabiani at geo-solutions.it
Mon Jan 25 09:28:13 PST 2016


Yep, this is a test that I can try.

I'll let you know if there is any update on this thread.

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.
==

Ing. Alessio Fabiani
@alfa7691
Founder/Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:     +39 0584 1660272
mob:   +39  331 6233686

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.



The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

On Mon, Jan 25, 2016 at 6:19 PM, Simone Dalmasso <simone.dalmasso at gmail.com>
wrote:

> Thanks Alessio for the extensive tests.
>
> One of the early developments that we should do is to upgrade Django to
> the 1.8 and if that fixes that all good.
> A test you could try is to just use django 1.8 to dump the data model and
> reload it (even ifgeonode won't work) and see whether it fixed.
>
> What do you think?
>
> 2016-01-25 18:13 GMT+01:00 Alessio Fabiani <
> alessio.fabiani at geo-solutions.it>:
>
>> Finally I found the issue (but not yet a beautiful solution).
>>
>> The problem is the following one:
>>
>> - We have a resourcebase class which has a manytomany relationship with
>> contactroles and links. The relationships are correctly defined, having a
>> non-null constraint on the resourcebase foreign key and on the role/link
>> ones too.
>>
>> - We have layers, inherited by the resourcebase.
>>
>> The issue:
>>
>> When we dump the data everything works just fine. The data can be
>> successfully dumped to JSON and, paying attention to the order, can be
>> almost all deserialized again on a fresh database.
>>
>> There is no problem in restoring everything except the Layers.
>>
>> In fact, it seems like DJango tries to create the Layer but somehow loses
>> the information on the resourcebase id when trying to restore the many to
>> many relationships. In other words, it tries to create again the
>> contactroles and the links associated to the Layer/ResourceBase, but this
>> time imposing the resourcebase foreignkey as NULL.
>>
>> Relaxing the model and allowing contactroles and links to accept also
>> NULL resources (which means basically have orphan records into the DB in
>> the end), everything works fine, except that we have duplicated
>> contactroles and links, half with the correct resourcebase id reference and
>> half with a null reference.
>>
>> The DJango mailing list seems not providing any solution for this, or at
>> least I was not able to find any at this time. Most probably this issue has
>> been fixed from DJango 1.7a+ since there are similar issues around related
>> to duplicate keys.
>>
>> The trivial workaround:
>>
>> - Modify a bit the model by allowing contactroles and links to accept
>> null values for the resourcebase primary key.
>> The whole integrity of the model is preserved, the only issue is that
>> there will be duplicate values in the DB after the restore.
>>
>> The non-trivial possible solutions:
>>
>> 1) Try to see if newer versions of DJango fix the issue and envisage a
>> GeoNode upgrade
>>
>> 2) Investigate the possibility of refactoring a bit the GeoNode model and
>> see if the many to many relationships can be better/differently defined
>> without breaking the GeoNode DB (possible backward compatibility issues)
>>
>> 3) See if there is a way to somehow patch the model by introducing
>> fields/methods that helps the fixtures to be deserialized correctly.
>>
>> Any feedback/help/idea welcomed.
>>
>> Regards,
>> Alessio.
>>
>>
>>
>> ==
>> GeoServer Professional Services from the experts! Visit
>> http://goo.gl/NWWaa2 for more information.
>> ==
>>
>> Ing. Alessio Fabiani
>> @alfa7691
>> Founder/Technical Lead
>>
>> GeoSolutions S.A.S.
>> Via Poggio alle Viti 1187
>> 55054  Massarosa (LU)
>> Italy
>> phone: +39 0584 962313
>> fax:     +39 0584 1660272
>> mob:   +39  331 6233686
>>
>> http://www.geo-solutions.it
>> http://twitter.com/geosolutions_it
>>
>> -------------------------------------------------------
>>
>> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>>
>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
>> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
>> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
>> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
>> darcene notizia via e-mail e di procedere alla distruzione del messaggio
>> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
>> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
>> principi dettati dal D.Lgs. 196/2003.
>>
>>
>>
>> The information in this message and/or attachments, is intended solely
>> for the attention and use of the named addressee(s) and may be confidential
>> or proprietary in nature or covered by the provisions of privacy act
>> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
>> Code).Any use not in accord with its purpose, any disclosure, reproduction,
>> copying, distribution, or either dissemination, either whole or partial, is
>> strictly forbidden except previous formal approval of the named
>> addressee(s). If you are not the intended recipient, please contact
>> immediately the sender by telephone, fax or e-mail and delete the
>> information in this message that has been received in error. The sender
>> does not give any warranty or accept liability as the content, accuracy or
>> completeness of sent messages and accepts no responsibility  for changes
>> made after they were sent or for other risks which arise as a result of
>> e-mail transmission, viruses, etc.
>>
>> On Mon, Jan 25, 2016 at 1:17 PM, Alessio Fabiani <
>> alessio.fabiani at geo-solutions.it> wrote:
>>
>>> About this issue,
>>> attached you can find a log of an import for a Layer into a fresh DB.
>>>
>>> It is quite strange, the issue seems to be related to the
>>> "base_contactrole" relation.
>>>
>>> In the log you can see ho the "resource_id" for the "base_link" is
>>> correctly parsed while for the "base_contactrole" is NULL.
>>>
>>> I suspect some issues on the GeoNode model definitions.
>>>
>>> Any idea?
>>>
>>> Thanks,
>>> Alessio.
>>>
>>>
>>> ==
>>> GeoServer Professional Services from the experts! Visit
>>> http://goo.gl/NWWaa2 for more information.
>>> ==
>>>
>>> Ing. Alessio Fabiani
>>> @alfa7691
>>> Founder/Technical Lead
>>>
>>> GeoSolutions S.A.S.
>>> Via Poggio alle Viti 1187
>>> 55054  Massarosa (LU)
>>> Italy
>>> phone: +39 0584 962313
>>> fax:     +39 0584 1660272
>>> mob:   +39  331 6233686
>>>
>>> http://www.geo-solutions.it
>>> http://twitter.com/geosolutions_it
>>>
>>> -------------------------------------------------------
>>>
>>> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>>>
>>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>>> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
>>> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
>>> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
>>> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
>>> darcene notizia via e-mail e di procedere alla distruzione del messaggio
>>> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>>> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
>>> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
>>> principi dettati dal D.Lgs. 196/2003.
>>>
>>>
>>>
>>> The information in this message and/or attachments, is intended solely
>>> for the attention and use of the named addressee(s) and may be confidential
>>> or proprietary in nature or covered by the provisions of privacy act
>>> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
>>> Code).Any use not in accord with its purpose, any disclosure, reproduction,
>>> copying, distribution, or either dissemination, either whole or partial, is
>>> strictly forbidden except previous formal approval of the named
>>> addressee(s). If you are not the intended recipient, please contact
>>> immediately the sender by telephone, fax or e-mail and delete the
>>> information in this message that has been received in error. The sender
>>> does not give any warranty or accept liability as the content, accuracy or
>>> completeness of sent messages and accepts no responsibility  for changes
>>> made after they were sent or for other risks which arise as a result of
>>> e-mail transmission, viruses, etc.
>>>
>>> On Fri, Jan 22, 2016 at 6:01 PM, Simone Dalmasso <
>>> simone.dalmasso at gmail.com> wrote:
>>>
>>>> Hi Alessio,
>>>>
>>>> can you check if there is actually a resourcebase with that id into the
>>>> db?
>>>>
>>>> the problem I see is that the id is assigned by the db so it may change
>>>> unless the resurcebase sequence is restored to 0. but the there is the
>>>> reference to the resourcebase into the layer,
>>>>
>>>> 2016-01-22 17:38 GMT+01:00 Alessio Fabiani <
>>>> alessio.fabiani at geo-solutions.it>:
>>>>
>>>>> Taking a look into the logs, the first JSON is correct. The issue is
>>>>> on the reloading of the base_contactrole dependency.
>>>>>
>>>>> ==
>>>>> GeoServer Professional Services from the experts! Visit
>>>>> http://goo.gl/NWWaa2 for more information.
>>>>> ==
>>>>>
>>>>> Ing. Alessio Fabiani
>>>>> @alfa7691
>>>>> Founder/Technical Lead
>>>>>
>>>>> GeoSolutions S.A.S.
>>>>> Via Poggio alle Viti 1187
>>>>> 55054  Massarosa (LU)
>>>>> Italy
>>>>> phone: +39 0584 962313
>>>>> fax:     +39 0584 1660272
>>>>> mob:   +39  331 6233686
>>>>>
>>>>> http://www.geo-solutions.it
>>>>> http://twitter.com/geosolutions_it
>>>>>
>>>>> -------------------------------------------------------
>>>>>
>>>>> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>>>>>
>>>>> Le informazioni contenute in questo messaggio di posta elettronica e/o
>>>>> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
>>>>> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
>>>>> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
>>>>> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
>>>>> darcene notizia via e-mail e di procedere alla distruzione del messaggio
>>>>> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>>>>> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
>>>>> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
>>>>> principi dettati dal D.Lgs. 196/2003.
>>>>>
>>>>>
>>>>>
>>>>> The information in this message and/or attachments, is intended solely
>>>>> for the attention and use of the named addressee(s) and may be confidential
>>>>> or proprietary in nature or covered by the provisions of privacy act
>>>>> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
>>>>> Code).Any use not in accord with its purpose, any disclosure, reproduction,
>>>>> copying, distribution, or either dissemination, either whole or partial, is
>>>>> strictly forbidden except previous formal approval of the named
>>>>> addressee(s). If you are not the intended recipient, please contact
>>>>> immediately the sender by telephone, fax or e-mail and delete the
>>>>> information in this message that has been received in error. The sender
>>>>> does not give any warranty or accept liability as the content, accuracy or
>>>>> completeness of sent messages and accepts no responsibility  for changes
>>>>> made after they were sent or for other risks which arise as a result of
>>>>> e-mail transmission, viruses, etc.
>>>>>
>>>>> On Fri, Jan 22, 2016 at 5:26 PM, Alessio Fabiani <
>>>>> alessio.fabiani at geo-solutions.it> wrote:
>>>>>
>>>>>> Dear list,
>>>>>> working on the GeoNode Backup and Restore task I'm struggling with an
>>>>>> issue while trying to deserialize data.
>>>>>>
>>>>>> Basically what I'm trying to do is dump the GeoNode fixtures as JSON
>>>>>> files and trying to load them later on an empty (prepared) GeoNode DB.
>>>>>>
>>>>>> Now ... by playing a bit with the json outputs and load order I am
>>>>>> able to restore almost all the fixtures, but I have an issue with the
>>>>>> Layers, inherited from ResourceBases.
>>>>>>
>>>>>> I am able to restore succesfully the ResourceBases but whenever I try
>>>>>> to deserialize a Layer I'm getting and error with the primary key.
>>>>>>
>>>>>> Using the default JSON like this:
>>>>>>
>>>>>> [
>>>>>> {
>>>>>>   "pk": 1,
>>>>>>   "model": "layers.layer",
>>>>>>   "fields": {
>>>>>>     "styles": [
>>>>>>       1
>>>>>>     ],
>>>>>>     "store": "datastore",
>>>>>>     "name": "san_andres_y_providencia_administrative",
>>>>>>     "service": null,
>>>>>>     "charset": "UTF-8",
>>>>>>     "title_en": "San Andres Y Providencia Administrative",
>>>>>>     "abstract_en": "No abstract provided",
>>>>>>     "regions": [],
>>>>>>     "typename": "geonode:san_andres_y_providencia_administrative",
>>>>>>     "default_style": null,
>>>>>>     "purpose_en": null,
>>>>>>     "workspace": "geonode",
>>>>>>     "upload_session": 9,
>>>>>>     "supplemental_information_en": "No information provided",
>>>>>>     "data_quality_statement_en": null,
>>>>>>     "storeType": "dataStore",
>>>>>>     "distribution_description_en": null,
>>>>>>     "constraints_other_en": null
>>>>>>   }
>>>>>> }
>>>>>> ]
>>>>>>
>>>>>> I get an error message like: null violates constraint for
>>>>>> "resource_id"
>>>>>>
>>>>>> If I try to modify the JSON like this:
>>>>>>
>>>>>> [
>>>>>> {
>>>>>>   "*resourcebase_ptr_id*": 1,
>>>>>>   "model": "layers.layer",
>>>>>>   "fields": {
>>>>>>     "styles": [
>>>>>>       1
>>>>>>     ],
>>>>>>     "store": "datastore",
>>>>>>     "name": "san_andres_y_providencia_administrative",
>>>>>>     "service": null,
>>>>>>     "charset": "UTF-8",
>>>>>>     "title_en": "San Andres Y Providencia Administrative",
>>>>>>     "abstract_en": "No abstract provided",
>>>>>>     "regions": [],
>>>>>>     "typename": "geonode:san_andres_y_providencia_administrative",
>>>>>>     "default_style": null,
>>>>>>     "purpose_en": null,
>>>>>>     "workspace": "geonode",
>>>>>>     "upload_session": 9,
>>>>>>     "supplemental_information_en": "No information provided",
>>>>>>     "data_quality_statement_en": null,
>>>>>>     "storeType": "dataStore",
>>>>>>     "distribution_description_en": null,
>>>>>>     "constraints_other_en": null
>>>>>>   }
>>>>>> }
>>>>>> ]
>>>>>>
>>>>>> I get the following error:
>>>>>>
>>>>>> ResourceBase matching query does not exist.
>>>>>>
>>>>>> Any idea/help ?
>>>>>>
>>>>>> ==
>>>>>> GeoServer Professional Services from the experts! Visit
>>>>>> http://goo.gl/NWWaa2 for more information.
>>>>>> ==
>>>>>>
>>>>>> Ing. Alessio Fabiani
>>>>>> @alfa7691
>>>>>> Founder/Technical Lead
>>>>>>
>>>>>> GeoSolutions S.A.S.
>>>>>> Via Poggio alle Viti 1187
>>>>>> 55054  Massarosa (LU)
>>>>>> Italy
>>>>>> phone: +39 0584 962313
>>>>>> fax:     +39 0584 1660272
>>>>>> mob:   +39  331 6233686
>>>>>>
>>>>>> http://www.geo-solutions.it
>>>>>> http://twitter.com/geosolutions_it
>>>>>>
>>>>>> -------------------------------------------------------
>>>>>>
>>>>>> *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*
>>>>>>
>>>>>> Le informazioni contenute in questo messaggio di posta elettronica
>>>>>> e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
>>>>>> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
>>>>>> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
>>>>>> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
>>>>>> darcene notizia via e-mail e di procedere alla distruzione del messaggio
>>>>>> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
>>>>>> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
>>>>>> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
>>>>>> principi dettati dal D.Lgs. 196/2003.
>>>>>>
>>>>>>
>>>>>>
>>>>>> The information in this message and/or attachments, is intended
>>>>>> solely for the attention and use of the named addressee(s) and may be
>>>>>> confidential or proprietary in nature or covered by the provisions of
>>>>>> privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data
>>>>>> Protection Code).Any use not in accord with its purpose, any disclosure,
>>>>>> reproduction, copying, distribution, or either dissemination, either whole
>>>>>> or partial, is strictly forbidden except previous formal approval of the
>>>>>> named addressee(s). If you are not the intended recipient, please contact
>>>>>> immediately the sender by telephone, fax or e-mail and delete the
>>>>>> information in this message that has been received in error. The sender
>>>>>> does not give any warranty or accept liability as the content, accuracy or
>>>>>> completeness of sent messages and accepts no responsibility  for changes
>>>>>> made after they were sent or for other risks which arise as a result of
>>>>>> e-mail transmission, viruses, etc.
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> geonode-devel mailing list
>>>>> geonode-devel at lists.osgeo.org
>>>>> http://lists.osgeo.org/mailman/listinfo/geonode-devel
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Simone
>>>>
>>>
>>>
>>
>
>
> --
> Simone
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/geonode-devel/attachments/20160125/ab66360b/attachment-0001.html>


More information about the geonode-devel mailing list