<div dir="ltr">Thanks Alessio for the extensive tests.<div><br></div><div>One of the early developments that we should do is to upgrade Django to the 1.8 and if that fixes that all good.</div><div>A test you could try is to just use django 1.8 to dump the data model and reload it (even ifgeonode won't work) and see whether it fixed.</div><div><br></div><div>What do you think? </div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-01-25 18:13 GMT+01:00 Alessio Fabiani <span dir="ltr"><<a href="mailto:alessio.fabiani@geo-solutions.it" target="_blank">alessio.fabiani@geo-solutions.it</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Finally I found the issue (but not yet a beautiful solution).<div><br></div><div>The problem is the following one:</div><div><br></div><div>- We have a resourcebase class which has a manytomany relationship with contactroles and links. The relationships are correctly defined, having a non-null constraint on the resourcebase foreign key and on the role/link ones too.</div><div><br></div><div>- We have layers, inherited by the resourcebase.</div><div><br></div><div>The issue:</div><div><br></div><div>When we dump the data everything works just fine. The data can be successfully dumped to JSON and, paying attention to the order, can be almost all deserialized again on a fresh database.</div><div><br></div><div>There is no problem in restoring everything except the Layers.</div><div><br></div><div>In fact, it seems like DJango tries to create the Layer but somehow loses the information on the resourcebase id when trying to restore the many to many relationships. In other words, it tries to create again the contactroles and the links associated to the Layer/ResourceBase, but this time imposing the resourcebase foreignkey as NULL.</div><div><br></div><div>Relaxing the model and allowing contactroles and links to accept also NULL resources (which means basically have orphan records into the DB in the end), everything works fine, except that we have duplicated contactroles and links, half with the correct resourcebase id reference and half with a null reference.</div><div><br></div><div>The DJango mailing list seems not providing any solution for this, or at least I was not able to find any at this time. Most probably this issue has been fixed from DJango 1.7a+ since there are similar issues around related to duplicate keys.</div><div><br></div><div>The trivial workaround:</div><div><br></div><div>- Modify a bit the model by allowing contactroles and links to accept null values for the resourcebase primary key.</div><div>The whole integrity of the model is preserved, the only issue is that there will be duplicate values in the DB after the restore.</div><div><br></div><div>The non-trivial possible solutions:</div><div><br></div><div>1) Try to see if newer versions of DJango fix the issue and envisage a GeoNode upgrade</div><div><br></div><div>2) Investigate the possibility of refactoring a bit the GeoNode model and see if the many to many relationships can be better/differently defined without breaking the GeoNode DB (possible backward compatibility issues)</div><div><br></div><div>3) See if there is a way to somehow patch the model by introducing fields/methods that helps the fixtures to be deserialized correctly.</div><div><br></div><div>Any feedback/help/idea welcomed.</div><div><br></div><div>Regards,</div><div>Alessio.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>==<br>GeoServer Professional Services from the experts! Visit<br><a href="http://goo.gl/NWWaa2" style="color:rgb(17,85,204)" target="_blank">http://goo.gl/NWWaa2</a> for more information.<br>==<br></div><div><br></div><div>Ing. Alessio Fabiani</div><div>@alfa7691</div><div>Founder/Technical Lead</div><div><br></div><div>GeoSolutions S.A.S.</div><div>Via Poggio alle Viti 1187</div><div>55054 Massarosa (LU)</div><div>Italy</div><div>phone: <a href="tel:%2B39%200584%20962313" value="+390584962313" target="_blank">+39 0584 962313</a></div><div>fax: <a href="tel:%2B39%200584%201660272" value="+3905841660272" target="_blank">+39 0584 1660272</a></div><div>mob: <a href="tel:%2B39%20%C2%A0331%206233686" value="+393316233686" target="_blank">+39 331 6233686</a></div><div><br></div><div><a href="http://www.geo-solutions.it" target="_blank">http://www.geo-solutions.it</a></div><div><a href="http://twitter.com/geosolutions_it" target="_blank">http://twitter.com/geosolutions_it</a></div><div><br></div><div>-------------------------------------------------------</div><div><br></div><div><p><span lang="IT"><font size="1"><b>AVVERTENZE AI SENSI DEL D.Lgs. 196/2003</b></font></span></p><p><span lang="IT"><font size="1">Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.</font></span></p><p><span lang="IT"><font size="1"> </font></span></p><p><font size="1">The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.</font></p></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Mon, Jan 25, 2016 at 1:17 PM, Alessio Fabiani <span dir="ltr"><<a href="mailto:alessio.fabiani@geo-solutions.it" target="_blank">alessio.fabiani@geo-solutions.it</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">About this issue,<div>attached you can find a log of an import for a Layer into a fresh DB.</div><div><br></div><div>It is quite strange, the issue seems to be related to the "base_contactrole" relation.</div><div><br></div><div>In the log you can see ho the "resource_id" for the "base_link" is correctly parsed while for the "base_contactrole" is NULL.</div><div><br></div><div>I suspect some issues on the GeoNode model definitions.</div><div><br></div><div>Any idea?</div><div><br></div><div>Thanks,</div><div>Alessio.</div><div><br></div></div><div class="gmail_extra"><span><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>==<br>GeoServer Professional Services from the experts! Visit<br><a href="http://goo.gl/NWWaa2" style="color:rgb(17,85,204)" target="_blank">http://goo.gl/NWWaa2</a> for more information.<br>==<br></div><div><br></div><div>Ing. Alessio Fabiani</div><div>@alfa7691</div><div>Founder/Technical Lead</div><div><br></div><div>GeoSolutions S.A.S.</div><div>Via Poggio alle Viti 1187</div><div>55054 Massarosa (LU)</div><div>Italy</div><div>phone: <a href="tel:%2B39%200584%20962313" value="+390584962313" target="_blank">+39 0584 962313</a></div><div>fax: <a href="tel:%2B39%200584%201660272" value="+3905841660272" target="_blank">+39 0584 1660272</a></div><div>mob: <a href="tel:%2B39%20%C2%A0331%206233686" value="+393316233686" target="_blank">+39 331 6233686</a></div><div><br></div><div><a href="http://www.geo-solutions.it" target="_blank">http://www.geo-solutions.it</a></div><div><a href="http://twitter.com/geosolutions_it" target="_blank">http://twitter.com/geosolutions_it</a></div><div><br></div><div>-------------------------------------------------------</div><div><br></div><div><p><span lang="IT"><font size="1"><b>AVVERTENZE AI SENSI DEL D.Lgs. 196/2003</b></font></span></p><p><span lang="IT"><font size="1">Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.</font></span></p><p><span lang="IT"><font size="1"> </font></span></p><p><font size="1">The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.</font></p></div></div></div></div></div></div></div></div>
<br></span><div><div><div class="gmail_quote">On Fri, Jan 22, 2016 at 6:01 PM, Simone Dalmasso <span dir="ltr"><<a href="mailto:simone.dalmasso@gmail.com" target="_blank">simone.dalmasso@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Alessio,<div><br></div><div>can you check if there is actually a resourcebase with that id into the db?</div><div><br></div><div>the problem I see is that the id is assigned by the db so it may change unless the resurcebase sequence is restored to 0. but the there is the reference to the resourcebase into the layer,</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>2016-01-22 17:38 GMT+01:00 Alessio Fabiani <span dir="ltr"><<a href="mailto:alessio.fabiani@geo-solutions.it" target="_blank">alessio.fabiani@geo-solutions.it</a>></span>:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Taking a look into the logs, the first JSON is correct. The issue is on the reloading of the base_contactrole dependency.</div><div class="gmail_extra"><span><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>==<br>GeoServer Professional Services from the experts! Visit<br><a href="http://goo.gl/NWWaa2" style="color:rgb(17,85,204)" target="_blank">http://goo.gl/NWWaa2</a> for more information.<br>==<br></div><div><br></div><div>Ing. Alessio Fabiani</div><div>@alfa7691</div><div>Founder/Technical Lead</div><div><br></div><div>GeoSolutions S.A.S.</div><div>Via Poggio alle Viti 1187</div><div>55054 Massarosa (LU)</div><div>Italy</div><div>phone: <a href="tel:%2B39%200584%20962313" value="+390584962313" target="_blank">+39 0584 962313</a></div><div>fax: <a href="tel:%2B39%200584%201660272" value="+3905841660272" target="_blank">+39 0584 1660272</a></div><div>mob: <a href="tel:%2B39%20%C2%A0331%206233686" value="+393316233686" target="_blank">+39 331 6233686</a></div><div><br></div><div><a href="http://www.geo-solutions.it" target="_blank">http://www.geo-solutions.it</a></div><div><a href="http://twitter.com/geosolutions_it" target="_blank">http://twitter.com/geosolutions_it</a></div><div><br></div><div>-------------------------------------------------------</div><div><br></div><div><p><span lang="IT"><font size="1"><b>AVVERTENZE AI SENSI DEL D.Lgs. 196/2003</b></font></span></p><p><span lang="IT"><font size="1">Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.</font></span></p><p><span lang="IT"><font size="1"> </font></span></p><p><font size="1">The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.</font></p></div></div></div></div></div></div></div></div>
<br></span><div><div><div class="gmail_quote">On Fri, Jan 22, 2016 at 5:26 PM, Alessio Fabiani <span dir="ltr"><<a href="mailto:alessio.fabiani@geo-solutions.it" target="_blank">alessio.fabiani@geo-solutions.it</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Dear list,<div>working on the GeoNode Backup and Restore task I'm struggling with an issue while trying to deserialize data.</div><div><br></div><div>Basically what I'm trying to do is dump the GeoNode fixtures as JSON files and trying to load them later on an empty (prepared) GeoNode DB.</div><div><br></div><div>Now ... by playing a bit with the json outputs and load order I am able to restore almost all the fixtures, but I have an issue with the Layers, inherited from ResourceBases.</div><div><br></div><div>I am able to restore succesfully the ResourceBases but whenever I try to deserialize a Layer I'm getting and error with the primary key.</div><div><br></div><div>Using the default JSON like this:</div><div><br></div><div><div>[</div><div>{</div><div> "pk": 1, </div><div> "model": "layers.layer", </div><div> "fields": {</div><div> "styles": [</div><div> 1</div><div> ], </div><div> "store": "datastore", </div><div> "name": "san_andres_y_providencia_administrative", </div><div> "service": null, </div><div> "charset": "UTF-8", </div><div> "title_en": "San Andres Y Providencia Administrative", </div><div> "abstract_en": "No abstract provided", </div><div> "regions": [], </div><div> "typename": "geonode:san_andres_y_providencia_administrative", </div><div> "default_style": null, </div><div> "purpose_en": null, </div><div> "workspace": "geonode", </div><div> "upload_session": 9, </div><div> "supplemental_information_en": "No information provided", </div><div> "data_quality_statement_en": null, </div><div> "storeType": "dataStore", </div><div> "distribution_description_en": null, </div><div> "constraints_other_en": null</div><div> }</div><div>}</div><div>]</div></div><div><br></div><div>I get an error message like: null violates constraint for "resource_id"</div><div><br></div><div>If I try to modify the JSON like this:</div><div><br></div><div><div>[</div><div>{</div><div> "<b>resourcebase_ptr_id</b>": 1, </div><div> "model": "layers.layer", </div><div> "fields": {</div><div> "styles": [</div><div> 1</div><div> ], </div><div> "store": "datastore", </div><div> "name": "san_andres_y_providencia_administrative", </div><div> "service": null, </div><div> "charset": "UTF-8", </div><div> "title_en": "San Andres Y Providencia Administrative", </div><div> "abstract_en": "No abstract provided", </div><div> "regions": [], </div><div> "typename": "geonode:san_andres_y_providencia_administrative", </div><div> "default_style": null, </div><div> "purpose_en": null, </div><div> "workspace": "geonode", </div><div> "upload_session": 9, </div><div> "supplemental_information_en": "No information provided", </div><div> "data_quality_statement_en": null, </div><div> "storeType": "dataStore", </div><div> "distribution_description_en": null, </div><div> "constraints_other_en": null</div><div> }</div><div>}</div><div>]</div></div><div><br></div><div>I get the following error:</div><div><br></div><div>ResourceBase matching query does not exist.<br></div><div><br></div><div>Any idea/help ?</div><div><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>==<br>GeoServer Professional Services from the experts! Visit<br><a href="http://goo.gl/NWWaa2" style="color:rgb(17,85,204)" target="_blank">http://goo.gl/NWWaa2</a> for more information.<br>==<br></div><div><br></div><div>Ing. Alessio Fabiani</div><div>@alfa7691</div><div>Founder/Technical Lead</div><div><br></div><div>GeoSolutions S.A.S.</div><div>Via Poggio alle Viti 1187</div><div>55054 Massarosa (LU)</div><div>Italy</div><div>phone: <a href="tel:%2B39%200584%20962313" value="+390584962313" target="_blank">+39 0584 962313</a></div><div>fax: <a href="tel:%2B39%200584%201660272" value="+3905841660272" target="_blank">+39 0584 1660272</a></div><div>mob: <a href="tel:%2B39%20%C2%A0331%206233686" value="+393316233686" target="_blank">+39 331 6233686</a></div><div><br></div><div><a href="http://www.geo-solutions.it" target="_blank">http://www.geo-solutions.it</a></div><div><a href="http://twitter.com/geosolutions_it" target="_blank">http://twitter.com/geosolutions_it</a></div><div><br></div><div>-------------------------------------------------------</div><div><br></div><div><p><span lang="IT"><font size="1"><b>AVVERTENZE AI SENSI DEL D.Lgs. 196/2003</b></font></span></p><p><span lang="IT"><font size="1">Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.</font></span></p><p><span lang="IT"><font size="1"> </font></span></p><p><font size="1">The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.</font></p></div></div></div></div></div></div></div></div>
</div></div>
</blockquote></div><br></div></div></div>
<br></div></div><span>_______________________________________________<br>
geonode-devel mailing list<br>
<a href="mailto:geonode-devel@lists.osgeo.org" target="_blank">geonode-devel@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/geonode-devel" rel="noreferrer" target="_blank">http://lists.osgeo.org/mailman/listinfo/geonode-devel</a><br>
<br></span></blockquote></div><span><font color="#888888"><br><br clear="all"><span class="HOEnZb"><font color="#888888"><div><br></div>-- <br><div>Simone </div>
</font></span></font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br></font></span></div></div></div>
</blockquote></div><br></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Simone </div>
</div>