[GeoNode-devel] download original datasets feature randomly fails

Thu May 23 09:22:41 PDT 2019

Sure I will do this.
Paolo

On Thu, May 23, 2019 at 3:17 AM Alessio Fabiani
<alessio.fabiani at geo-solutions.it> wrote:
>
> Hi Paolo,
> yes, sure, it is feasible.
> Do you have some resources to work on that?
>
> Il giorno mer 22 mag 2019 alle ore 22:15 Paolo Corti <pcorti at gmail.com> ha scritto:
>>
>> Hi Alessio
>> I guess that would be feasible with the management command. I wouldn't
>> do that when saving the metadata as it can slow down things.
>> How about having a setting which disable the creation of the "Original
>> Dataset" link?
>> thanks for helping with this
>> Paolo
>>
>> On Mon, May 20, 2019 at 4:24 AM Alessio Fabiani
>> <alessio.fabiani at geo-solutions.it> wrote:
>> >
>> > Hello Paolo,
>> > yes it was tested... not recently to be honest.
>> > Maybe we can improve it in order to look first for the original dataset and advertise the user suddenly.
>> > Moreover in the latest versions the management command "set_all_layers_metadata" checks and fixes the links, and refreshes the metadata accordingly.
>> > We could envisage the code creating "Original Dataset" links to check for the original resource before creating it.
>> >
>> > Il giorno ven 17 mag 2019 alle ore 21:49 Paolo Corti <pcorti at gmail.com> ha scritto:
>> >>
>> >> Dear Alessio
>> >> thanks a lot for the detailed explanation.
>> >> The reason in our instances could be that for many layers we haven't
>> >> anymore the original shapefile but just the data in PostGIS.
>> >> Which means that the best option for us is to disable that link.
>> >> ps: by the way, was this tested both with rest and the importer
>> >> uploader? (we are using the latter)
>> >> Have a great weekend
>> >> Paolo
>> >>
>> >> On Fri, May 17, 2019 at 3:47 AM Alessio Fabiani
>> >> <alessio.fabiani at geo-solutions.it> wrote:
>> >> >
>> >> > Hello Paolo,
>> >> > don't know exactly why is failing in your case, it is usually working, at least on my experience.
>> >> >
>> >> > However, the idea is the following.
>> >> > All the download methods exposed on GeoNode, relies on OGC calls. For vectors WFS, raster WCS and other formats WMS.
>> >> > That means that you overload GeoServer and potentially, especially for huge datasets, a single download can stuck your server by consuming all the machine resources available.
>> >> > Notice also that GeoServer always does a format conversion when you ask for a raw data format.
>> >> >
>> >> > The latest versions of GeoServer have some protection mechanisms allowing you to configure the cap of resources to be consumed.
>> >> > That means that for big datasets the download won't be allowed at all.
>> >> >
>> >> > The intention of "Original Dataset" download link, is to allow GeoNode to package the uploaded files into the filesystem along with all the SLDs defined and metadata formats, and send back to the user
>> >> > without making OGC requests.
>> >> > Of course this will successfully work only if:
>> >> >
>> >> > 1. The original uploaded file still exists and it is accessible on the filesystem
>> >> > 2. The layer is not a pure table on a DB but has a file associated with it
>> >> >
>> >> > Other than this, and this could be the cause why it is failing in your case, the "upload session" metadata stored on GeoNode database should point to the correct original files uploaded into GeoNode.
>> >> >
>> >> > That said, if you have the chance to investigate a bit more on the cause of the failures, we can try to make it a bit more reliable.
>> >> >
>> >> > Cheers,
>> >> > A.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > Il giorno ven 17 mag 2019 alle ore 00:41 Paolo Corti <pcorti at gmail.com> ha scritto:
>> >> >>
>> >> >> Hi devs
>> >> >> when downloading a layer, "Zipped Shapefile" works as expected, while
>> >> >> "Original Datasets" randomly fails. Also, I am not sure what is the
>> >> >> latter for: when I am able to download it successfully, I see a zip
>> >> >> file containing a bunch of .sld and geojson file.
>> >> >> Would be glad to learn more about this
>> >> >> thanks
>> >> >> Paolo
>> >> >>
>> >> >> --
>> >> >> Paolo Corti
>> >> >> Geospatial software developer
>> >> >> web: http://www.paolocorti.net
>> >> >> twitter: @capooti
>> >> >> skype: capooti
>> >> >> #drt3jc1
>> >> >> _______________________________________________
>> >> >> geonode-devel mailing list
>> >> >> geonode-devel at lists.osgeo.org
>> >> >> https://lists.osgeo.org/mailman/listinfo/geonode-devel
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> >
>> >> > ==
>> >> >
>> >> > GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
>> >> > ==
>> >> > Ing. Alessio Fabiani
>> >> >
>> >> > @alfa7691
>> >> > Founder/Technical Lead
>> >> >
>> >> >
>> >> > GeoSolutions S.A.S.
>> >> > Via di Montramito 3/A - 55054  Massarosa (LU) - Italy
>> >> > phone: +39 0584 962313
>> >> > fax:     +39 0584 1660272
>> >> > mob:   +39 331 6233686
>> >> >
>> >> >
>> >> > http://www.geo-solutions.it
>> >> > http://twitter.com/geosolutions_it
>> >> > -------------------------------------------------------
>> >> >
>> >> > Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.
>> >> >
>> >> >
>> >> > This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.
>> >>
>> >>
>> >>
>> >> --
>> >> Paolo Corti
>> >> Geospatial software developer
>> >> web: http://www.paolocorti.net
>> >> twitter: @capooti
>> >> skype: capooti
>> >> #drt3jc1
>> >
>> >
>> >
>> > --
>> >
>> > ==
>> >
>> > GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
>> > ==
>> > Ing. Alessio Fabiani
>> >
>> > @alfa7691
>> > Founder/Technical Lead
>> >
>> >
>> > GeoSolutions S.A.S.
>> > Via di Montramito 3/A - 55054  Massarosa (LU) - Italy
>> > phone: +39 0584 962313
>> > fax:     +39 0584 1660272
>> > mob:   +39 331 6233686
>> >
>> >
>> > http://www.geo-solutions.it
>> > http://twitter.com/geosolutions_it
>> > -------------------------------------------------------
>> >
>> > Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.
>> >
>> >
>> > This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.
>>
>>
>>
>> --
>> Paolo Corti
>> Geospatial software developer
>> web: http://www.paolocorti.net
>> twitter: @capooti
>> skype: capooti
>> #drt3jc1
>
>
>
> --
>
> ==
>
> GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
> ==
> Ing. Alessio Fabiani
>
> @alfa7691
> Founder/Technical Lead
>
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A - 55054  Massarosa (LU) - Italy
> phone: +39 0584 962313
> fax:     +39 0584 1660272
> mob:   +39 331 6233686
>
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
> -------------------------------------------------------
>
> Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.
>
>
> This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

-- 
Paolo Corti
Geospatial software developer
web: http://www.paolocorti.net
twitter: @capooti
skype: capooti
#drt3jc1