[GeoNode-users] Large Capabilties document causes create mapcrash

Fri Dec 1 05:00:51 PST 2017

Hi Paolo, 

No problem! Thanks for your response. 

Do you mean having the capabilities document physically in the file system? Is there an option in GeoServer that can do this?

The problem I see is that we publish layers weekly so there should a batch process creating the file every week I guess.

Thanks.

Regards.

Agur bero bat,

David Alda Fernández de Lezea
Área de Sistemas de Información Geográfica, Planificación Territorial y Forestal Informazio Geografikoen Sistemak, Lurralde eta Baso Antolaketaren Arloa.
dalda at hazi.eus | www.hazi.eus
T 945 003 240 – M 627 923 170 – F 945 003 290 
Hazi | Granja Modelo de Arkaute s/n | 01192 Arkaute – Araba

*********************  LEGE OHARRA   *******************   AVISOLEGAL   *******************   DISCLAIMER   *****************************
Mezu hau pertsonala eta isilpekoa da eta baimenik gabeko erabilera debekatua dago legalki. Jasotzailea ez bazara ezabatu mezua, bidali eta kontserbatu gabe.
Este mensaje es personal y confidencial y su uso no autorizado está prohibido legalmente. Si usted no es el destinatario, proceda a borrarlo, sin reenviarlo ni conservarlo.
This message is personal and confidential, unauthorised use is legally prohibited. If you are not the intended recipient, delete it without resending or backing it.

-----Mensaje original-----
De: Paolo Corti [mailto:pcorti at gmail.com] 
Enviado el: jueves, 30 de noviembre de 2017 23:38
Para: David Alda Fernandez de Lezea
CC: geonode-users at lists.osgeo.org
Asunto: Re: [GeoNode-users] Large Capabilties document causes create mapcrash

Hi David
sorry for late answer.

When requesting the GetCapabilities document, GeoServer is checking GeoFence permissions for each layer. This is not scalable when having a large number of layers, so it can be time consuming.
It would be better to use virtual services to limit the returned number of layers.
If applicable you could cache the GetCapabilities response in your web server. This will have the advantage to return more quickly the GetCapabilities document has GeoServer does not need to resolve it another time (but the document can still be very large), but it will use the permissions of the user which generated the document in the cache.

hope this helps
cheers
p

On Fri, Nov 24, 2017 at 4:47 AM, David Alda Fernandez de Lezea <dalda at hazi.eus> wrote:
> Hi Paolo,
>
> I don't understand you.. What do you mean by " that permissions check slow down the response "?
>
> And "about caching it in your webserver" do you mean caching the capabilities document?
>
> Thanks!
>
> Regards,
>
> Agur bero bat,
>
>
> David Alda Fernández de Lezea
> Área de Sistemas de Información Geográfica, Planificación Territorial y Forestal Informazio Geografikoen Sistemak, Lurralde eta Baso Antolaketaren Arloa.
> dalda at hazi.eus | www.hazi.eus
> T 945 003 240 – M 627 923 170 – F 945 003 290 Hazi | Granja Modelo de 
> Arkaute s/n | 01192 Arkaute – Araba
>
> *********************  LEGE OHARRA   *******************   AVISOLEGAL   *******************   DISCLAIMER   *****************************
> Mezu hau pertsonala eta isilpekoa da eta baimenik gabeko erabilera debekatua dago legalki. Jasotzailea ez bazara ezabatu mezua, bidali eta kontserbatu gabe.
> Este mensaje es personal y confidencial y su uso no autorizado está prohibido legalmente. Si usted no es el destinatario, proceda a borrarlo, sin reenviarlo ni conservarlo.
> This message is personal and confidential, unauthorised use is legally prohibited. If you are not the intended recipient, delete it without resending or backing it.
>
> -----Mensaje original-----
> De: Paolo Corti [mailto:pcorti at gmail.com] Enviado el: martes, 21 de 
> noviembre de 2017 17:33
> Para: David Alda Fernandez de Lezea
> CC: geonode-users at lists.osgeo.org
> Asunto: Re: [GeoNode-users] Large Capabilties document causes create 
> mapcrash
>
> Hi David
>
>>
>> - Connecting to the GeoServer that holds that large number of layers from QGIS works quite fast but If I try to connect to the GeoNode-Geoserver from QGIS I get a time-out exception, so I don't really know if it's something related to the number layers or to the Cascading WMS.
>>
>
> This makes me thinking. It could be that permissions check slow down the response.
> How about caching it in your webserver?
>
> Let us know if this helps
> cheers
> p
>
> On Tue, Nov 21, 2017 at 3:08 AM, David Alda Fernandez de Lezea <dalda at hazi.eus> wrote:
>> Hi Paolo,
>>
>> I forgot to mention that:
>>
>> - The WMSCSOURCE is a cascaded WMS inside my GeoNode-Geoserver
>>
>> - As I mentioned before if I remove the WMSCSOURCE of the local_settings the "Create Map" page works perfect. Then if I add the server via "Add remote server" in the GeoExplorer it loads the capabilities without any problem, it takes its time since they're still around 5000 layers, but it doesn't crash.
>>
>> - Connecting to the GeoServer that holds that large number of layers from QGIS works quite fast but If I try to connect to the GeoNode-Geoserver from QGIS I get a time-out exception, so I don't really know if it's something related to the number layers or to the Cascading WMS.
>>
>> Paolo, I'll answer you below:
>>
>> 1) make sure you are not exposing lots of unneeded projections. --> I've removed unneeded projections, but without success.
>> 2) if applicable, restrict the layers returned by the GetCapabilities using virtual services by filtering on a workspace --> We're already working with Virtual services, perhaps we could improve here by re-ordering / thinking a new architecture.
>>
>> Apart from being a large number of layers and that we can improve that a little bit, I feel that the problem seems to be the cascaded configuration perhaps? Is there anything that I can tune there?
>>
>> Thanks in advance.
>>
>> Regards,
>>
>> Agur bero bat,
>>
>>
>> David Alda Fernández de Lezea
>> Área de Sistemas de Información Geográfica, Planificación Territorial y Forestal Informazio Geografikoen Sistemak, Lurralde eta Baso Antolaketaren Arloa.
>> dalda at hazi.eus | www.hazi.eus
>> T 945 003 240 – M 627 923 170 – F 945 003 290 Hazi | Granja Modelo de 
>> Arkaute s/n | 01192 Arkaute – Araba
>>
>> *********************  LEGE OHARRA   *******************   AVISOLEGAL   *******************   DISCLAIMER   *****************************
>> Mezu hau pertsonala eta isilpekoa da eta baimenik gabeko erabilera debekatua dago legalki. Jasotzailea ez bazara ezabatu mezua, bidali eta kontserbatu gabe.
>> Este mensaje es personal y confidencial y su uso no autorizado está prohibido legalmente. Si usted no es el destinatario, proceda a borrarlo, sin reenviarlo ni conservarlo.
>> This message is personal and confidential, unauthorised use is legally prohibited. If you are not the intended recipient, delete it without resending or backing it.
>>
>>
>> -----Mensaje original-----
>> De: Paolo Corti [mailto:pcorti at gmail.com] Enviado el: lunes, 20 de 
>> noviembre de 2017 18:08
>> Para: David Alda Fernandez de Lezea
>> CC: geonode-users at lists.osgeo.org
>> Asunto: Re: [GeoNode-users] Large Capabilties document causes create 
>> mapcrash
>>
>> Hi David
>>
>> unfortunately that is not going to scale well.
>>
>> Any chance you can restrict the size of the document? Two things are worth to mention:
>> 1) make sure you are not exposing lots of unneeded projections.
>> Sometime this is causing the document to grow pretty much!
>> 2) if applicable, restrict the layers returned by the GetCapabilities 
>> using virtual services by filtering on a workspace
>>
>> regards
>> p
>>
>> On Mon, Nov 20, 2017 at 4:38 AM, David Alda Fernandez de Lezea <dalda at hazi.eus> wrote:
>>> Hi ,
>>>
>>> I'm facing a problem with a "WMSCSOURCE"... In my local_config.py file I've added an WMSCSOURCE pointing to another geoserver we have in our organization. That geoserver is handling right now 5000 layers approximately, the thing is that as the number of layers is growing the load time of the "Create Map" page has slowed down in the same rate causing finally a "memory crash" in the browser...
>>>
>>> I've tried disabling that WMSCSOURCE and the "Create Map" page works perfect, so the problem is in the process of loading that capabilities document.
>>>
>>> The capabilities document is about 12,5 MB size
>>>
>>> I'm working on 2.4 version.
>>>
>>> Any ideas where can I improve the load time or something?
>>>
>>> Thanks in advance.
>>>
>>> Regards,
>>>
>>> Agur bero bat,
>>>
>>>
>>> David Alda Fernández de Lezea
>>> Área de Sistemas de Información Geográfica, Planificación Territorial y Forestal Informazio Geografikoen Sistemak, Lurralde eta Baso Antolaketaren Arloa.
>>> dalda at hazi.eus | www.hazi.eus
>>> T 945 003 240 - M 627 923 170 - F 945 003 290 Hazi | Granja Modelo 
>>> de Arkaute s/n | 01192 Arkaute - Araba
>>>
>>> *********************  LEGE OHARRA   *******************   AVISOLEGAL   *******************   DISCLAIMER   *****************************
>>> Mezu hau pertsonala eta isilpekoa da eta baimenik gabeko erabilera debekatua dago legalki. Jasotzailea ez bazara ezabatu mezua, bidali eta kontserbatu gabe.
>>> Este mensaje es personal y confidencial y su uso no autorizado está prohibido legalmente. Si usted no es el destinatario, proceda a borrarlo, sin reenviarlo ni conservarlo.
>>> This message is personal and confidential, unauthorised use is legally prohibited. If you are not the intended recipient, delete it without resending or backing it.
>>>
>>> _______________________________________________
>>> geonode-users mailing list
>>> geonode-users at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/geonode-users
>>
>>
>>
>> --
>> Paolo Corti
>> Geospatial software developer
>> web: http://www.paolocorti.net
>> twitter: @capooti
>> skype: capooti
>
>
>
> --
> Paolo Corti
> Geospatial software developer
> web: http://www.paolocorti.net
> twitter: @capooti
> skype: capooti

--
Paolo Corti
Geospatial software developer
web: http://www.paolocorti.net
twitter: @capooti
skype: capooti