[fdo-internals] RE: [mapguide-internals] bad behaviour in low memory situation verified on yesterday build from build server #480,

[Trevor Wekel]

I suspect we are looking at a memory leak somewhere in the MapGuide code base.  I am currently investigating memory leaks in MapGuide and may know more in a couple of days.

UV, were you using a strictly SDF based map?  How many tiles were generated before you started running into memory issues? How many layers were on the map?  Were you using any of the advanced stylization?  I'm just trying to scope the problem a bit.

Thanks,
Trevor

________________________________________
From: mapguide-internals-bounces at lists.osgeo.org [mapguide-internals-bounces at lists.osgeo.org] On Behalf Of UV [uvwild at gmail.com]
Sent: Sunday, April 19, 2009 10:50 PM
To: MapGuide Internals Mail List; FDO Internals Mail List
Subject: [mapguide-internals] bad behaviour in low memory situation verified on yesterday build from build server  #480,

To verify the low memory behaviour of the server I was rendering the
tiles of our big map
on a 1GB windows server machine using a most recent build from the build
server.

I can observer a relative consistent behaviour now....
First  the FDO reader falls over trying to load a large file. (SDF
database access)
After this happened a few times, the MappingService fails to allocate
memory itself.
In the meanwhile SUCCESS gets returned for wrong tiles.

I think the OoM exception inside FDO does not get propagated to MapGuide
correctly nor handled in a useful manner.
I remember seeing some exception free code under the sdf reader....
Similarly the OoM exception does not get handled in a useful manner in
MapGuide.

Generally, the behaviour of the server when memory runs out is now
almost worse than before because the tileserver returns SUCCESS for
wrongly rendered tiles instead of INVALID STREAM HEADER.
I am happy to help but I need help.
I supplied the retry routine for the ByteSink which was a step forward
in stability,
But the underlying problem of this multi-threaded server when it comes
to memory shortage IS NOT SOLVED.

[Approach1]
A wait retry loop around each memory allocation to catch out of memory
exception .... this should serialize memory requests sufficiently to
make things stable. We could create a clever macro for all memory
allocation which would do exactly this.
Also the allocator base class for std library containers can be used for
that. I think an important place would be the FDO readers which open the
few hundred MB files. There is no point in doing that in MapGuide only.
This could be done by propagating more meaningful errors from FDO to
mapguide so this can be dealt with correctly (recognize OoM and attempt
retry) Black boxing FDO and having only a general exception there seems
the wrong way to deal with such problems.

This is a lot of work but I assume that doing that for the few memory
hungry parts will sort most problems. (see also ByteSink)

[Approach2]
The minimal useful workaround would be to have the server return an
error message as a HTTP response or on the tile without creating the
tile in the cache! This way the client layer can attempt to retry later.

# Log Type: Error Log
# Log Parameters: CLIENT,CLIENTIP,USER,ERROR,STACKTRACE
<2009-04-20T13:30:01>     2088
 Success: Server started.
<2009-04-20T14:04:41>     5212        192.168.99.99    Anonymous
 Error: Failed to stylize layer: aug 06-ROADS - HWY
        Cannot establish connection.
 StackTrace:
  - MgMappingUtil.StylizeLayers line 763 file
c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp
<2009-04-20T14:07:28>     5540        192.168.99.99    Anonymous
 Error: Failed to stylize layer: aug 06-ROADS - FWY
        An exception occurred in FDO component.
        An exception occurred in FDO component.
        An error occurred during SDF database access.
 StackTrace:
  - MgMappingUtil.StylizeLayers line 763 file
c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp
<2009-04-20T14:07:29>     1768        192.168.99.99    Anonymous
 Error: Failed to stylize layer: aug 06-ROADS - MAJ
        An exception occurred in FDO component.
        An exception occurred in FDO component.
        An error occurred during SDF database access.
 StackTrace:
  - MgMappingUtil.StylizeLayers line 763 file
c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp
<2009-04-20T14:23:54>     3380        192.168.99.99    Anonymous
 Error: Failed to stylize layer: Feb 08-ABORIG_WA
        Out of memory.
        bad allocation
 StackTrace:
  - MgMappingUtil.StylizeLayers line 763 file
c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp
.....
_______________________________________________
mapguide-internals mailing list
mapguide-internals at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapguide-internals