[fdo-internals] bad behavior in low memory situation verified on yesterday build from build server #480,

[Dan Stoica]

There is memory leak in the Expression Engine (see Ticket #474) which has been fixed in the Trunk.
Practically all the geometries returned by a spatial query were leaking. This might explain the bad behavior.

And for the fix itself there is a better solution (more consistent with the EE implementation) suggested by Romica. 

Dan.


-----Original Message-----
From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Haris Kurtagic
Sent: Monday, April 20, 2009 4:01 AM
To: FDO Internals Mail List
Subject: RE: [fdo-internals] bad behavior in low memory situation verified on yesterday build from build server #480, 

I don't think we can do much in FDO. It is about providers what they do
with memory.
I doubt that SDF provider is using lot of memory regardless of size of
SDF file. 
I believe, the worst provider regarding memory usage could be KML
provider. It loads whole KML into memory. Although once loaded, FDO
readers created after do not use additional memory.

Where exactly you think is problems with memory allocation. Perhaps it
is in MG when serializing FDO reader, rather in reader itself ?

Haris

-----Original Message-----
From: fdo-internals-bounces at lists.osgeo.org
[mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of UV
Sent: Monday, April 20, 2009 6:50 AM
To: MapGuide Internals Mail List; FDO Internals Mail List
Subject: [fdo-internals] bad behaviour in low memory situation verified
on yesterday build from build server #480,

To verify the low memory behaviour of the server I was rendering the 
tiles of our big map
on a 1GB windows server machine using a most recent build from the build

server.

I can observer a relative consistent behaviour now....
First  the FDO reader falls over trying to load a large file. (SDF 
database access)
After this happened a few times, the MappingService fails to allocate 
memory itself.
In the meanwhile SUCCESS gets returned for wrong tiles.

I think the OoM exception inside FDO does not get propagated to MapGuide

correctly nor handled in a useful manner.
I remember seeing some exception free code under the sdf reader....
Similarly the OoM exception does not get handled in a useful manner in 
MapGuide.

Generally, the behaviour of the server when memory runs out is now 
almost worse than before because the tileserver returns SUCCESS for 
wrongly rendered tiles instead of INVALID STREAM HEADER.
I am happy to help but I need help.
I supplied the retry routine for the ByteSink which was a step forward 
in stability,
But the underlying problem of this multi-threaded server when it comes 
to memory shortage IS NOT SOLVED.

[Approach1]
A wait retry loop around each memory allocation to catch out of memory 
exception .... this should serialize memory requests sufficiently to 
make things stable. We could create a clever macro for all memory 
allocation which would do exactly this.
Also the allocator base class for std library containers can be used for

that. I think an important place would be the FDO readers which open the

few hundred MB files. There is no point in doing that in MapGuide only.
This could be done by propagating more meaningful errors from FDO to 
mapguide so this can be dealt with correctly (recognize OoM and attempt 
retry) Black boxing FDO and having only a general exception there seems 
the wrong way to deal with such problems.

This is a lot of work but I assume that doing that for the few memory 
hungry parts will sort most problems. (see also ByteSink)

[Approach2]
The minimal useful workaround would be to have the server return an 
error message as a HTTP response or on the tile without creating the 
tile in the cache! This way the client layer can attempt to retry later.

# Log Type: Error Log
# Log Parameters: CLIENT,CLIENTIP,USER,ERROR,STACKTRACE
<2009-04-20T13:30:01>     2088            
 Success: Server started.    
<2009-04-20T14:04:41>     5212        192.168.99.99    Anonymous
 Error: Failed to stylize layer: aug 06-ROADS - HWY
        Cannot establish connection.
 StackTrace:
  - MgMappingUtil.StylizeLayers line 763 file 
c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp

<2009-04-20T14:07:28>     5540        192.168.99.99    Anonymous
 Error: Failed to stylize layer: aug 06-ROADS - FWY
        An exception occurred in FDO component.
        An exception occurred in FDO component.
        An error occurred during SDF database access.  
 StackTrace:
  - MgMappingUtil.StylizeLayers line 763 file 
c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp

<2009-04-20T14:07:29>     1768        192.168.99.99    Anonymous
 Error: Failed to stylize layer: aug 06-ROADS - MAJ
        An exception occurred in FDO component.
        An exception occurred in FDO component.
        An error occurred during SDF database access.  
 StackTrace:
  - MgMappingUtil.StylizeLayers line 763 file 
c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp

<2009-04-20T14:23:54>     3380        192.168.99.99    Anonymous
 Error: Failed to stylize layer: Feb 08-ABORIG_WA
        Out of memory.
        bad allocation
 StackTrace:
  - MgMappingUtil.StylizeLayers line 763 file 
c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp

.....
_______________________________________________
fdo-internals mailing list
fdo-internals at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/fdo-internals
_______________________________________________
fdo-internals mailing list
fdo-internals at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/fdo-internals