[fdo-internals] bad behaviour in low memory situation verified on yesterday build from build server #480,

[UV]

To verify the low memory behaviour of the server I was rendering the 
tiles of our big map
on a 1GB windows server machine using a most recent build from the build 
server.

I can observer a relative consistent behaviour now....
First  the FDO reader falls over trying to load a large file. (SDF 
database access)
After this happened a few times, the MappingService fails to allocate 
memory itself.
In the meanwhile SUCCESS gets returned for wrong tiles.

I think the OoM exception inside FDO does not get propagated to MapGuide 
correctly nor handled in a useful manner.
I remember seeing some exception free code under the sdf reader....
Similarly the OoM exception does not get handled in a useful manner in 
MapGuide.

Generally, the behaviour of the server when memory runs out is now 
almost worse than before because the tileserver returns SUCCESS for 
wrongly rendered tiles instead of INVALID STREAM HEADER.
I am happy to help but I need help.
I supplied the retry routine for the ByteSink which was a step forward 
in stability,
But the underlying problem of this multi-threaded server when it comes 
to memory shortage IS NOT SOLVED.

[Approach1]
A wait retry loop around each memory allocation to catch out of memory 
exception .... this should serialize memory requests sufficiently to 
make things stable. We could create a clever macro for all memory 
allocation which would do exactly this.
Also the allocator base class for std library containers can be used for 
that. I think an important place would be the FDO readers which open the 
few hundred MB files. There is no point in doing that in MapGuide only.
This could be done by propagating more meaningful errors from FDO to 
mapguide so this can be dealt with correctly (recognize OoM and attempt 
retry) Black boxing FDO and having only a general exception there seems 
the wrong way to deal with such problems.

This is a lot of work but I assume that doing that for the few memory 
hungry parts will sort most problems. (see also ByteSink)

[Approach2]
The minimal useful workaround would be to have the server return an 
error message as a HTTP response or on the tile without creating the 
tile in the cache! This way the client layer can attempt to retry later.

# Log Type: Error Log
# Log Parameters: CLIENT,CLIENTIP,USER,ERROR,STACKTRACE
<2009-04-20T13:30:01>     2088            
 Success: Server started.    
<2009-04-20T14:04:41>     5212        192.168.99.99    Anonymous
 Error: Failed to stylize layer: aug 06-ROADS - HWY
        Cannot establish connection.
 StackTrace:
  - MgMappingUtil.StylizeLayers line 763 file 
c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp    
<2009-04-20T14:07:28>     5540        192.168.99.99    Anonymous
 Error: Failed to stylize layer: aug 06-ROADS - FWY
        An exception occurred in FDO component.
        An exception occurred in FDO component.
        An error occurred during SDF database access.  
 StackTrace:
  - MgMappingUtil.StylizeLayers line 763 file 
c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp    
<2009-04-20T14:07:29>     1768        192.168.99.99    Anonymous
 Error: Failed to stylize layer: aug 06-ROADS - MAJ
        An exception occurred in FDO component.
        An exception occurred in FDO component.
        An error occurred during SDF database access.  
 StackTrace:
  - MgMappingUtil.StylizeLayers line 763 file 
c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp    
<2009-04-20T14:23:54>     3380        192.168.99.99    Anonymous
 Error: Failed to stylize layer: Feb 08-ABORIG_WA
        Out of memory.
        bad allocation
 StackTrace:
  - MgMappingUtil.StylizeLayers line 763 file 
c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp   
.....