[fdo-internals] bad behavior in low memory situation verified on yesterday build from build server #480,

[UV]

Thanks for the input!
Any information regarding such component internals is greatly appreciated.

So far not understanding too much of the details I just made a wild 
guess as my test case opens
a 250MB SDF file after which the error occurs.. somewhere... so 
depending on the way it is implemented
this could use more or less memory.

So if you suggest that the reader does not allocate lots of memory 
except for the serialization in mapguide
we might have a good candidate to start looking for things. Thanks

Cheers
UV

PS We dont use KML in our map but its good to know that this is a memory 
hungry component :)

Haris Kurtagic wrote:
> I don't think we can do much in FDO. It is about providers what they do
> with memory.
> I doubt that SDF provider is using lot of memory regardless of size of
> SDF file. 
> I believe, the worst provider regarding memory usage could be KML
> provider. It loads whole KML into memory. Although once loaded, FDO
> readers created after do not use additional memory.
>
> Where exactly you think is problems with memory allocation. Perhaps it
> is in MG when serializing FDO reader, rather in reader itself ?
>
> Haris
>
> -----Original Message-----
> From: fdo-internals-bounces at lists.osgeo.org
> [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of UV
> Sent: Monday, April 20, 2009 6:50 AM
> To: MapGuide Internals Mail List; FDO Internals Mail List
> Subject: [fdo-internals] bad behaviour in low memory situation verified
> on yesterday build from build server #480,
>
> To verify the low memory behaviour of the server I was rendering the 
> tiles of our big map
> on a 1GB windows server machine using a most recent build from the build
>
> server.
>
> I can observer a relative consistent behaviour now....
> First  the FDO reader falls over trying to load a large file. (SDF 
> database access)
> After this happened a few times, the MappingService fails to allocate 
> memory itself.
> In the meanwhile SUCCESS gets returned for wrong tiles.
>
> I think the OoM exception inside FDO does not get propagated to MapGuide
>
> correctly nor handled in a useful manner.
> I remember seeing some exception free code under the sdf reader....
> Similarly the OoM exception does not get handled in a useful manner in 
> MapGuide.
>
> Generally, the behaviour of the server when memory runs out is now 
> almost worse than before because the tileserver returns SUCCESS for 
> wrongly rendered tiles instead of INVALID STREAM HEADER.
> I am happy to help but I need help.
> I supplied the retry routine for the ByteSink which was a step forward 
> in stability,
> But the underlying problem of this multi-threaded server when it comes 
> to memory shortage IS NOT SOLVED.
>
> [Approach1]
> A wait retry loop around each memory allocation to catch out of memory 
> exception .... this should serialize memory requests sufficiently to 
> make things stable. We could create a clever macro for all memory 
> allocation which would do exactly this.
> Also the allocator base class for std library containers can be used for
>
> that. I think an important place would be the FDO readers which open the
>
> few hundred MB files. There is no point in doing that in MapGuide only.
> This could be done by propagating more meaningful errors from FDO to 
> mapguide so this can be dealt with correctly (recognize OoM and attempt 
> retry) Black boxing FDO and having only a general exception there seems 
> the wrong way to deal with such problems.
>
> This is a lot of work but I assume that doing that for the few memory 
> hungry parts will sort most problems. (see also ByteSink)
>
> [Approach2]
> The minimal useful workaround would be to have the server return an 
> error message as a HTTP response or on the tile without creating the 
> tile in the cache! This way the client layer can attempt to retry later.
>
> # Log Type: Error Log
> # Log Parameters: CLIENT,CLIENTIP,USER,ERROR,STACKTRACE
> <2009-04-20T13:30:01>     2088            
>  Success: Server started.    
> <2009-04-20T14:04:41>     5212        192.168.99.99    Anonymous
>  Error: Failed to stylize layer: aug 06-ROADS - HWY
>         Cannot establish connection.
>  StackTrace:
>   - MgMappingUtil.StylizeLayers line 763 file 
> c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp
>
> <2009-04-20T14:07:28>     5540        192.168.99.99    Anonymous
>  Error: Failed to stylize layer: aug 06-ROADS - FWY
>         An exception occurred in FDO component.
>         An exception occurred in FDO component.
>         An error occurred during SDF database access.  
>  StackTrace:
>   - MgMappingUtil.StylizeLayers line 763 file 
> c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp
>
> <2009-04-20T14:07:29>     1768        192.168.99.99    Anonymous
>  Error: Failed to stylize layer: aug 06-ROADS - MAJ
>         An exception occurred in FDO component.
>         An exception occurred in FDO component.
>         An error occurred during SDF database access.  
>  StackTrace:
>   - MgMappingUtil.StylizeLayers line 763 file 
> c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp
>
> <2009-04-20T14:23:54>     3380        192.168.99.99    Anonymous
>  Error: Failed to stylize layer: Feb 08-ABORIG_WA
>         Out of memory.
>         bad allocation
>  StackTrace:
>   - MgMappingUtil.StylizeLayers line 763 file 
> c:\ci\mapguide\build\mgdev\server\src\services\mapping\MappingUtil.cpp
>
> .....
> _______________________________________________
> fdo-internals mailing list
> fdo-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/fdo-internals
> _______________________________________________
> fdo-internals mailing list
> fdo-internals at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/fdo-internals
>
>