[mapguide-internals] Exception Handling for Memory limitations + Patch for #480 & #520

[UV]

Hi there,

I am trying to make the mgserver behave in the case of limited memory..
We are using a massive map for our client and the server keeps on 
falling over. (did not reproduce in 2.0.2)

I followed the trace from StylizeLayerinto the SimpleSDFReader which 
fails with a (ret != SQLiteDB_OK)

This interface to the underlying SQLiteTable::get call does not do 
exception handling so we have lost the underlying reason.....
I assume its a failed malloc somewhere as I am investigating out of 
memory behaviour.

Furthermore the MappingUtil::StylizeLayers method simply eats up the 
exception.

I am not sure if this is really the desired behaviour. Maybe this should 
be examinated in more detail.
I am just slowly getting fed up reading a lot of code that has clearly 
not been written to be understood by other people.
So other approaches are needed.

== Smart Handling of Out of Memory Exceptions ==

I suggest to wrap all memory allocation deeper in the code into retry 
loops with a time delay doubling at each repetition (like CSMA/CD).
Another option used in my ByteSink patch simply decreases the allocated 
buffer as it can in this particular case.

This should increase server stability massively as memory is used and 
returned a lot in this multi-threaded server.
This should provide a way for the halted threads to get the memory they 
need at a later point in time.
I think waiting is a realistic engeering approach to solve the OoM 
problem.  (I learned this style while coding on a realtime audio engine :)

== Patch for #480 #520 ==

The patch cleans up the file handle and removes empty files. Furthermore 
an intelligent exception handler uses a buffer of an available size if 
1MB is not available anymore.
(1MB looks like a quite a hefty estimation of the needed size in the 
first place.....)

-------------- next part --------------
Index: ByteSink.cpp
===================================================================
--- ByteSink.cpp	(revision 3778)
+++ ByteSink.cpp	(working copy)
@@ -188,7 +188,7 @@
 void MgByteSink::ToFile(CREFSTRING filename)
 {
     unsigned char* bytes = 0;
-
+	FILE* f = 0;	// needs to be in top scope to be able to clean it up in exception handler
     MG_TRY()
 
     // See if we are sourced from a temporary file byte source.  If we are, just
@@ -202,7 +202,7 @@
     else
     {   // Do it the hard way.  Create a new file and write it a block at a time
         ///Use ACE_OS::fopen() for Linux compatibility
-        FILE* f = ACE_OS::fopen(MG_WCHAR_TO_TCHAR(filename), ACE_TEXT("wb"));
+        f = ACE_OS::fopen(MG_WCHAR_TO_TCHAR(filename), ACE_TEXT("wb"));
 
         if (f == NULL)
         {
@@ -221,11 +221,23 @@
             }
         }
 
-        bytes = new unsigned char[BSINK_BUFFER_SIZE];
+		// allocate a huge buffer (usually 1MB) at first
+		int bSinkBufferSize = BSINK_BUFFER_SIZE;
+		do 
+		{	// retry allocation with half the size until we give up at 16kB
+			try {
+				bytes = new unsigned char[bSinkBufferSize];
+			} catch (exception& e) {
+				if (typeid(e) == typeid(bad_alloc) && bSinkBufferSize >= 1024*16)
+					bSinkBufferSize >>= 1;
+				else
+					throw e;
+			}
+		} while (bytes == 0);
 
         size_t bytesReceived, bytesWritten;
 
-        while ((bytesReceived = (size_t)m_reader->Read(bytes, BSINK_BUFFER_SIZE)) > 0)
+        while ((bytesReceived = (size_t)m_reader->Read(bytes, bSinkBufferSize)) > 0)
         {
             if((bytesWritten = ACE_OS::fwrite(bytes, 1, bytesReceived, f)) != bytesReceived)
             {
@@ -245,6 +257,14 @@
     MG_CATCH(L"MgByteSink::ToFile")
 
     delete [] bytes;
+	if (NULL != f)	ACE_OS::fclose(f);	// close the file if not done yet
 
-    MG_THROW()
+	if (NULL != mgException && ) {
+		struct _stat statInfo;
+		MgFileUtil::GetFileStatus(MG_WCHAR_TO_TCHAR(filename), statInfo);
+		// delete only if empty... we dont want to destroy files with useful content ....
+		if (statInfo.st_size == 0)
+			MgFileUtil::DeleteFile(MG_WCHAR_TO_TCHAR(filename));
+	}
+	MG_THROW( )
 }