[fdo-internals] PATCH: fixes for WFS memory problems on Windows

[Robin Newton]

Hi,

I hope this is OK, but I've attached two patches for memory problems 
seen when using the WFS provider on Windows to read large datasets. The 
product where I've been seeing these problems currently uses FDO 3.2, so 
that's where I've been doing my investigation, but the affected bits of 
code have barely changed between 3.2 and the current trunk, so I believe 
the problems would still be present, and the patches would still apply.

One of the problems is that memory usage steadily increases when reading 
features from WFS and, depending on the machine being used and the 
amount of data being read, this can cause an application to run out of 
memory. The reason is that FdoXmlFeatureReaderImpl builds up a 
collection of all the features it has seen, not releasing them until it 
is destroyed. My fix for this is in FeatureReaderImpl_patch.txt.

As far as I get tell, there is nothing that would ever access elements 
in m_featureCollection prior to the current feature, and 
m_curFeatureIndex never goes backwards, so I think this is a safe change.

The other problem is a bit like that in ticket 732: reading a feature 
with many coordinates can cause an application to terminate on Windows 
due to a stack overflow. This is less important to me than the first 
problem, as I have a work-around for it.

The problem here is that FdoXmlUtilXrcs::Xrcs2Unicode can allocate up to 
512K of stack space - whereas e.g. a Java application running on 32-bit 
Windows will by default a 320K stack. (The work-around is to change the 
stack size, which is doable but inconvenient, and potentially wastes 
space in heavily multi-threaded applications.) I assume that there are 
performance reasons for not troubling the heap allocator with lots of 
small, fiddly memory requests; but I think the limit for stack 
allocation could reasonably be reduced, so in UtilXrcs_patch.txt I've 
suggested 64K.

Thanks,
Robin Newton


Keep up to date with the latest 1Spatial news and events

1Spatial: Steering you towards efficient and reliable location-based data with the 1Spatial Data Improvement Process

http://www.1spatial.com/improve_your_data/

1Spatial Group Limited; Registered in England No. 4785688 VAT Reg. No. 816329821; Registered Office: Tennyson House; Cambridge Business Park; Cambridge; CB4 0WZ; United Kingdom

IMPORTANT NOTICE
 
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender immediately and delete this e-mail from your system. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of 1Spatial Group Limited, its subsidiaries or associated companies, except where the author specifically states them to be the views of 1Spatial Group Limited, its subsidiaries or associated companies.
 
1Spatial Group Limited, its subsidiaries and associated companies will not be held liable for any legally binding obligations that are not the subject of an official 1Spatial purchase order or as part of a contract signed by a director of one of the aforementioned companies.
 
Although the company has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. You should understand and accept that, when communicating with us by e-mail, it is not a totally secure communications medium.

-------------- next part --------------
Index: Fdo/Unmanaged/Src/Common/Xml/UtilXrcs.cpp
===================================================================
--- Fdo/Unmanaged/Src/Common/Xml/UtilXrcs.cpp	(revision 6214)
+++ Fdo/Unmanaged/Src/Common/Xml/UtilXrcs.cpp	(working copy)
@@ -43,7 +43,7 @@
         outString = (wchar_t*) chars;  // fast assignment step (no conversion)
     else
     {
-		bool inHeap = sizeof(wchar_t)*(length+1) > 512 * 1024 ? true : false;
+		bool inHeap = sizeof(wchar_t)*(length+1) > 64 * 1024 ? true : false;
         wchar_t* target;
 		if (inHeap)
 			target = (wchar_t*)new unsigned char[sizeof(wchar_t)*(length+1)];
-------------- next part --------------
Index: Fdo/Unmanaged/Src/Fdo/Xml/FeatureReaderImpl.cpp
===================================================================
--- Fdo/Unmanaged/Src/Fdo/Xml/FeatureReaderImpl.cpp	(revision 6214)
+++ Fdo/Unmanaged/Src/Fdo/Xml/FeatureReaderImpl.cpp	(working copy)
@@ -96,6 +96,16 @@
             m_featurePropertyReader->SetFeatureSchemas(m_schemas);
         }
 		
+		// if we are reading a new feature, release old ones
+		if (m_curFeatureIndex >= (int)m_featureCollection.size())
+		{
+			while(!m_featureCollection.empty()) {
+				FDO_SAFE_RELEASE(m_featureCollection.back());
+				m_featureCollection.pop_back();
+				m_curFeatureIndex --;
+			}
+		}
+
 		m_featurePropertyReader->Parse(this, NULL, m_incrementalParsing);
 	}