[fdo-internals] PATCH: fixes for WFS memory problems on Windows

[Greg Boone]

OK. Your changes look good. I will submit them in the next day or so.

Greg

-----Original Message-----
From: fdo-internals-bounces at lists.osgeo.org [mailto:fdo-internals-bounces at lists.osgeo.org] On Behalf Of Robin Newton
Sent: Wednesday, August 24, 2011 2:45 PM
To: fdo-internals at lists.osgeo.org
Subject: [fdo-internals] PATCH: fixes for WFS memory problems on Windows

Hi,

I hope this is OK, but I've attached two patches for memory problems seen when using the WFS provider on Windows to read large datasets. The product where I've been seeing these problems currently uses FDO 3.2, so that's where I've been doing my investigation, but the affected bits of code have barely changed between 3.2 and the current trunk, so I believe the problems would still be present, and the patches would still apply.

One of the problems is that memory usage steadily increases when reading features from WFS and, depending on the machine being used and the amount of data being read, this can cause an application to run out of memory. The reason is that FdoXmlFeatureReaderImpl builds up a collection of all the features it has seen, not releasing them until it is destroyed. My fix for this is in FeatureReaderImpl_patch.txt.

As far as I get tell, there is nothing that would ever access elements in m_featureCollection prior to the current feature, and m_curFeatureIndex never goes backwards, so I think this is a safe change.

The other problem is a bit like that in ticket 732: reading a feature with many coordinates can cause an application to terminate on Windows due to a stack overflow. This is less important to me than the first problem, as I have a work-around for it.

The problem here is that FdoXmlUtilXrcs::Xrcs2Unicode can allocate up to 512K of stack space - whereas e.g. a Java application running on 32-bit Windows will by default a 320K stack. (The work-around is to change the stack size, which is doable but inconvenient, and potentially wastes space in heavily multi-threaded applications.) I assume that there are performance reasons for not troubling the heap allocator with lots of small, fiddly memory requests; but I think the limit for stack allocation could reasonably be reduced, so in UtilXrcs_patch.txt I've suggested 64K.

Thanks,
Robin Newton


Keep up to date with the latest 1Spatial news and events

1Spatial: Steering you towards efficient and reliable location-based data with the 1Spatial Data Improvement Process

http://www.1spatial.com/improve_your_data/

1Spatial Group Limited; Registered in England No. 4785688 VAT Reg. No. 816329821; Registered Office: Tennyson House; Cambridge Business Park; Cambridge; CB4 0WZ; United Kingdom

IMPORTANT NOTICE
 
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender immediately and delete this e-mail from your system. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of 1Spatial Group Limited, its subsidiaries or associated companies, except where the author specifically states them to be the views of 1Spatial Group Limited, its subsidiaries or associated companies.
 
1Spatial Group Limited, its subsidiaries and associated companies will not be held liable for any legally binding obligations that are not the subject of an official 1Spatial purchase order or as part of a contract signed by a director of one of the aforementioned companies.
 
Although the company has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. You should understand and accept that, when communicating with us by e-mail, it is not a totally secure communications medium.