[fdo-internals] SDF 3.0 FDO Provider Serious Memory Leak

[Kenneth Skovhede, GEOGRAF A/S]

I can confirm that I have seen the memory problem.
I was able to minimize it by re-using objects, rather than create new ones.
It took a lot of experiments to figure out which classes were capable of 
re-use though.

Regards, Kenneth Skovhede, GEOGRAF A/S



Carl Jokl skrev:
> For clarification I must explain that there are two applications which were
> written using the SDF FDO provider. I wrote a benchmarking application. Just
> a simple one which was designed just to give a rough impression of the
> comparative speed differences of the different providers. The aim of this
> was to help us to decide which providers / storage methods we wanted to use
> in our application as we deal with large data sets and fairly heavy load
> where performance matters. 
>
> The memory leak was not noticed in this application initially becasue during
> the testing it never threw an out of memory exception (and still hasn't so
> far). The problem occured when an application was written by a colleague
> which is used to migrate our large legacy sdf 2.0 data to SDF 3.0. This
> application had problems with out of memory exceptions which were cured by
> putting in explicit calls to the garbage collector. The explicit calls to
> garbage collection though result is a severe performance penalty. For this
> reason work is going on right now to try and only do these calls after some
> optimum number of reads as part of a general trial of breaking down the
> migration into smaller batches of reads and writes to see if that will help. 
>
> I went back to my benchmakr application and tested it to see if it exhibited
> the same memory problems. As I said before it did not run out of memory even
> on a larger data set but still used lots of memory and what is most worrying
> in my opinion was that it still held on to a lot even after all the readers
> and connections had been closed. I tried putting explicit garbage collection
> to run after the benchmark and it made no discernable difference to the
> ammount of memory being held on to after the benchmark completed.
>
> There is a windows forms component to the benchmark application but this
> only displays the time taken to complete the test and how many records were
> processed etc but not any of the actual data from the file. The data in the
> file is stored in a list up to a limited batch size after which point the
> data is just discarded and a new batch starts (as this is just a benchmark
> and nothing usefull is being done with the data). 
>
> The actual migration application to my knowledge doesn't use a GUI at all.
>