<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:D="DAV:" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='color:#1F497D'>Why don’t you try modifying the
benchmark to stop storing all the attribute values in a single dictionary and
see what the side effect is? If the memory usage drops to an acceptable level
then the issue is most likely the benchmark implementation, not the SDF
Provider.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>Greg<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Carl Jokl
[mailto:carl.jokl@keynetix.com] <br>
<b>Sent:</b> Friday, July 18, 2008 10:07 AM<br>
<b>To:</b> Greg Boone<br>
<b>Subject:</b> RE: SDF 3.0 Memory Leak<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'>The test for memory
was not the intention of the benchmark but just a side effect. It was hardly
the most professional approach but to observe the memory usage I opened windows
task manager and watched the memory usage of the FDOBenchmark process.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'>The benchmark reads
entries up to an arbitary batch size. This is part of the FDOBatch class. Once
the batch size is reached the time to load that single batch is saved and all
the data loaded in that batch is discarded. The reason being that if we were to
test with a really large file of the order of gigabites then the entries could
not all be loaded into memory. Using batches was supposed to make the
test scalable up to larger sizes due to the assumption that discarding the data
in a batch would free up the memory it has occupied ready to load another
batch. Bear in mind that the benchmark was just about timing how long data took
to load and not (at least in this version) doing anything with the data. Yesterday
evening while doing some other mapguide work which had me looking into the C++
sourcecode I noted that the Dispose() function would call the C++ delete on the
native object. This gave me the idea of altering the benchmark code so that
both the FDO batch and FDO Entry classes would have a dispose method which
would explicitly go through every map guide value object and call the dispose
method. My thinking was that perhaps the FDO wrappers to these data objects
were being garbage collected but the native components which they wrapped were
remaining in memory and could then explain the memory leak. I put this to the
test but having put the code in place to dispose of all the value objects the
memory usage of the application during and after the benchmark were still the
same. I also endeavoured to explicitly dispose of any FDO classes as soon as
they were no longer needed be it connections, feature readers, schemas, class
definitions. Disposing of them explicitly in this way still did not seem to
impact on the memory used when observing the footprint though task manager.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'>I am informed by a
colleague that AutoDesk are in position of some profiling tools which could
more effectively diagnose what is going on with the memory usage than I am able
to.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'>Regards<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'>Carl<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'><o:p> </o:p></span></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Greg Boone
[mailto:greg.boone@autodesk.com] <br>
<b>Sent:</b> 18 July 2008 14:50<br>
<b>To:</b> Carl Jokl<br>
<b>Subject:</b> RE: SDF 3.0 Memory Leak<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>Excuse my insistence here, but I
need to understand the memory read dynamics of the benchmark. <o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>So, the benchmark opens a 344MB
SDF file. It then reads all the features and stores a copy of all the Attribute
and Geometry values for each feature in memory in a dictionary. Is this the
case?<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>What is your expected result in
this situation. As I read it, memory usage will increase as each feature is
read and stored in the dictionary, up until memory usage reaches ~340 MB. <o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>Greg<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Carl Jokl
[mailto:carl.jokl@keynetix.com] <br>
<b>Sent:</b> Friday, July 18, 2008 9:43 AM<br>
<b>To:</b> Greg Boone<br>
<b>Subject:</b> RE: SDF 3.0 Memory Leak<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'>This was the intent
but on it’s own doing this isn’t of great value. The idea is to time how long
loading takes so it performs a loading operation from and SDF file times how
long it takes to load the contents of the file while discarding the data once
it has loaded. The time here on it’s own isn’t very useful but for the
benchmark and identical set of data was stored in a PostGIS database and the
time taken to load the contents of the SDF file through the SDF FDO provider
were compared with the time taken to load identical data from the PostGIS
database through the PostGIS provider.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'>The times were then
compared to see how the retrieval times compared. I was expecting PostGIS to be
a bit slower because the SDF just had to read in from a Flat file and this
would likely be happening in the same process as the calling benchmark. By
comparison the PostGIS data is in a PostGIS database running in it’s own
process and piping data out over TCP/IP. That was bound to be extra overhead vs
the SDF provider. <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'>The point of the
benchmark was that at Keynetix we are migrating a legacy MapGuide 6 application
to MapGuide enterprise. As part of this migration the question of the best way
to store the migrated data came up. The options are to continue using SDF flat
files, use PostGIS, use SQLServer or if we really had lots of money to use
Oracle. To help make that kind of decision I was asked to try and compare
the speed of data manipulation from the various data sources. If I had time I
would have done reading, writing and querying etc as was my original intention.
As I went through development however I was put under increasing pressure just
to get some figures back and so I just implemented reading in this version.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'>The benchmark showed
that the PostGIS FDO provider was much much slower than SDF, far more so than I
expected. I was a bit suspicious of this and so wrote a test Java program to
load the same data directly via JDBC as well as using PostGIS with PGAdmin and
running queries and that was much much faster (the Java program was able
to load the data faster than the SDF provider).<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'>This as well as some
log analysis on PostGIS pointed the speed problems to most likely be caused by
the quality of the PostGIS fdo provider implementation.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'>By this point the
benchmarks had served their purpose for the most part. It was dug out again
when my colleague was writing a migration application to migrate our legacy SDF
2.0 data to SDF 3.0. This also used FDO in parts but was plagued by the Memory
out of error problem. When that was discovered I dug out this benchmark
application to see if it too had problems with escalating requirements for
memory. I found that it did albeit not to the point of having a memory out of
error exception. <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'>I hope that explains
what the code was for. I commented out the PostGIS benchmark as it will not be
of much use to you unless you have a PostGIS data source to test with but that
provider did not appear to have a memory leak problem that I could tell. <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'>Regards<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'>Carl <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='color:#1F497D'><o:p> </o:p></span></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Greg Boone
[mailto:greg.boone@autodesk.com] <br>
<b>Sent:</b> 18 July 2008 14:22<br>
<b>To:</b> Carl Jokl<br>
<b>Subject:</b> RE: SDF 3.0 Memory Leak<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p>
<p class=MsoNormal>Can I ask what is the ultimate purpose of the Benchmark
code? After looking at the source code, it seems that the benchmark opens the
SDF file, reads all the features, both attributes and geometry values, and
stores them in memory. Is this the intent?<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Greg<span style='color:#1F497D'><o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Carl Jokl
[mailto:carl.jokl@keynetix.com] <br>
<b>Sent:</b> Friday, July 18, 2008 9:04 AM<br>
<b>To:</b> Greg Boone<br>
<b>Subject:</b> SDF 3.0 Memory Leak<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><span lang=EN-GB>Greg<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB>I attach a copy of the benchmark test
program. I apologise that it is not as well structured as I would like as I was
under pressure to get some results back quickly and originally the benchmarks
were going to cover reading writing querying and removing etc. In the end only
reading was implemented. The idea was that there is an FDOBenchmark class. Each
specific FDO provider would have its own subclass which implements setting up
an FDO connection to that source in the way that specific provider needs to but
with the actual benchmark tests executing in the common base code to make the
test as fair as possible.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB>The individual benchmarks ended up being
instantiated in the BenchmarkRunnerForm code behind. It was originally intended
to be cleaner that this with a specific benchmark runner class doing this
kind of thing but this was a quick shortcut to save time because I was under
pressure to get some results data. <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB>There is a line in the
BenchmarkRunnerForm.cs:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New"'>_sdfBenchmark
= <span style='color:blue'>new</span> <span style='color:#2B91AF'>SDF3Benchmark</span>(<span
style='color:blue'>new</span> <span style='color:#2B91AF'>FileInfo</span>(<span
style='color:#A31515'>"E:\\llWater.sdf"</span>), <span
style='color:blue'>new</span> <span style='color:#2B91AF'>FileInfo</span>(<span
style='color:#A31515'>"D:\\testing_file.sdf"</span>));<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB>You will have to change this line to point
to whatever SDF 3 data file you are going to use to test with as I don’t think
you would want me attaching the 344mb llWater.sdf to this email. The second
parameter was for use<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB>In the writing benchmark test but as that
did not get implemented then changing it does not really matter. I think that
all that happens in the code is that it may check for the presence of the
second file and delete it if it exists so as to create a fresh destination file
as part of the test.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB>If you need anything else please let me
know.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB>Carl Jokl<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-GB>Keynetix<o:p></o:p></span></p>
</div>
</body>
</html>