MapScript C# Memory Corruption Problem (MapServer 4.10)

[Hong]

Hi Tamas,

Since you have access to a Windows Server 2003 x64 R2 machine, can you tell me
your experience running MapServer on it? Did you experience random memory
corruption the way I did?

Hong

On Thu, Feb 15, 2007 at 11:17:45AM +0100, Tamas Szekeres wrote:
> Hi Hong,
> 
> Are you sure you've made an x64 build for your dependent libraries?
> You should start by enabling only the vital components in the build,
> and adding the new dependencies one by one after a successful build.
> The gd library is a vital component for mapserver. I would suggest to
> set up a configuration enabling only the gd component in nmake.opt.
> 
> I've attached my mapserver nmake.opt and the gd makefile.vc. I've
> tested these configurations on a Windows Server 2003 x64 R2 and AMD64.
> 
> You might also compile the SDE related stuff or obtain an x64
> compatible library version for linking against.
> 
> 
> Best regards,
> 
> Tamas
> 
> 
> 
> 2007/2/15, Hong <ljfong at sdf.lonestar.org>:
> >Ok, trying to compile everything in pure 64-bit mode, compilation went 
> >fine up
> >to the moment of linking:
> >
> >link /dll   mapbits.obj maphash.obj mapshape.obj mapxbase.obj...
> >
> >that ended up with symbol referencing error:
> >
> >...
> >Creating library mapserver_i.lib and object mapserver_i.exp
> >mapraster.obj : error LNK2001: unresolved external symbol gdImageDestroy
> >mapgd.obj : error LNK2001: unresolved external symbol gdImageDestroy
> >...
> >mapsde.obj : error LNK2019: unresolved external symbol SE_reginfo_free 
> >referenced
> >in function msSDELayerGetRowIDColumn
> >mapsde.obj : error LNK2019: unresolved external symbol
> >SE_reginfo_get_rowid_column referenced in function msSDELayerGetRowIDColumn
> >...
> >
> >All the libs are in correct places, location hard coded, and passed to 
> >LINK.
> >Searching on google revealed that someone had similar problem compiling 
> >GDAL
> >on 64-bit as well. The message on gdal-dev mentioned about the way symbol
> >names are decorated differently in 64-bit windows. I'm thinking bgd.def 
> >might
> >have trailing "@nn" that needs to be replaced with something else. Does 
> >anyone
> >have any experience dealing with this issue?
> >
> >The original message on gdal-dev:
> >
> >http://lists.maptools.org/pipermail/gdal-dev/2006-May/008885.html
> >
> >Thanks.
> >
> >Hong
> >
> >
> >On Fri, Feb 02, 2007 at 10:51:21PM +0100, Tamas Szekeres wrote:
> >> Since this issue seems to be quite an essential it would be useful to
> >> keep it tracked through a bug report. Would you submit one by
> >> summarizing the all the information we currently have
> >> (http://mapserver.gis.umn.edu/bugs/enter_bug.cgi)? Assign the bug to
> >> the C# binding.
> >>
> >> Here are some further tests that would worth trying:
> >>
> >> 1. Try to alter the SDE connection pooling behaviour. Adding
> >> PROCESSING "CLOSE_CONNECTION=DEFER"  would set it on a per layer
> >> basis. I don't know exactly which would be the best at this case, but
> >> in fact the connections are retained per threads. Maybe keeping the
> >> connections alive would affect how often the memory allocations
> >> related to the connections take place.
> >>
> >> 2. I've just added a possibility to use a global lock when invoking
> >> into the mapscript code. To enable this option you should uncomment
> >> #define USE_GLOBAL_LOCK in csmodule.i. and recompile the mapscript
> >> code. This setting would ensure that only one thread will invoke into
> >> libmap.dll in most cases simultaneously, and could help detecting some
> >> threading issues, though this solution is not equal as restricting the
> >> invocation to one thread (like the COM STA model would do for
> >> example).
> >>
> >> 3. Ensure disposing the cloned mapObj explicitly by the "using"
> >> directive for example, like
> >>
> >> using (map = ((mapObj)Session("m_mapObj")).clone() )
> >> {
> >>    // operate on map
> >> }
> >>
> >> The child references (if any) should also be controlled by an internal
> >> using blocks. All of the mapscript objects implement IDisposable.
> >>
> >>
> >> Best regards,
> >>
> >> Tamas
> >>
> >>
> >>
> >> 2007/2/2, Hong <ljfong at sdf.lonestar.org>:
> >> >On Thu, Feb 01, 2007 at 10:47:00PM +0100, Tamas Szekeres wrote:
> >> >> Hi Hong,
> >> >>
> >> >> Are you still getting the problem at the same function or the location
> >> >> may change in a random fashion? Do you have a stacktrace on it?
> >> >
> >> >I'll summarize the various states of MapServer for me so far:
> >> >
> >> >VS2003 compilation on 32-bit environment: stable (no memory corruption 
> >at
> >> >all)
> >> >
> >> >VS2005 compilation on 32-bit environment: behavior is unpredictable, 
> >memory
> >> >corruption at will, be that assigning filter string to layer, creating a
> >> >new
> >> >classification, etc.
> >> >
> >> >VS2003 compilation on 64-bit environment: 100% memory corruption.
> >> >
> >> >VS2005 compilation on 64-bit environment: behavior is unpredictable, 
> >memory
> >> >corruption at will. As a matter of curiosity, this happened:
> >> >With debugger attached to the main process running MapServer, I went to 
> >the
> >> >'immediate window' after a breakpoint and did
> >> >'?p_classObj.getExpressionString' a few times. The first few times, I 
> >got
> >> >the
> >> >expression string, on the 4th time or so, access violation happened.
> >> >Hmmm...
> >> >
> >> >A chunk of my project requires MapServer to run reliably on 64-bit
> >> >environment, seems like that is a no-go for any foreseeable future. 
> >Worst
> >> >of
> >> >all, when memory corruption happened with MapServer, the memory space of
> >> >ASP.NET worker process is messed up as well. Only through restarting the
> >> >worker process is the process back to its "normal" self.
> >> >
> >> >However, I haven't tried hard compiling MapServer in truly 64-bit way, 
> >that
> >> >is, through the VS2005 64-bit command prompt. Tried a few times before, 
> >but
> >> >failed miserably. This is what I will do next in my attempt to get it to
> >> >run
> >> >reliably on 64-bit environment.
> >> >
> >> >> At the moment I can think of 3 issues the memory problem might be 
> >related
> >> >> to.
> >> >>
> >> >> 1. Usage of different CRT heaps / using different dll versions
> >> >> You should make sure (once again) that only 1 version of the CRT dll
> >> >> is used in the dependency tree of mapscript.dll. In this case
> >> >> msvcr71.dll would be the expected version. If the same dll is
> >> >> referenced from multiple dll-s make sure that the references point to
> >> >> the same location. Make sure that the same dll has been loaded as you
> >> >> have compiled against.
> >> >> Kicking off the unused dependencies are also a good practice. (May be
> >> >> you have possibly done it)
> >> >
> >> >Double-checked all the dependencies. All the DLLs use the same CRT 
> >library.
> >> >
> >> >> 2. Early garbage collection
> >> >> Ensure that the lifespan of the child references (eg. layerObj) are
> >> >> not falling outside of the lifespan of the root object (eg. mapObj).
> >> >> Here is an example may cause potential problems:
> >> >>
> >> >> parentObj parent = new parentObj();
> >> >> childObj child = parent.getChildObj();
> >> >> // at this point there's no more references to 'parent' so it might be
> >> >> garbage collected
> >> >> child.OperateOnChild();  // will cause access violation
> >> >>
> >> >> So as to keep the parent alive until a certain point you might use
> >> >> GC.KeepAlive() as.
> >> >>
> >> >> parentObj parent = new parentObj();
> >> >> childObj child = parent.getChildObj();
> >> >> child.OperateOnChild();
> >> >> GC.KeepAlive(parent);
> >> >>
> >> >> will prevent from garbage collecting the parent until the
> >> >> OperateOnChild() returns. Try to add GC.KeepAlive()
> >> >> at the end of the function.
> >> >
> >> >Tried that before, no help. Now, I have no clear idea how GC.KeepAlive()
> >> >works
> >> >on object that is maintained across ASP.NET session.
> >> >
> >> >> 3. Multithreading issues.
> >> >> To properly isolate this issue a global lock should be added when
> >> >> invoking the mapscript code (practically at mapscript.dll). I will
> >> >> consider how it can be done easily. Until then you can make a test
> >> >> against the builtin shape or the posgis driver to see whether the
> >> >> problem is SDE related or not.
> >> >
> >> >I'm still yet to test this on a postgis driver or just the builtin shape
> >> >driver. I'll keep you posted on what happens when I do.
> >> >
> >> >Another thing curious: I'm also using ESRI ArcIMS in my project and the
> >> >ActiveX connector seems to run just fine in 64-bit environment, without 
> >any
> >> >special set up. Granted the thing is also a COM object, and COM object 
> >is
> >> >suppported in .NET framework officially. However, I don't feel it is the
> >> >same
> >> >with pINVOKE, which is responsible for MapServer on .NET. Maybe pINVOKE 
> >is
> >> >buggy under 64-bit of windows. My end goal is to ditch ArcIMS (it has 
> >its
> >> >own
> >> >set of problems) completely in my project and have MapServer in its 
> >place.
> >> >I
> >> >think I'm getting there, but still there is some way to go.
> >> >
> >> >I'll keep this thread updated on my progress.
> >> >
> >> >Thanks.
> >> >
> >> >Hong
> >> >
> >> >> 2007/2/1, Hong <ljfong at sdf.lonestar.org>:
> >> >> >An update for my progress:
> >> >> >
> >> >> >After calling clone() before calling draw() of mapObj, MapServer 
> >became
> >> >> >more
> >> >> >stable, but under heavier load (many more dynamic layers and config),
> >> >the
> >> >> >memory corruption still happened.
> >> >> >
> >> >> >The strange solution I had found was to compile MapServer using 
> >Visual
> >> >> >Studio
> >> >> >2003 instead of 2005. After doing so, MapServer no longer suffered
> >> >memory
> >> >> >corruption. For some reason, the binary produced from VS2003 running
> >> >under
> >> >> >.NET 2.0 did not have any memory management problem. This was done 
> >with
> >> >the
> >> >> >mapObj still being maintained through session.
> >> >> >
> >> >> >After a lot of brainstorming over this, I have come to the conjecture
> >> >that
> >> >> >the
> >> >> >garbage collector in the .NET framework might have been the culprit 
> >of
> >> >the
> >> >> >memory corruption. Since MapServer internal objects are managed
> >> >internally
> >> >> >via
> >> >> >PInvoke, they are unmanaged as far as the garbage collector is
> >> >concerned.
> >> >> >Somehow between ASP.NET page postbacks, the garbage collector came in
> >> >and
> >> >> >invalidated the references unmanaged objects prematurely, as they are
> >> >not
> >> >> >referenced by any managed object. Now, objects compiled for .NET 1.1 
> >(vs
> >> >> >2003)
> >> >> >are somehow treated differently by the garbage collector of .NET 
> >2.0. I
> >> >> >don't
> >> >> >know the exact mechanism. There are ways to control the garbage
> >> >collector,
> >> >> >however I would rather not explore them.
> >> >> >
> >> >> >I would like to hear any other thoughts/comments.
> >> >> >
> >> >> >Hong
> >> >> >
> >> >> >On Mon, Jan 22, 2007 at 10:39:38PM -0700, Hong wrote:
> >> >> >> I have reopened the bug report.
> >> >> >>
> >> >> >> On Mon, Jan 22, 2007 at 11:12:39PM +0100, Tamas Szekeres wrote:
> >> >> >> > 2007/1/22, Hong <ljfong at sdf.lonestar.org>:
> >> >> >> > >
> >> >> >> > >Now instead of using the mapObj stored in the session, I 
> >perform a
> >> >> >clone()
> >> >> >> > >on the mapObj from the session, and use the clone to perform
> >> >draw().
> >> >> >So
> >> >> >> > >far, I
> >> >> >> > >have not seen any crash after doing this. The only problem I
> >> >observed
> >> >> >was
> >> >> >> > >that
> >> >> >> > >the symbol transparencies was messed up after subsequent clone +
> >> >draw.
> >> >> >> > >Maybe
> >> >> >> > >the clone is not perfect, I'm not sure.
> >> >> >> > >
> >> >> >> >
> >> >> >> > That seems strange. You should describe the problem in more 
> >detail
> >> >and
> >> >> >> > reopen the following bug then:
> >> >> >> > http://mapserver.gis.umn.edu/bugs/show_bug.cgi?id=931
> >> >> >> >
> >> >> >> > Please attach the corresponding mapfile, the symbol definition 
> >file
> >> >> >> > and the symbol image to this bug.
> >> >> >> >
> >> >> >> > Best Regards,
> >> >> >> >
> >> >> >> > Tamas
> >> >> >
> >> >
> >