[mapguide-users] Out of memory exception when generating tiles from large raster imagery

[Trevor Wekel]

Hello Julien,

For a multi user load, the Autodesk raster provider should give you better performance.  The GDAL provider in MapGuide is currently limited to a single thread to improve stability.  ECW is an excellent format for MapGuide.  TIFF is also good but it requires a lot more disk space.  For ECW, you may get a more usable memory footprint by using a smaller number of larger ECW images.  I have not tried this myself so I cannot say for certain.

The out of memory exception is typically caused by exceeding the virtual memory on the MapGuide Server process, and not the working set.  Under Windows Task Manager, View...  Select Columns...  add Peak Memory Usage and Virtual Memory Size.  If either of these hit 1.5GB or more for the MapGuide Server process, you are likely to run into problems.

Once the beta for MapGuide 2.2 is out the door, I will be looking at stability and performance for the GDAL Raster Provider.

Regards,
Trevor


-----Original Message-----
From: mapguide-users-bounces at lists.osgeo.org [mailto:mapguide-users-bounces at lists.osgeo.org] On Behalf Of julienltr
Sent: March 30, 2010 10:17 AM
To: mapguide-users at lists.osgeo.org
Subject: [mapguide-users] Out of memory exception when generating tiles from large raster imagery


Hi MapGuide users,

Following the release to public of our application based on Autodesk
MapGuide Entreprise 2010, we are facing some annoying crashes of the service
(a minimum of one crash per day and sometime every couple of hours). Of
course, before the release to public, with only few users and not an
extensive use, but with the exact same MapGuide service and hardware, we did
not detect this problem. Thanks to the logs generated since last week, we
found that the error is almost every time the same: an "out of memory"
exception when generating tiles from large raster imagery.

Example of this error:
<2010-03-30T09:42:31> 	3380	Ajax Viewer	xx.xxx.xx.xx	Administrator
 Error: Failed to stylize layer: XXXXX
        Out of memory.
        bad allocation
 StackTrace:
  - MgMappingUtil.StylizeLayers() line 791 file
d:\build\testarossa_nightly\build_37.1\ent\os\server\src\services\mapping\MappingUtil.cpp

When this error occur, the service throws few unclassified exceptions during
1 or 2 minutes. Restarting the service is not an option as it refuses to
stop himself. The only solution is to kill the process through a taskkill
command (fortunately we developed a small application to do it
automatically, based upon a Jason's idea :
http://lists.osgeo.org/pipermail/mapguide-users/2006-May/001039.html).

We found a lot of thing to read ([mapguide "out of memory"] on Google = 1650
results), and the scenarios described are close to our :

- MG Error Out of memory, bad allocation, by juergenk
http://n2.nabble.com/MG-Error-Out-of-memory-bad-allocation-td3298687.html
This case uses TIFF files, but without tiled layers in MapGuide.

- Continued MGOS 2.0.1 Crashes, by Andy Morsell
http://n2.nabble.com/Continued-MGOS-2-0-1-Crashes-td1817284.html  
No answer to this old post (2008).

- Ticket #480 (reopened defect), by zspitzer
Crash with MgInvalidStreamHeaderException with GETTILEIMAGE (load test)
http://trac.osgeo.org/mapguide/ticket/480 
This old still open ticket demonstrates a way to reproduce a crash server
with an MgInvalidStreamHeaderException. We also have sometime this exception
just before the service crash. Some other time is an
OSGeo.MapGuide.MgDbXmlException "Error: Out of memory in NsSAX2Reader parse
File".

Furthermore, we also read carefully the "best practices" wiki on
http://sandbox.mapguide.com 


Some specifications:

- MapGuide installation
Autodesk MapGuide Entreprise 2010
Update 1b
.NET & IIS
FDO 3.4

- Server hardware
Windows Server 2003 Datacenter Edition SP2
Intel Xeon E5405 2.00 GHz x 4
8.00 GB RAM

We are suspicious about the "out of memory" exception (does MapGuide log the
"real" exception?) as MapGuide service uses "only" 300 to 400 MB (but it's
maybe already too high?). The CPU is never above 25%. However, just before
the crash, the CPU can be higher than 80% during a couple of minutes.


Here's my questions:

Our rasters are ECW files, tiles generated by MapGuide are JPEG files of
600px, the FDO provider is the Autodesk Raster provider and layers are only
display beneath a zoom of 1:30000.
- Is Autodesk Raster provider better than OpenSource GDAL provider?
- Is ECW a "good" (or the "best") format for raster imagery inside MapGuide?
- Maybe TIFF is preferable?

Our ECW files are around 25 MB and there are 200 files per folder (i.e. one
folder = 4.5 GB = 1 DVD!).
- Is there some guideline about number of files and size?

On the map definiton, we have one base layer group with five base layers (1
base layer = 1 DVD).
- Is it better to merge all files into a single base layer (and a single
folder on the file system) or to dispatch them into a lot of base layers? 


Thank for reading, I would appreciate some feedback.
-- 
View this message in context: http://n2.nabble.com/Out-of-memory-exception-when-generating-tiles-from-large-raster-imagery-tp4825554p4825554.html
Sent from the MapGuide Users mailing list archive at Nabble.com.
_______________________________________________
mapguide-users mailing list
mapguide-users at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapguide-users