[mapguide-users] MG Error Out of memory, bad allocation

[Jason Birch]

I'm using FWTools in combination with GnuWin32.  Incredibly handy, and almost feels like I'm on linux...

In my case, I processed each of my as-delivered TIFF+wld files as follows using a batch file I called for each image (^ is line continuation):

gdal_translate   ^
  -co "TILED=YES"   ^
  -co "PROFILE=GEOTIFF"   ^
  -co "INTERLEAVE=BAND"   ^
  -co "BLOCKXSIZE=512"   ^
  -co "BLOCKYSIZE=512"  ^
  -a_srs utm83-10.prj  ^
  input/%1.tif   ^
  output/%1.tif

gdaladdo -r gauss output.tif 2 4 8 16 32 64 128

This generated true GeoTIFFs (with embedded projection information) that were tiled optimally and contained overviews (successively resampled pyramids).

This left me with a large number of small images which are optimal at close zoom, but I still needed some merged overviews for when zoomed out.  The initial key to this was to generate a virtual image (basically a raster catalog) from these.  The "ufind" command is a renamed find.exe from the GnuWin32 distribution because windows find gets in the way otherwise:

ufind input/ -name "*.tif" | xargs gdalbuildvrt -resolution highest all.vrt

Because I wanted a white background on my overviews, I then edited the all.vrt, adding a <NoDataValue/> section at the top of each <VRTRasterBand /> section:

  <VRTRasterBand dataType="Byte" band="1">
    <NoDataValue>255</NoDataValue>

The gdalinfo command gave me the dimensions of the virtual image, which I then divided by 4 to give me a reasonable sized (3GB) overview image.  Careful with sizes here.  If you're using MapGuide 2.1, you can use the -CO "BIGTIFF=YES" option if you need to generate files larger than 4GB.

gdal_translate   ^
  -outsize 53120 14000   ^
  -co "TILED=YES"   ^
  -co "PROFILE=GEOTIFF"   ^
  -co "INTERLEAVE=BAND"   ^
  -co "BLOCKXSIZE=512"   ^
  -co "BLOCKYSIZE=512"   ^
  all.vrt   ^
  all.tif

In my case, I didn't want to keep around the extra metadata that gdal generates so I deleted the .aux.xml file:

del all.tif.aux.xml

And finally, I generated overviews on the resultant:

gdaladdo -r gauss all.tif 2 4 8 16 32 64 128

Depending on the size of the data, there are other optimisations that could be done, such as first rolling the area up into 16ths, quarters, etc, and then generating additional resampled overview layers.

I need to thank the awesome folks on the #gdal IRC channel (in this case, I think it was Even, Howard, Frank) who helped me out when I ran into questions on this process...  Amazing support.

Links:

FWTools Downloads: 
http://home.gdal.org/fwtools/

GetGnuWin32 (easy way to download GnuWin32):
http://getgnuwin32.sourceforge.net/

gdal_translate docs:
http://www.gdal.org/gdal_translate.html

gdaladdo docs:
http://www.gdal.org/gdaladdo.html

gdalbuildvrt docs:
http://www.gdal.org/gdalbuildvrt.html

GeoTIFF format docs:
http://www.gdal.org/frmt_gtiff.html

VRT format docs:
http://www.gdal.org/gdal_vrttut.html

Jason

-----Original Message-----
From: Dave Wilson
Sent: Tuesday, July 21, 2009 7:47 AM
Subject: RE: [mapguide-users] MG Error Out of memory, bad allocation

What tool are you using to convert the files? Are people using FW Tools or something else?

Dave