[Mapserver-users] Very Large tiff output

[Frank Warmerdam]

Carl Anderson wrote:
> 
> I am at a loss......
> 
> Sometime ago I remember a comment that production images of 20,000 x  
> 20,000 pixels were being created using mapserver.
> 
> I am testing compositing images into much larger tiles (building  
> seamless.usgs.gov into quarter quads).  When the requested image size  
> exceeds about 12,000 x 12,000 pixels I am getting
> 
>  msSaveImageGDAL(): General error message. Failed to create MEM  dataset.
> 
> using "top" the peak memory use is 2.7Gb
> 
> The test system has 4Gb or real memory and 6Gb of swap
> and map.h has
>    #define MS_MAXIMGSIZE 35000
> 
> standard gdal tools gdalwarp and gdal_translate are fine with a 25,000  
> x 25,000 GTiff file.

Carl,

There are two reasons that MapServer is not very memory efficient for large
output files.

The first is that the actual drawing is done into a "GD" image in memory.
This means there is one complete copy of the output image held in ram.  If you
are doing 24bit output this is 4bytes per pixel or about 576MB in your case.

If you your result to a GDAL support file (instead of using the GD output
drivers for PNG for instance) then the msSaveImageGDAL() function actually makes
another complete memory copy of the image in RAM as a "virtual" image from
which GDAL can create the output image.   This is generally a similar amount
of memory to the GD image, so 576MB if producing a full 4 band (RGBA) output.

This of course doesn't come very near the 2.7GB of RAM you are reporting.  I
don't know where the other memory is being consumed.  Some sort of detailed
analysis would likely be needed.

There isn't really much that can be done about the GD image.   To work around
this size you would basically need to split your output size down to a
managable size for the system you are running on, and then mosaic the results.

However, it seems that MapServer is making it through the rendering to the
stage where a virtual copy of the dataset is made.  At this stage there are
various things that could be done in the code to avoid the extra copy.  Either
implement virtual reading functions that can read directly from GD, or do
direct writing to the output file (only works for some GDAL output formats).

The msSaveImageGDAL() code creates a complete copy of the image for convenience
and under the assumption that output image sizes are not very large.  While I
could correct this, it isn't very high on my priority list.   However, if it is
a pressing concern let me know and I will see what I can do.

Best regards,

-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent