[mapserver-users] What resourse blocks any bigger GeoTIFF output from WCS?

[Even Rouault]

On Wednesday 05 August 2015 09:43:14 Rahkonen Jukka wrote:
> Hi,
> 
> I started to experiment with WCS 2.0.1 with Mapserver 7.1-dev and I managed
> to get WCS to work rather easily. However, I can only get data with rather
> small GetCoverage requests when using image/tiff format.
> 
> My environment:
> MapServer 7.1-dev which I installed from the 32-bit Windows binaries from
> gisinternals
> http://download.gisinternals.com/sdk/downloads/release-1600-gdal-mapserver.
> zip. For installation I used existing MS4W installation by making a new
> cgi-bin directory where I dropped the new binaries (dll files and
> mapserv.exe). My computer is 64-bit Windows 7 with 8 GB of memory. I am
> stuck with 32-bit Mapserver because using MS4W on the bottom is the only
> way to install that I can handle.
> 
> Output is successful when the resulting image is 8000x8000 pixels (192 MB)
> but if I try to get an image with 10000x10000 pixels (300 MB) I am
> receiving an error:
> 
> <ows:ExceptionText>msWCSWriteFile20(): General error message. msSaveImage()
> failed msSaveImageGDAL(): General error message. Failed to create output
> GTiff file. TIFFAppendToStrip:Write error at scanline 6500
> I have no problem at all with output size of 12000x12000 pixels if I use
> image/png or image/jpeg.
> 
> If I play with subset ranges I can see that the GeoTIFF write error happens
> at different scanlines. It makes me think that it could be some memory
> problem. But how could I give more resources for GDAL so it could make its
> job?  I have not yet thought how big images I will need from WCS but lets
> say the aim is at 50000x50000 pixels which would make 7.5 gigabyte GeoTIFF
> as an uncompressed, 8-bit, 3-band image.

Jukka,

My understanding on what is involved is the following. MapServer will first 
allocate a memory buffer for the whole resulting image, do the rendering on it, 
create a GDAL MEM dataset with the content of the mapserver image, and then 
CreateCopy() it to the final in-memory image file and then stream it out through 
HTTP. So in the case of the 10K x 10K image, you'll have a 300 MB buffer for 
the mapserver image + 300 MB for the GDAL MEM dataset + 300 MB for the 
resulting TIFF stored in a /vsimem/ file. Due to the way the TIFF writer works, 
by appending data and growing the /vsimem/ file progressively by successive 
realloc(), I suspect that heap memory fragmentation occurs and that on the 
available 2 GB of the 32 bit process, much more than the theoretically 3x300 
MB needed is consumed, reaching the 2 GB limit.
I don't see any easy workaround, but a few possible improvements:
- (GDAL) In the case of uncompressed TIFF where the final file size is known the 
GDAL GTiff writer could perhaps be improved to actually pre-allocate the whole 
buffer instead of growing it progressively, so as to avoid heap fragmentation.
- (MapServer) Another/complementary option would be in the case of GTiff, and 
other formats that supports the GDALCreate() API, to avoid the use of the MEM 
dataset.
- (MapServer) Another/complementary option would be to offer an option to have 
a on-disk temporary file to save some RAM.  For non-virtual enabled GDAL 
formats, such as netCDF, this is what is used.
- (MapServer) Another/complementary option would be to use in MapServer the 
new capability in GDAL 2.0 of the GTiff driver to generate directly streamable 
file in the case of a uncompressed file.

But even in the best case (just the mapserver image buffer), the maximum file 
you could produce would be 2 GB with a 32bit MapServer build. In theory, I 
believe that WCS GetCoverage could generite arbitrary file size with modest RAM 
requirements but that would likely require major architectural changes in 
MapServer.

Even

> 
> -Jukka Rahkonen-
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com