[mapserver-users] What resourse blocks any bigger GeoTIFF output from WCS?

[Rahkonen Jukka (MML)]

Hi,

Thank you very much about the information. How much about the troubles remain if 64-bit system is used, and would there be any difference between Windows and Linux then?  2 GB outputs are not unrealistically small for a web service but WCS coverages can be very large and as far as I know WCS has no standard way for paging.  I don't even feel very relaxed with paging by making subsequent GetCoverages by sliding the subset range without checking carefully how the pixels at the seams behave.

-Jukka Rahkonen-


Even Rouault wrote:

On Wednesday 05 August 2015 09:43:14 Rahkonen Jukka wrote:
> Hi,
> 
> I started to experiment with WCS 2.0.1 with Mapserver 7.1-dev and I 
> managed to get WCS to work rather easily. However, I can only get data 
> with rather small GetCoverage requests when using image/tiff format.
> 
> My environment:
> MapServer 7.1-dev which I installed from the 32-bit Windows binaries 
> from gisinternals 
> http://download.gisinternals.com/sdk/downloads/release-1600-gdal-mapserver.
> zip. For installation I used existing MS4W installation by making a 
> new cgi-bin directory where I dropped the new binaries (dll files and 
> mapserv.exe). My computer is 64-bit Windows 7 with 8 GB of memory. I 
> am stuck with 32-bit Mapserver because using MS4W on the bottom is the 
> only way to install that I can handle.
> 
> Output is successful when the resulting image is 8000x8000 pixels (192 
> MB) but if I try to get an image with 10000x10000 pixels (300 MB) I am 
> receiving an error:
> 
> <ows:ExceptionText>msWCSWriteFile20(): General error message. 
> msSaveImage() failed msSaveImageGDAL(): General error message. Failed 
> to create output GTiff file. TIFFAppendToStrip:Write error at scanline 
> 6500 I have no problem at all with output size of 12000x12000 pixels 
> if I use image/png or image/jpeg.
> 
> If I play with subset ranges I can see that the GeoTIFF write error 
> happens at different scanlines. It makes me think that it could be 
> some memory problem. But how could I give more resources for GDAL so 
> it could make its job?  I have not yet thought how big images I will 
> need from WCS but lets say the aim is at 50000x50000 pixels which 
> would make 7.5 gigabyte GeoTIFF as an uncompressed, 8-bit, 3-band image.

Jukka,

My understanding on what is involved is the following. MapServer will first allocate a memory buffer for the whole resulting image, do the rendering on it, create a GDAL MEM dataset with the content of the mapserver image, and then
CreateCopy() it to the final in-memory image file and then stream it out through HTTP. So in the case of the 10K x 10K image, you'll have a 300 MB buffer for the mapserver image + 300 MB for the GDAL MEM dataset + 300 MB for the resulting TIFF stored in a /vsimem/ file. Due to the way the TIFF writer works, by appending data and growing the /vsimem/ file progressively by successive realloc(), I suspect that heap memory fragmentation occurs and that on the available 2 GB of the 32 bit process, much more than the theoretically 3x300 MB needed is consumed, reaching the 2 GB limit.
I don't see any easy workaround, but a few possible improvements:
- (GDAL) In the case of uncompressed TIFF where the final file size is known the GDAL GTiff writer could perhaps be improved to actually pre-allocate the whole buffer instead of growing it progressively, so as to avoid heap fragmentation.
- (MapServer) Another/complementary option would be in the case of GTiff, and other formats that supports the GDALCreate() API, to avoid the use of the MEM dataset.
- (MapServer) Another/complementary option would be to offer an option to have a on-disk temporary file to save some RAM.  For non-virtual enabled GDAL formats, such as netCDF, this is what is used.
- (MapServer) Another/complementary option would be to use in MapServer the new capability in GDAL 2.0 of the GTiff driver to generate directly streamable file in the case of a uncompressed file.

But even in the best case (just the mapserver image buffer), the maximum file you could produce would be 2 GB with a 32bit MapServer build. In theory, I believe that WCS GetCoverage could generite arbitrary file size with modest RAM requirements but that would likely require major architectural changes in MapServer.

Even

> 
> -Jukka Rahkonen-
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users

--
Spatialys - Geospatial professional services http://www.spatialys.com