[mapserver-users] What resourse blocks any bigger GeoTIFF output from WCS?

[Even Rouault]

On Wednesday 05 August 2015 12:16:49 Rahkonen Jukka wrote:
> Hi,
> 
> Thank you very much about the information. How much about the troubles
> remain if 64-bit system is used,

With a 64 bit system, you would still need to be able to allocate 3 times the 
size of the output file, either from RAM only or RAM + swap space.

> and would there be any difference between
> Windows and Linux then?

On 64 bit probably little. On 32 bit, the Linux memory allocator could 
potentially be better at limiting fragmentation, but I'm not sure of that.

> 2 GB outputs are not unrealistically small for a
> web service but WCS coverages can be very large and as far as I know WCS
> has no standard way for paging.  I don't even feel very relaxed with paging
> by making subsequent GetCoverages by sliding the subset range without
> checking carefully how the pixels at the seams behave.

There might indeed be border effects if resampling is involved, although 
ideally MapServer should query a bit more than the requested area (I think it 
does)

> 
> -Jukka Rahkonen-
> 
> 
> Even Rouault wrote:
> 
> On Wednesday 05 August 2015 09:43:14 Rahkonen Jukka wrote:
> > Hi,
> > 
> > I started to experiment with WCS 2.0.1 with Mapserver 7.1-dev and I
> > managed to get WCS to work rather easily. However, I can only get data
> > with rather small GetCoverage requests when using image/tiff format.
> > 
> > My environment:
> > MapServer 7.1-dev which I installed from the 32-bit Windows binaries
> > from gisinternals
> > http://download.gisinternals.com/sdk/downloads/release-1600-gdal-mapserver
> > .
> > zip. For installation I used existing MS4W installation by making a
> > new cgi-bin directory where I dropped the new binaries (dll files and
> > mapserv.exe). My computer is 64-bit Windows 7 with 8 GB of memory. I
> > am stuck with 32-bit Mapserver because using MS4W on the bottom is the
> > only way to install that I can handle.
> > 
> > Output is successful when the resulting image is 8000x8000 pixels (192
> > MB) but if I try to get an image with 10000x10000 pixels (300 MB) I am
> > receiving an error:
> > 
> > <ows:ExceptionText>msWCSWriteFile20(): General error message.
> > msSaveImage() failed msSaveImageGDAL(): General error message. Failed
> > to create output GTiff file. TIFFAppendToStrip:Write error at scanline
> > 6500 I have no problem at all with output size of 12000x12000 pixels
> > if I use image/png or image/jpeg.
> > 
> > If I play with subset ranges I can see that the GeoTIFF write error
> > happens at different scanlines. It makes me think that it could be
> > some memory problem. But how could I give more resources for GDAL so
> > it could make its job?  I have not yet thought how big images I will
> > need from WCS but lets say the aim is at 50000x50000 pixels which
> > would make 7.5 gigabyte GeoTIFF as an uncompressed, 8-bit, 3-band image.
> 
> Jukka,
> 
> My understanding on what is involved is the following. MapServer will first
> allocate a memory buffer for the whole resulting image, do the rendering on
> it, create a GDAL MEM dataset with the content of the mapserver image, and
> then CreateCopy() it to the final in-memory image file and then stream it
> out through HTTP. So in the case of the 10K x 10K image, you'll have a 300
> MB buffer for the mapserver image + 300 MB for the GDAL MEM dataset + 300
> MB for the resulting TIFF stored in a /vsimem/ file. Due to the way the
> TIFF writer works, by appending data and growing the /vsimem/ file
> progressively by successive realloc(), I suspect that heap memory
> fragmentation occurs and that on the available 2 GB of the 32 bit process,
> much more than the theoretically 3x300 MB needed is consumed, reaching the
> 2 GB limit. I don't see any easy workaround, but a few possible
> improvements:
> - (GDAL) In the case of uncompressed TIFF where the final file size is known
> the GDAL GTiff writer could perhaps be improved to actually pre-allocate
> the whole buffer instead of growing it progressively, so as to avoid heap
> fragmentation. - (MapServer) Another/complementary option would be in the
> case of GTiff, and other formats that supports the GDALCreate() API, to
> avoid the use of the MEM dataset. - (MapServer) Another/complementary
> option would be to offer an option to have a on-disk temporary file to save
> some RAM.  For non-virtual enabled GDAL formats, such as netCDF, this is
> what is used. - (MapServer) Another/complementary option would be to use in
> MapServer the new capability in GDAL 2.0 of the GTiff driver to generate
> directly streamable file in the case of a uncompressed file.
> 
> But even in the best case (just the mapserver image buffer), the maximum
> file you could produce would be 2 GB with a 32bit MapServer build. In
> theory, I believe that WCS GetCoverage could generite arbitrary file size
> with modest RAM requirements but that would likely require major
> architectural changes in MapServer.
> 
> Even
> 
> > -Jukka Rahkonen-
> > _______________________________________________
> > mapserver-users mailing list
> > mapserver-users at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapserver-users
> 
> --
> Spatialys - Geospatial professional services http://www.spatialys.com

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com