[gdal-dev] memory problems with huge raster sizes

[Dennis Gocke]

Hi,



We've stumbled upon memory problems when using huge raster levels.



For instance for a common TMS/WMS server on the full level you will have
something like:

Raster Size: 268435456x268435456

Block Size: 256x256



With SUBBLOCK_SIZE 64 this results in:

nSubBlocksPerRow = 268435456 / 256 / 64 = 16384

nSubBlocksPerColumn = 268435456 / 256 / 64 = 16384



for each band papoBlocks will therefore have the size:

sizeof(void*) * nSubBlocksPerRow * nSubBlocksPerColumn => 2GB (64bit) and
1GB (32bit)



On 32bit system you will have out of memory exception with just one RGB
raster and on 64bit you will of course also get problems very fast.



One solution would be to increase SUBBLOCK_SIZE:

#define SUBBLOCK_SIZE 256

#define TO_SUBBLOCK(x) ((x) >> 8)

#define WITHIN_SUBBLOCK(x) ((x) & 0xff)



This will result in 128MB (64bit) for the size of papoBlocks instead of 2GB.



Perhaps an even better solution would be to make SUBBLOCK_SIZE not a
constant but instead dependent on nBlocksPerRow.

For instance:

toSubBlockShift = log2(nBlocksPerRow)/2;

subBlockSize =  1 << toSubBlockShift;

withinSubBlockMask = subBlockSize-1;

For the above example this would result in subBlockSize=1024 and just 8MB
(64bit) for papoBlocks.


This will handle even bigger rasters better and scale better for smaller.

Hopefully the performance impact for not having simple defines will not be
noticeable, but I don't think so.



Kind regards,

Dennis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20140213/4bb57592/attachment.html>