[gdal-dev] subblock of gdal source code

[Even Rouault]

Le Sunday 07 June 2009 12:18:24 gispowerfan, vous avez écrit :
> thank you very much! I understand the concept of subblocking in brief.
> It is the skill of memory management of large image block using the default
> paras such as SUBBLOCK_SIZE=64.

Not sure what you mean by "large image block" : to be more precise, 
subblocking is not related to the dimensions of a block itself, but to the 
number of blocks. For a constant image size, the larger the dimensions of a 
block, the lesser the number of blocks : nBlocksPerRow = ceil(nRasterXSize / 
nBlockXSize)

> The strategy of memory block management in 
> GDAL is LRU, I got the idea from mail list archives, and now are there only
> block list for all GDAL datasets?or one block list for one GDAL datasets?

There is only one block list shared by all GDAL datasets. It is accessed by 
the poOlded and poNewest pointers in gdalrasterblock.cpp. As this is used for 
all datasets, there is a global mutex to protect its access for 
multi-threading usage.
The role of the block list is to control the total memory used for block 
caching of all the datasets opened by GDAL.

> Is papoBlocks member the block list of memory?

No, the papoBlocks member is an array of pointer to blocks (or sub-blocks), 
maintained per rasterband (thus indirectly per dataset).

When the LRU block is discarded from the block list because the size of the 
block cache has reached its maximum (40 MB by default), it is freed and the 
related pointer in the papoBlocks of its rasterband is reset to NULL.
On the contrary when a new block is accessed, memory is allocated for it. The 
entry of the papoBlocks array points to that memory and the block is linked 
to the others in the block list.

>
> thanks
>
> gispowerfan
>
> 2009-06-07
>
>
>
>
>
> 发件人： Even Rouault
> 发送时间： 2009-06-07  16:56:17
> 收件人： gdal-dev
> 抄送： gispowerfan
> 主题： Re: [gdal-dev] subblock of gdal source code
>
> Unless you want to hack into gdalrasterband.cpp, you generally don't have
> to care about that concept of subblocking. It is just an internal
> implementation detail to improve memory usage when a raster has a big
> number of blocks. Usually, GDAL will create an array of pointers to blocks
> of size nBlocksPerRow x nBlocksPerColumn, stored in the papoBlocks member.
> Let's say your raster is of dimension 1000 x 1000 and the block size is
> 10x10. This would lead to the creation of an array of 100 x 100 pointers.
> If the number of blocks in one dimension is greater than SUBBLOCK_SIZE/2,
> then GDAL will create instead an array of ceil(nBocksPerRow /
> SUBBLOCK_SIZE) * ceil(nBocksPerColumn / SUBBLOCK_SIZE) where SUBBLOCK_SIZE
> = 64. In the example, as 100 > 32, an array of 4 x 4 is created in
> papoBlocks. Each element of this array can contain a sub-array of 64x64
> pointer to blocks. But at the beginning those sub-arrays are not created,
> in order to save memory. When you need to access block (i, j), GDAL will
> first compute the subblock in which the block is contained : (i', j') = (i
> / 64, j / 64). If the element at (i', j') is NULL, GDAL will first
> instanciate the 64x64 array. Then it will reserve memory for the block
> located at (i'', j'') = (i % 64, j % 64) in the subblock.
> In case of huge rasters, this concept could be extended to a third level of
> indirection ("sub-sub-blocks"), but this has not been implemented.
> Not sure I've been very clear. But now if you read the code, you should
> understand what I mean ;-)
>
> Le Sunday 07 June 2009 09:21:57 gispowerfan, vous avez écrit :
> > hi,
> >       Recently I read the sorce code of gdal, and I found that there are
> > subblocking in GDALRasterBand, can anyone give some examples of subblock,
> > and how to use it? some doc in detail will be better!
> >
> > thanks in advance!
> >
> > gispowerfan
> >
> > 2009-06-07
> >
> >
> >
> > gispowerfan