[mapserver-users] MapCache - A way to reduce BerkeleyDB Cache size ?

[thomas bonfort]

For reference, when the disk cache has symlinking enabled:

case 1:
96M

case 2:
147M


On Tue, May 29, 2012 at 1:06 PM, thomas bonfort
<thomas.bonfort at gmail.com> wrote:
> Salut Ludovic, hi Andrew,
>
> I've investigated a bit more on the disk space issue with berkeley db
> backends, and here are some of my findings. To summarize, the space
> occupancy/efficiency  is closely tied to the size of the images stored
> in the cache, and the configured pagesize of the bdb backend.
>
> For a cache that contains many empty tiles, i.e. that are just a few
> hundred to a couple thousand bytes, here are the space occupancies
> (the disk cache uses 4k blocks). This is for reference only, usual
> caches will not resemble this one. Also note that the disk cache was
> not configured to symlink blank tiles, and that the bdb cache backend
> has some code in it to reduce the tile data stored on empty tiles.
>
> disk: 197M
> bdb pagesize=64k: 106M
> bdb pagesize=4k: 96M
> bdb pagesize=1k: 73M
> sqlite: 103M
> mbtiles (sqlite with blank tile detection): 62M
>
> So these numbers actually aren't so bad for bdb in all configurations,
> but in this case the 4k filesytem blocksize is really inefficient.
>
> These are the occupancies for another tileset, where the tiles are
> much heavier (tiles around 40-45k). There are also quite a few empty
> tiles.
>
> disk:153M
> bdb pagesize=64k: 209M
> bdb pagesize=4k: 148M
> bdb pagesize=1k: 144M
> sqlite: 140M
> mbtiles (sqlite with blank tile detection): 137M
>
>
> I had initally stayed away from using small pagesizes in the bdb
> backend because of some stability issues, but in the current tests I
> have not run into any issues. From these tests, I think the natural
> way forward is to change the default pagesize, and maybe make it
> configurable in mapcache.xml as an advanced configuration option.
> If you are able to confirm my findings on your instances, that would
> be great. To set the pagesize you'll have to recompile/reinstall,
> after having changed lib/cache_bdb.c by replacing
>
> #define PAGESIZE 64*1024
> by
> #define PAGESIZE 1*1024
>
> near line 50.
>
> regards,
> thomas
>
>
>
> On Wed, May 23, 2012 at 3:55 PM, Ludovic Gnemmi <lgnemmi at rgd73-74.fr> wrote:
>> Hi list,
>> I would like to experiment Berkeley DB cache type but i'm surprised of the
>> resulting db size compared to sqlite cache type. Here is my results for
>> about 13000 tiles:
>>
>> Total tiles size  = 44 Mb
>> Sqlite = 48 Mb
>> Berkeley DB = 61 Mb
>> Disk = 66 Mb (with 4kb filesystem block size)
>>
>> I thought i could reduce storage capacity with Berkeley DB cache type as
>> with Sqlite but it seems there is not much difference with disk cache type.
>> Is there a way to reduce Berkeley DB cache size ?
>>
>> Thanks for your help
>>
>> Ludovic
>>
>> --
>> View this message in context: http://osgeo-org.1560.n6.nabble.com/MapCache-A-way-to-reduce-BerkeleyDB-Cache-size-tp4976403.html
>> Sent from the Mapserver - User mailing list archive at Nabble.com.
>> _______________________________________________
>> mapserver-users mailing list
>> mapserver-users at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapserver-users