[mapserver-users] MapCache - A way to reduce BerkeleyDB Cache size ?

[thomas bonfort]

Salut Ludovic, hi Andrew,

I've investigated a bit more on the disk space issue with berkeley db
backends, and here are some of my findings. To summarize, the space
occupancy/efficiency  is closely tied to the size of the images stored
in the cache, and the configured pagesize of the bdb backend.

For a cache that contains many empty tiles, i.e. that are just a few
hundred to a couple thousand bytes, here are the space occupancies
(the disk cache uses 4k blocks). This is for reference only, usual
caches will not resemble this one. Also note that the disk cache was
not configured to symlink blank tiles, and that the bdb cache backend
has some code in it to reduce the tile data stored on empty tiles.

disk: 197M
bdb pagesize=64k: 106M
bdb pagesize=4k: 96M
bdb pagesize=1k: 73M
sqlite: 103M
mbtiles (sqlite with blank tile detection): 62M

So these numbers actually aren't so bad for bdb in all configurations,
but in this case the 4k filesytem blocksize is really inefficient.

These are the occupancies for another tileset, where the tiles are
much heavier (tiles around 40-45k). There are also quite a few empty
tiles.

disk:153M
bdb pagesize=64k: 209M
bdb pagesize=4k: 148M
bdb pagesize=1k: 144M
sqlite: 140M
mbtiles (sqlite with blank tile detection): 137M


I had initally stayed away from using small pagesizes in the bdb
backend because of some stability issues, but in the current tests I
have not run into any issues. From these tests, I think the natural
way forward is to change the default pagesize, and maybe make it
configurable in mapcache.xml as an advanced configuration option.
If you are able to confirm my findings on your instances, that would
be great. To set the pagesize you'll have to recompile/reinstall,
after having changed lib/cache_bdb.c by replacing

#define PAGESIZE 64*1024
by
#define PAGESIZE 1*1024

near line 50.

regards,
thomas



On Wed, May 23, 2012 at 3:55 PM, Ludovic Gnemmi <lgnemmi at rgd73-74.fr> wrote:
> Hi list,
> I would like to experiment Berkeley DB cache type but i'm surprised of the
> resulting db size compared to sqlite cache type. Here is my results for
> about 13000 tiles:
>
> Total tiles size  = 44 Mb
> Sqlite = 48 Mb
> Berkeley DB = 61 Mb
> Disk = 66 Mb (with 4kb filesystem block size)
>
> I thought i could reduce storage capacity with Berkeley DB cache type as
> with Sqlite but it seems there is not much difference with disk cache type.
> Is there a way to reduce Berkeley DB cache size ?
>
> Thanks for your help
>
> Ludovic
>
> --
> View this message in context: http://osgeo-org.1560.n6.nabble.com/MapCache-A-way-to-reduce-BerkeleyDB-Cache-size-tp4976403.html
> Sent from the Mapserver - User mailing list archive at Nabble.com.
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users