[gdal-dev] Gdaladdo feels slow with Rasterlite

[Even Rouault]

Le mercredi 04 juillet 2012 19:27:40, Jukka Rahkonen a écrit :
> Hi,
> 
> I feel that gdaladdo is not working optimally with any bigger rasterlite
> tables. For example for my 153600x249600 pixel sized test image the
> initial write into rasterlite took 3 hours and 10 minutes while adding
> overviews with gdaladdo took 10 hours.

I tried a bit the gdal_translate part, but didn't have the patience to wait 
for it to complete (~ 1 hour on my PC). A sysprof profile shows that most of 
the CPU time ( ~ 75% ) is spent in zlib and libpng, which isn't surprising. 
I've tried to use tiles in geotiff with deflate compression, and it seems around 
twice faster, so png has a non neglectable overhead over plain zlib 
compression. But png seems to be smaller too.

> Gdaladdo seems to use remarkable
> much time every time it has finished with creating one overview layer.

Yes, this is expected. Each overview layer is wrapped into a BEGIN / COMMIT 
transaction, so the commit at the end will take some time. Without that, I 
think things would be globally slower. 

> In
> addition, the creation of the last overview layers does not feel as much
> faster than I would have guessed keeping in mind that the number of tiles
> to write goes 3/4 down at each step.

As far as the perceived slowness on the last overview layer, reviewing the 
code, I don't see any obvious hint that would explain that. I tried on a much 
smaller dataset, and I could see that building overview level (n+1) uses as 
expected results from level n. So there's something more subtle. Once the 
overviews are built, is the reading performance of the overviews OK ? I'm 
wondering if there are not some indexes that should be added on the metadata 
table, perhaps on pixel_x_size and pixel_y_size, to speed-up tile retrieval.

More generally, the rasterlite overview building does not use the generic 
overview code, but uses RasterIO() decimation. Not sure if this affects notably 
performance.

> 
> I put my test image available for testing
> http://latuviitta.org/documents/t80.tif (550 MB, deflate compressed tiff)
> I will keep it on a server for a week or so.
> 
> Commands I used were:
> gdal_translate -of rasterlite -co driver=png t80.tif t80.sqlite
> gdaladdo rasterlite:t80.sqlite,table=t80
>  2 4 8 16 32 64 128 256 512 1024
> 
> I tried to compare times with Rasterlite utilites rasterlite_load and
> raster_pyramid but they both failed totally with my test image.
> 
> -Jukka Rahkonen-
> 
> 
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev