[gdal-dev] Gdaladdo feels slow with Rasterlite
even.rouault at mines-paris.org
Wed Jul 4 14:39:42 PDT 2012
Le mercredi 04 juillet 2012 19:27:40, Jukka Rahkonen a écrit :
> I feel that gdaladdo is not working optimally with any bigger rasterlite
> tables. For example for my 153600x249600 pixel sized test image the
> initial write into rasterlite took 3 hours and 10 minutes while adding
> overviews with gdaladdo took 10 hours.
I tried a bit the gdal_translate part, but didn't have the patience to wait
for it to complete (~ 1 hour on my PC). A sysprof profile shows that most of
the CPU time ( ~ 75% ) is spent in zlib and libpng, which isn't surprising.
I've tried to use tiles in geotiff with deflate compression, and it seems around
twice faster, so png has a non neglectable overhead over plain zlib
compression. But png seems to be smaller too.
> Gdaladdo seems to use remarkable
> much time every time it has finished with creating one overview layer.
Yes, this is expected. Each overview layer is wrapped into a BEGIN / COMMIT
transaction, so the commit at the end will take some time. Without that, I
think things would be globally slower.
> addition, the creation of the last overview layers does not feel as much
> faster than I would have guessed keeping in mind that the number of tiles
> to write goes 3/4 down at each step.
As far as the perceived slowness on the last overview layer, reviewing the
code, I don't see any obvious hint that would explain that. I tried on a much
smaller dataset, and I could see that building overview level (n+1) uses as
expected results from level n. So there's something more subtle. Once the
overviews are built, is the reading performance of the overviews OK ? I'm
wondering if there are not some indexes that should be added on the metadata
table, perhaps on pixel_x_size and pixel_y_size, to speed-up tile retrieval.
More generally, the rasterlite overview building does not use the generic
overview code, but uses RasterIO() decimation. Not sure if this affects notably
> I put my test image available for testing
> http://latuviitta.org/documents/t80.tif (550 MB, deflate compressed tiff)
> I will keep it on a server for a week or so.
> Commands I used were:
> gdal_translate -of rasterlite -co driver=png t80.tif t80.sqlite
> gdaladdo rasterlite:t80.sqlite,table=t80
> 2 4 8 16 32 64 128 256 512 1024
> I tried to compare times with Rasterlite utilites rasterlite_load and
> raster_pyramid but they both failed totally with my test image.
> -Jukka Rahkonen-
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
More information about the gdal-dev