[gdal-dev] Gdaladdo feels slow with Rasterlite

[Rahkonen Jukka]

Even Rouault wrote:

> Le mercredi 04 juillet 2012 19:27:40, Jukka Rahkonen a écrit :
>> Hi,
>>
>> I feel that gdaladdo is not working optimally with any bigger rasterlite
>> tables. For example for my 153600x249600 pixel sized test image the
>> initial write into rasterlite took 3 hours and 10 minutes while adding
>> overviews with gdaladdo took 10 hours.

> I tried a bit the gdal_translate part, but didn't have the patience to wait
> for it to complete (~ 1 hour on my PC). A sysprof profile shows that most of
> the CPU time ( ~ 75% ) is spent in zlib and libpng, which isn't surprising.
> I've tried to use tiles in geotiff with deflate compression, and it seems around
> twice faster, so png has a non neglectable overhead over plain zlib
> compression. But png seems to be smaller too.

I repeated my test with another computer and I could not repeat the bad behaviour.
Overviews were generated at reasonable time and highest levels in the pyramid 
were just as much faster to create as could be estimated.  I can only imagine
that my first test computer was suffering from lack of some resources. Right 
now I cannot imagine what resource it could have been but I do not think
that it is so important either because there does not seem to be anything in 
gdaladdo that is totally wrong.

One issue there is still.  My original was a highly compressed (deflate) tiff 
with a file size of 550 MB.  Initial transform into Rasterlite with png tiles yielded
a 1.2 GB Rasterlite db. Condiderably bigger but still understandable.  However,
gdaladdo has no option to compress internal overviews which make most
sense with Rasterlite.  Because of this the Rasterlite overviews are 
uncompressed tiff tiles and after creating the whole set of overlays the file
size is not less than 37 GB.  It is naturally far too much and gdaladdo is
unusable for the kind of original maps I used in the test.
If gdaladdo cannot be modified to make compressed internal overviews when 
target format is suitable for that (I suppose at least all database rasters 
are such: Rasterlite, PostGIS raster, Oracle georasters) then it might be worth 
mentioning in gdaladdo manual page that the that the need of disk space
may be surprisingly high.

-Jukka Rahkonen-


>> Gdaladdo seems to use remarkable
>> much time every time it has finished with creating one overview layer.

> Yes, this is expected. Each overview layer is wrapped into a BEGIN / COMMIT
> transaction, so the commit at the end will take some time. Without that, I
> think things would be globally slower.

>> In addition, the creation of the last overview layers does not feel as much
>> faster than I would have guessed keeping in mind that the number of tiles
>> to write goes 3/4 down at each step.

> As far as the perceived slowness on the last overview layer, reviewing the
> code, I don't see any obvious hint that would explain that. I tried on a much
> smaller dataset, and I could see that building overview level (n+1) uses as
> expected results from level n. So there's something more subtle. Once the
> overviews are built, is the reading performance of the overviews OK ? I'm
> wondering if there are not some indexes that should be added on the metadata
> table, perhaps on pixel_x_size and pixel_y_size, to speed-up tile retrieval.

> More generally, the rasterlite overview building does not use the generic
> overview code, but uses RasterIO() decimation. Not sure if this affects notably
> performance.

>
>> I put my test image available for testing
>> http://latuviitta.org/documents/t80.tif (550 MB, deflate compressed tiff)
>> I will keep it on a server for a week or so.
>>
>> Commands I used were:
>> gdal_translate -of rasterlite -co driver=png t80.tif t80.sqlite
>> gdaladdo rasterlite:t80.sqlite,table=t80
>>  2 4 8 16 32 64 128 256 512 1024
>
>> I tried to compare times with Rasterlite utilites rasterlite_load and
>> raster_pyramid but they both failed totally with my test image.
>>
>> -Jukka Rahkonen-
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev