[gdal-dev] Optimisation of ECW files

[Armin Burger]

On 08/11/2011 22:27, Yves Jacolin (free) wrote:

>
> It seems there was a performance issue with this files. I though using a lot of
> small ECW files was not a good practice. Am I wrong?


Yves

my experience is that making Mapserver (via GDAL) reading lots of small 
ECW files becomes quite inefficient. I would say having more than maybe 
50 or 100 files to be read for one Mapserver request starts to slow down 
the response time of Mapserver. Geotiff with overview files I found a 
bit better in this respect, but still suboptimal. ECW works well for 
very large single files, having them even with several 10's of GB will 
not cause any problems.

If you have frequent updates of the single files then merging the files 
into few large ones might not be a feasible option. I would then try to 
use Geotiff with overviews, it should be at least a bit better than the 
ECW. Automatically converting the ECW files to Geotiff is done quickley. 
Regarding the required space, Geotiff with overviews need ~5 to 15 times 
more disc space, depending on the ECW compression rate.

If you need to display the image layer already at very low scales (hence 
requiring many files to read) then a single merged low-resolution file 
with ~10-20% resolution (and with overviews) as active layer for low 
scales would help quite much. Then only when the request is at higher 
scales, the small images are read, but the number of files will be much 
lower.

OK, this low-resolution files needs to be updated every time you replace 
some of the ECW files, but it should be much faster to create a 10% 
resolution file than a full-resolution one. Using high values for 
"--config GDAL_CACHEMAX" with "gdal_translate" can speed up quite much 
merging the small files.


armin