Image formats: Performance Testing: Test 1

[Gregor Mosheh]

A customer asked me to do some tests on TIFF, JPEG2000, and ECW, and 
rate their relative merits in terms of disk usage, performance/speed, 
and image quality.

I can't say that the results are surprising, but at least I have some 
procedures and empirical numbers to back it up. For the impatient ones, 
skip to the Results section at the end. :)

I will be repeating this with larger files, as well as testing the 
effects of overviews and internal tiling, on my own time. This test of 
200 MB files was intended to represent the customer's typical use case.


*** DATA AND SETUP

The dataset used for this test is infrared false-color imagery of 
Eureka, California.
The pixel size of the data is 10684x6983 pixels.
The spatial extent of the data is:
    -124.1987173 40.7379805 -124.0836036 40.8132182

The original TIFF was reprojected from UTM to geographic using gdalwarp.
It was then converted using gdal_translate to generate the ECW and 
JPEG2000 versions.
As such, all images are identical data and all are in the same projection.

A mapfile was created listing three layers, named ECW, TIFF, and 
JPEG2000, each reading from the corresponding data source.


*** TESTING

I chose to use the shp2img program, rather than web-based rendering 
using OpenLayers or Mapscript, in order to eliminate many of the 
variables associated with timing, e.g. network lag and delays within the 
client.

I chose PNG-24 as the output format since this is what we ually use, for 
its combination of quality and color fdelity

The commands, runtimes, and URLs of the resulting images are as follows:

shp2img -m mapfile.map -l TIFF -o tiff-fullview.png
Time: 0.725s
https://www.cartograph.com/~gregor/formats/tiff-fullview.png

shp2img -m mapfile.map -l ECW -o ecw-fullview.png
Time: 3.069s
https://www.cartograph.com/~gregor/formats/ecw-fullview.png

shp2img -m mapfile.map -l JPEG2000 -o jpeg-fullview.png
Time: 6.071s
https://www.cartograph.com/~gregor/formats/jpeg-fullview.png

shp2img -m mapfile.map -l TIFF -e -124.1987173 40.7379805 -124.14116045 
40.77559935 -o tiff-quad.png
Time: 0.833s
https://www.cartograph.com/~gregor/formats/tiff-quad.png

shp2img -m mapfile.map -l ECW -e -124.1987173 40.7379805 -124.14116045 
40.77559935 -o ecw-quad.png
Time: 3.070s
https://www.cartograph.com/~gregor/formats/ecw-quad.png

shp2img -m mapfile.map -l JPEG2000 -e -124.1987173 40.7379805 
-124.14116045 40.77559935 -o jpeg-quad.png
Time: 9.073s
https://www.cartograph.com/~gregor/formats/jpeg-quad.png



*** RESULTS

Storage space:

TIFF: 213 MB, ECW: 60 MB, JPEG2000: 87 MB

Both ECW and JPEG offered sustantial reductions in storage space 
requirements, ECW best of all.

Rendering speed:

TIFF was the fastest to render, while ECW was significantly slower. 
JPEG2000 was very slow. When seeking a subset of the data, these results 
continued, though JPEG2000 showed a very significant slowdown on this 
very common task.

Image quality:

The original photo had a grainy quality to it. Although unattractive to 
the eye, these grains do represent fine-detail data, useful for 
analyzing the loss of such quality.

As expected, TIFF retained all of the fine detail. ECW also showed high 
fidelity to the original image. JPEG2000 showed the most loss of the 
grainy detail, although in this case it did make for a "smoothing" 
effect which was not unattractive.


*** CONCLUSION

ECW and JPEG2000 both provide very good compression, resulting in a 65% 
reduction in storage requirements.

Both ECW and JPEG2000 showed some loss of image quality, but the results 
are quite acceptable considering the reduction in storage space.

At the file sizes which we typically use (up to 300 MB TIFFs), both ECW 
and JPEG2000 are probably too slow for our needs. With very large TIFFs, 
the disk access can often outweight the CPU speed and cause TIFF to be 
slower than other formats, but in the 200 MB size range, this is not the 
case.