[Gdal-dev] Advice on Raster Formats

Tue Aug 30 17:57:17 EDT 2005

Bill,

we made some tests with Quickbird and Ikonos images using ECW and 
MapServer. The ECW image size ranges from 100 MB to 1 GB. Putting the 
images in an image catalog and displaying them with Mapserver typically 
works very well. The biggest image catalog is made out of 80 
pan-sharpened Ikonos images that have about 12 GB altogether.

If you display all of them together at low scale (which I agree in 
principal makes no sense) still just takes some seconds on a standard 
server even with a not really fast disk array. The more you zoom in the 
faster the processing is, because the server just has to read a few 
images. Doing the same thing with TIFF (even with overviews created) was 
typically somewhat slower.

The ECW has big advantages if you can merge your images into just a few 
big files. If your data changes frequently then of course this approach 
is not the best for you. We created one big ECW from Landsat data for 
most parts of Europe. The size of the ECW is about 4 GB (compression 
ratio about 30 times). The visual quality is not much lower than the 
original files and a map creation with Mapserver takes 0.5 to 2 seconds.

I have to admit that there were some strange problems when zooming to 
higher scales in some parts of the big image. That made the server 
process crash. But I think that could be related to the ECW library 
(still using the 3.1 beta on Linux) and will hopefully resolved in newer 
versions.

I found it a very convenient functionality in newer Mapserver versions 
that you can specify an image catalog now more flexibly. We store all 
the image extents as polygons in PostGIS (instead of shapefiles) and the 
catalog in the map file just uses a filter expression to select the 
images we need from this catalog. I think the current documentation does 
not mention this definition but there was a mail last year on the 
Mapserver mailing list that explained it.

Armin

Bill Binko wrote:
> Hello everyone,
> 
> I need some advice on how to store and process raster images I'm working
> with.  The images have generally come in as either ECW (or JP2000 -- my
> choice) or MrSID format.  They are aerial images (DOQQs generally at
> 1meter accuracy).
> 
> I need to be able to serve these up efficiently using mapserver (through 
> GDAL, of course).  The output will be a 24 bit JPEG (with overlay data 
> unfortunately compressed as well) that will be sent over the wire to the 
> client.
> 
> My problem is that both the ECW and the MrSIDs take a very long time to 
> decompress, but GeoTIFF takes an ungodly amount of disk space.  Here are 
> some numbers I just grabbed at random:
> 
> I took the same data in GeoTIFF, ECW, and MrSID formats and decompressed 
> them (to MEM which I thought would be a good comparison to reading into 
> mapserver).
> 
> $ time gdal_translate -of MEM Q2918se.tiff /dev/null
> Input file size is 6337, 7082
> 0...10...20...30...40...50...60...70...80...90...100 - done.
> 0.72user 0.52system 0:02.79elapsed 44%CPU (0avgtext+0avgdata 
> 0maxresident)k 0inputs+0outputs (0major+37327minor)pagefaults 0swaps
> 
> $ time gdal_translate -of MEM Q2918se.sid /dev/null
> Input file size is 6337, 7082
> 0...10...20...30...40...50...60...70...80...90...100 - done.
> 55.80user 7.78system 1:12.57elapsed 87%CPU (0avgtext+0avgdata 
> 0maxresident)k 0inputs+0outputs (3major+631879minor)pagefaults 0swaps
> 
> $ time gdal_translate -of MEM Q2918se.ecw /dev/null
> Input file size is 6337, 7082
> 0...10...20...30...40...50...60...70...80...90...100 - done.
> 25.92user 1.43system 0:29.80elapsed 91%CPU (0avgtext+0avgdata 
> 0maxresident)k 0inputs+0outputs (0major+38181minor)pagefaults 0swaps
> 
> As you can see, MrSID is more than twice as slow as ECW, and GeoTIFF just
> flies.  The flip side (disk space) can be seen here in a directory
> listing:
> 
>  19226192 Aug 30 02:05 Q2918se.ecw
>   7601858 Aug 30 02:21 Q2918se.sid
> 134983605 Aug 30 02:00 Q2918se.tiff
> 
> What I'm considering is keeping the full-scale files in MrSID format, and
> keeping overviews (pyramids) of lower resolution images in ECW.  Assuming
> I need four layers (dividing each side by 2 -- and the area by 4) or so
> for adequate performance, the image above would take:
> 
> 7.6 (base) + 19/4MB (layer1) + 19/16MB (layer2)+ 19/64MB (layer3) + 
> 19/256MB (layer4).
> 
> That adds up to 13.9MB, a far cry from the 130MB TIFF.
> 
> This also gives me the ability to keep a lossless (MrSID) file in case I 
> ever want to convert to GeoTIFFs in the future.
> 
> Does this make sense?  Are there better ways to get performance out of 
> MrSIDs?  Should some of the higher (and smaller) layers stay GeoTIFF?
> 
> I'm sure that someone will say "Hard Drives are Cheap", and I agree, but 
> it seems that this data is growing at outrageous rates (uncompressed), and 
> that the time spent reading the entire GeoTIFF becomes a bottleneck 
> anyway.
> 
> I'd really appreciate any advice,
> 
> Thanks
> Bill
> 
> _______________________________________________
> Gdal-dev mailing list
> Gdal-dev at lists.maptools.org
> http://lists.maptools.org/mailman/listinfo/gdal-dev
> 
> 