[Gdal-dev] Advice on Raster Formats

[Jacob Delfos]

Hi Bill,

I am not an expert on this like most people on the list, but in my experience performance in mapserver is not really comparable to decompressing an entire image in its entirety. I suppose it depends on your system, whether harddisk speed is a bottleneck versus CPU. I actually get the best performance on my sites when I use ECW's (better than TIFF), whether my ECW's are big (2.5GB) or small (10MB). Probably because my bottleneck is the harddisk speed, not the CPU (Running a normal pc as server). I notice people running larger sites with many visitors prefer to reduce the load on the CPU by running fast harddisks with uncompressed TIFFs. So I guess you need to see whether you expect a lot of visitors (>100 / hour?). If so, your CPU may not deal well with compressed imagery, slowing down your site.

Hope this helps...

regards,

Jacob

 

» -----Original Message-----
» From: gdal-dev-bounces at lists.maptools.org 
» [mailto:gdal-dev-bounces at lists.maptools.org] On Behalf Of Bill Binko
» Sent: 30 August 2005 14:30
» To: GDAL Developer List
» Subject: [Gdal-dev] Advice on Raster Formats
» 
» Hello everyone,
» 
» I need some advice on how to store and process raster images 
» I'm working
» with.  The images have generally come in as either ECW (or 
» JP2000 -- my
» choice) or MrSID format.  They are aerial images (DOQQs generally at
» 1meter accuracy).
» 
» I need to be able to serve these up efficiently using 
» mapserver (through 
» GDAL, of course).  The output will be a 24 bit JPEG (with 
» overlay data 
» unfortunately compressed as well) that will be sent over the 
» wire to the 
» client.
» 
» My problem is that both the ECW and the MrSIDs take a very 
» long time to 
» decompress, but GeoTIFF takes an ungodly amount of disk 
» space.  Here are 
» some numbers I just grabbed at random:
» 
» I took the same data in GeoTIFF, ECW, and MrSID formats and 
» decompressed 
» them (to MEM which I thought would be a good comparison to 
» reading into 
» mapserver).
» 
» $ time gdal_translate -of MEM Q2918se.tiff /dev/null
» Input file size is 6337, 7082
» 0...10...20...30...40...50...60...70...80...90...100 - done.
» 0.72user 0.52system 0:02.79elapsed 44%CPU (0avgtext+0avgdata 
» 0maxresident)k 0inputs+0outputs (0major+37327minor)pagefaults 0swaps
» 
» $ time gdal_translate -of MEM Q2918se.sid /dev/null
» Input file size is 6337, 7082
» 0...10...20...30...40...50...60...70...80...90...100 - done.
» 55.80user 7.78system 1:12.57elapsed 87%CPU (0avgtext+0avgdata 
» 0maxresident)k 0inputs+0outputs (3major+631879minor)pagefaults 0swaps
» 
» $ time gdal_translate -of MEM Q2918se.ecw /dev/null
» Input file size is 6337, 7082
» 0...10...20...30...40...50...60...70...80...90...100 - done.
» 25.92user 1.43system 0:29.80elapsed 91%CPU (0avgtext+0avgdata 
» 0maxresident)k 0inputs+0outputs (0major+38181minor)pagefaults 0swaps
» 
» As you can see, MrSID is more than twice as slow as ECW, and 
» GeoTIFF just
» flies.  The flip side (disk space) can be seen here in a directory
» listing:
» 
»  19226192 Aug 30 02:05 Q2918se.ecw
»   7601858 Aug 30 02:21 Q2918se.sid
» 134983605 Aug 30 02:00 Q2918se.tiff
» 
» What I'm considering is keeping the full-scale files in MrSID 
» format, and
» keeping overviews (pyramids) of lower resolution images in 
» ECW.  Assuming
» I need four layers (dividing each side by 2 -- and the area 
» by 4) or so
» for adequate performance, the image above would take:
» 
» 7.6 (base) + 19/4MB (layer1) + 19/16MB (layer2)+ 19/64MB (layer3) + 
» 19/256MB (layer4).
» 
» That adds up to 13.9MB, a far cry from the 130MB TIFF.
» 
» This also gives me the ability to keep a lossless (MrSID) 
» file in case I 
» ever want to convert to GeoTIFFs in the future.
» 
» Does this make sense?  Are there better ways to get 
» performance out of 
» MrSIDs?  Should some of the higher (and smaller) layers stay GeoTIFF?
» 
» I'm sure that someone will say "Hard Drives are Cheap", and I 
» agree, but 
» it seems that this data is growing at outrageous rates 
» (uncompressed), and 
» that the time spent reading the entire GeoTIFF becomes a bottleneck 
» anyway.
» 
» I'd really appreciate any advice,
» 
» Thanks
» Bill
» 
» _______________________________________________
» Gdal-dev mailing list
» Gdal-dev at lists.maptools.org
» http://lists.maptools.org/mailman/listinfo/gdal-dev
» 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20050830/244a28b6/attachment.html