[Gdal-dev] Advice on Raster Formats

[Bill Binko]

Thank you Frank, Ed and everyone else for your input.

As mentioned, perhaps I should back up and explain my problem a bit more.

I will have DOQQs for the entire state of Florida at 1m (or 3 ft,
depending on the area) accuracy.  Currently, I have 2/3 of the state, and 
it's taking about 30GB compressed, which is completely reasonable.  
However, with a 10:1 decompression rate (at least on the MrSIDS), the 
400GB needed for a full decompressed set (plus the overview size), it 
rapidly becomes untenable.

Certainly the hard-drive space can be overcome: 400GB drives can be had
for $229, and decent SCSI drives at that size are only twice that price.  
However, there are other issues w/the uncompressed files that are 
troublesome:

1) Backups become more difficult as you either need to a) compress the 
files or b) use very expensive tapes or c) use additional (portable) hard 
drives.

2) As my site will be colocated, I will not be able to update the DOQQs 
over the network, and will have to take a portable drive down to the colo 
center.

3) While the GeoTIFFs are far (far!) faster to load, when mapserver 
pointed to the limited set I created, I saw a marked increase in io 
utilization (which is I suppose expected).  With multiple concurrent 
users, I think this may move my bottleneck to the drives.

With all that said, I am certainly open to upgrading my drives if that's 
the best solution.

Now, as to the actual performance problem.

When I zoom into a DOQQ (to the street level or so), the performance is 
fine with both ECW and MrSIDs.  As I zoom out _even within the same DOQQ_, 
it rapidly slows down to the point of being unusable.  When I get to the 
point where I'm at four tiles, ECW is still standing (about 1 minute 
render time), and MrSID is a joke (I haven't waited for it).  I generally 
have to go kill the mapserv processes when I mistakenly start a 
large-scale render on the MrSIDs.

Now, I thought that I would create some overviews with gdaladdo, but that 
doesn't really help.  ECWs have "built-in" overviews that you can't seem 
to tweak, and MrSIDs are not writeable.  Perhaps there's a way to create 
external overviews for MrSIDs, but I have not found one.

So, my next approach was to create pyramid layers that the mapserver 
process would switch between as it zoomed in and out.  This would reduce 
both the decompression time and the resampling time.  So I started writing 
a script to build those pyramids using python and gdal.  I have made 
progress, but have run into some issues.  I can work past all of them (I 
think) with some help from you wonderful people, but I would like to know 
the target: which formats should I be creating these overview layers in.

I hope I haven't been too long-winded, and that this is a bit clearer.

Thanks again,
Bill

PS: regarding the "lossless" MrSIDs -- I am just quoting the agency I got 
the DOQQs from.  I can post the output of gdalinfo if you think it's 
pertinent.