[Gdal-dev] Advice on Raster Formats

[Frank Warmerdam]

On 8/30/05, Bill Binko <bill at binko.net> wrote:

> $ time gdal_translate -of MEM Q2918se.tiff /dev/null
> Input file size is 6337, 7082
> 0...10...20...30...40...50...60...70...80...90...100 - done.
> 0.72user 0.52system 0:02.79elapsed 44%CPU (0avgtext+0avgdata
> 0maxresident)k 0inputs+0outputs (0major+37327minor)pagefaults 0swaps
> 
> $ time gdal_translate -of MEM Q2918se.sid /dev/null
> Input file size is 6337, 7082
> 0...10...20...30...40...50...60...70...80...90...100 - done.
> 55.80user 7.78system 1:12.57elapsed 87%CPU (0avgtext+0avgdata
> 0maxresident)k 0inputs+0outputs (3major+631879minor)pagefaults 0swaps
> 
> $ time gdal_translate -of MEM Q2918se.ecw /dev/null
> Input file size is 6337, 7082
> 0...10...20...30...40...50...60...70...80...90...100 - done.
> 25.92user 1.43system 0:29.80elapsed 91%CPU (0avgtext+0avgdata
> 0maxresident)k 0inputs+0outputs (0major+38181minor)pagefaults 0swaps
> 
> As you can see, MrSID is more than twice as slow as ECW, and GeoTIFF just
> flies.  The flip side (disk space) can be seen here in a directory
> listing:

Bill, 

I don't think the above performance is particularly meaningful with
regard to how fast things would be when using MrSID (or ECW) in
MapServer.  In particular, conversion with gdal_translate generally
results in data from the source being requested one line at a time.
This will generally result in the ECW or MrSID drivers falling back to
a tile based strategy.  In the case of MrSID it will populate the tile
cache with 1024x128 blocks to satisfy requests.  Doing many smallish
requests like this is significantly slower with formats like MrSID, ECW
or JPEG2000 than doing one big pass through.  But because the
driver has trouble predicting what future requests will be made, I 
had to some up with some strategy.

In the case of MapServer though, we take care to do a single
big RasterIO() request so the MrSID, ECW or JP2 readers can
just make one windowed request for data (well sometimes 3
for RGB data since I am not savvy about doing all three bands
at once).  In short, MapServer may have substantially better
performance characteristics than you would expect from the
above numbers. 

I believe you have already tried directly to MrSID with MapServer
and got poor performance (as you showed me in IRC some weeks ago?).
Some of my other clients have also been somewhat disappointed with
performance and I have found performance a bit unreliable in my
local tests (though not too bad).  I'm not sure what to say.   My doing
three passes to get red, green and blue may be a significant factor.
It may also be that there is a non-trivial "connection time" to opening
the MrSIDs and that what we really need is an approach where the
MrSID file(s) are kept open between requests using FastCGI and 
some gdal file handle caching mechanism.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent