[Benchmarking] Too much raster data?

[Andrea Aime]

Hi,
I had some concerns with the raster data size and
Michael helped me out double check that angle yesterday.

I'm in vacation sea side and I cannot find any server
admin I can get in touch with that would allow me to
use a torrent on their servers, but I had the strong
feeling 6GB compressed ECW were a bit too much.

So Michael took one of the ECW and converted it into
uncompressed TIFF, and then added the overviews.

gdal_translate  -of GTiff PNOA_MA_OF_ETRS89_HU31_h50_0118B.ecw
PNOA_MA_OF_ETRS89_HU31_h50_0118B.tif

gdaladdo -r average PNOA_MA_OF_ETRS89_HU31_h50_0118B.tif 2 4 8 16 32 64
128 256

Original size: 6.1gb
With Overviews: 8.1gb

Now, I also hear the ECW files are 74. If they are all same size 
(widthxheight) that will result in 600GB worth of TIFF files with
overviews.

Which is way too much. For the baseline test, assuming that everybody
can read bigtiff, we can have a single 600GB bigtiff file (ugh?) or
a mosaic of 8GB files (or something like that).

But when it comes to best effort testing every team will come up
with their customized format, which I expect to be often some
variant of an uncompressed image pyramid (and teams might want to
try out multiple variants, for example, disk based vs database
backed pyramid).
We simply don't have space to host 600*8 GB of data on the disks

The original target size was 100GB of imagery, which would be reached
by taking a subset of 12 files instead (a 4x3 block)

Soo... shall we select a subset of files?

Cheers
Andrea