[gdal-dev] How create a simplified raster from an uge raster dataset

[Even Rouault]

Selon Andrea Peri <aperi2007 at gmail.com>:

> Hi,
> I have a huge set of tiffs rasters.
> Any single tiff is a geotiff tiled (256x256) and as an internal overview.
>
> To can use this raster dataset to lower scales (full extension) I need to
> create a single simplified tiff raster of all this dataset.
> Now I need to create a single simplified tiff raster .
>
> I don't know how is the better way to produce this simplified raster.
> This because the original huge set of rasters are about 2 TB. Every raster
> is about 15000 x 15000 px.
>
> SO I start creating a catalog named "cat_fogli.vrt" using gdalbuildvrt.
> And after this was create, I tryed to produce an external overview of this
> catalog.
> using this command:
>
> gdaladdo -r average -ro --config BIGTIFF_OVERVIEW YES cat_fogli.vrt 256 512
> 1024 2048 4096
>
> Unfortunately gdaladdo seem to be lock. It is take really many time and is
> always on 0%.
>
> I don't know if is only a bug time due to the huge size of original raster
> dataset or I'm using the wrong way.

I might not be surprised that it would really take a lot of time. You might want
to increase significatively the block cache to hopefully speed up things. You
can do that by specifying --config GDAL_CACHEMAX some_value_in_megabytes. See
http://trac.osgeo.org/gdal/wiki/ConfigOptions#GDAL_CACHEMAX for more details.
Note that if you want to use more than 2 gigabytes of RAM, you need a 64-bit
build of GDAL.

Another factor which might explain the slowness is the value of 256 as the first
overview level, which is really a huge value. Computing a block of 256x256 of
that level requires processing 65536x65536 pixels of the base image ! So I would
recommand specifying much lower values before reaching 256. For example 4 16 64.

I'm also skeptical on the fact that computing such huge overview levels will
lead to a visually acceptable result. Sometimes you want to use other source of
data than the high resolution data when displaying low resolution overviews.

(Ah, and using gdaltindex will not help for your use case. gdaltindex output is
only used by MapServer AFAIK.)

Best regards,

Even