[gdal-dev] Scale dependent VRT for overviews

[Jukka Rahkonen]

Hi,

This is a long letter and finally it will lead to a suggestion about
implementing some kind of a Virtual Overview (VOV) system for organising a
number of GDAL datasets into a one top level dataset in such a way that GDAL can
automatically select the best source for the requested scale. Same thing is used
in Mapserver through a layer group and scale dependent layers.
---

I noticed today that I had same problem than Andrea Peri some time ago
http://article.gmane.org/gmane.comp.gis.gdal.devel/31177

I had built an biggish image mosaic as VRT. Size of the mosaic was 288000 by
480000 pixels and it was combined fron 400 paletted raster maps (8-bit with a
palette. Naturally I wanted to build some overviews for the VRT and naturally I
started to create them with gdaladdo.  After two hours it was still showing me
zero percant progress. I stopped the job, read Andrea's mail and concluded that
perhaps gdaladdo does not work very well with big VRT files.

I started to think about alternative methods. ECW and JPEG2000 would be pretty
good because they offer overviews for free. Unfortunately ECW itself is not free
and free GDAL JPEG2000 drivers are not fast enough. And even I can mysself write
JPEG2000 with a good licenced SDK our users do not have licensed readers and
slow JPEG2000 drivers are slow in reading too. Thus I will keep on waiting for a
JPEG2000 driver that is at the same time flaming fast AND open.

Then I had a try with dropping the resolution of VRT into half by using
gdalwarp. It managed to convert the VRT into GeoTIFF with double sized pixels in
three hours. Gdalwarp gives also an option to select the resampling method which
is nice sometimes. Gdaladdo has that option too but you may remember that the
program itself seems to be unusable for this purpose.

Three hours felt still rather slow. Then I measured how long it would take to
convert the individual images forming up the VRT with gdal_translate and
-outsize 50% 50%. It took only half on hour. Running gdalbuildvrt took a few
seconds more and I had the first reduced resolution level ready.

First run with gdal_translate gave reasonably good quality but the next reduced
level looked very bad. Several nearest neighbour samplings in a row is obviously
not so good method. I had now paletted raster maps and they do not tolerate
subsampling well with any method. I made some more tests and found that
converting images one be one with gdalwarp was a suitable compromise for me. It
was slower than gdal_translate but gave better looking result and converting all
individual files one by one was faster than converting the VRT as a whole.

After playing with a whole many variations with gdal_translate and gdalwarp,
including some convertions of subareas frpom the VRT for getting bigger tiles
and less files for low resolution scales I had a nice set in my hands. I had no
idea about how to ulilise them as a one GDAL data set but because I have some
experience on Mapserver I just gathered the images for each resolution level
with gdaltindex and then build a scale dependent WMS layer group and all the GIS
programs consuming WMS are happy. However it would be awfull to teach the
process for someone and even I can make a well working WMS service for our
customers I cannot deliver the data which is an odd mess now and only usable as
a one data source for Mapserver.

Now finally to my suggestion. To me GDAL virtual format is quite a lot similar
than Mapserver tileindex, though much more feature rich. But with Mapserver it
is very easy to combine layers into scale dependent layer groups be setting
suitable MINSCALEDENOM and MAXSCALEDENOM values for the individual layers.

Dear GDAL developers, I hope you can find some nice place inside the VRT XML
file for setting MINSCALEDEMON and MAXSCALEDENOM for "SimpleSource" or for some
new "SimpleSourceGroup" or whatever you feel suitable.

Regards,


-Jukka Rahkonen-