[gdal-dev] VRT of VRTs with overviews. They do not seem to be used.

Fri Aug 3 03:44:08 PDT 2018

Hi,

I have a use case where I would like to update some parts of my dataset
and not have to re-render all the overviews for all the other parts of
the globe. My idea was to use 5x5 degree boxes for which I would create
overviews of levels e.g. 2, 4, 8, 16. Then I would use a global VRT of
these VRT files to create the higher level overviews 32, 64 and so on.
So I I update a part of the dataset I only have to create the overviews
for the new part and the high level overviews which I thought would be
fast since the lower level overviews are already there.

But that does not seem to work since gdal seems to go back to the
original files even when the overviews are already there.

This simplified example might illustrate my problem:

This is the input file I took for illustration purposes

$ gdal-config --version
2.3.1

$ gdalinfo test.tif 
Driver: GTiff/GeoTIFF
Files: test.tif
Size is 1000, 1000
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (10.000000000000000,10.000000000000000)
Pixel Size = (0.010000000000000,-0.010000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  10.0000000,  10.0000000) ( 10d 0' 0.00"E, 10d 0' 0.00"N)
Lower Left  (  10.0000000,   0.0000000) ( 10d 0' 0.00"E,  0d 0' 0.01"N)
Upper Right (  20.0000000,  10.0000000) ( 20d 0' 0.00"E, 10d 0' 0.00"N)
Lower Right (  20.0000000,   0.0000000) ( 20d 0' 0.00"E,  0d 0' 0.01"N)
Center      (  15.0000000,   5.0000000) ( 15d 0' 0.00"E,  5d 0' 0.00"N)
Band 1 Block=256x256 Type=Float64, ColorInterp=Gray

$ gdalbuildvrt first_level.vrt test.tif 
$ gdaladdo test.tif 2 4 8 16
$ export CPL_DEBUG=ON
$ gdalbuildvrt second_level.vrt first_level.vrt
GNM: GNMRegisterAllInternal
GNM: RegisterGNMFile
GNM: RegisterGNMdatabase
0...10...20...30...40...50...60...70...80...90...100 - done.
GDAL: GDALOpen(first_level.vrt, this=0xb88140) succeeds as VRT.
GDAL: GDALClose(first_level.vrt, this=0xb88140)
GDAL: In GDALDestroy - unloading GDAL shared library.

$ gdaladdo second_level.vrt 32
GNM: GNMRegisterAllInternal
GNM: RegisterGNMFile
GNM: RegisterGNMdatabase
GDAL: GDALOpen(second_level.vrt, this=0x11990c0) succeeds as VRT.
GDAL: GDALDefaultOverviews::OverviewScan()
GDAL: GDALOpen(second_level.vrt.ovr, this=0x1159950) succeeds as GTiff.
GDAL: GDALDefaultOverviews::OverviewScan()
0GDAL: GDALOpen(first_level.vrt, this=0x11abd90) succeeds as VRT.
GDAL: GDALOpen(test.tif, this=0x11aef80) succeeds as GTiff.
GDAL: GDAL_CACHEMAX = 785 MB
...10...20...30...40...50...60...70...80...90...100 - done.
GDAL: GDALClose(second_level.vrt.ovr, this=0x1159950)
GDAL: GDALOpen(second_level.vrt.ovr, this=0x1159950) succeeds as GTiff.
GDAL: GDALDefaultOverviews::OverviewScan()
GDAL: GDALClose(second_level.vrt, this=0x11990c0)
GDAL: GDALClose(test.tif, this=0x11aef80)
GDAL: GDALClose(first_level.vrt, this=0x11abd90)
GDAL: GDALClose(second_level.vrt.ovr, this=0x1159950)
GDAL: In GDALDestroy - unloading GDAL shared library.

What is also strange that gdalinfo of the first_level.vrt file does not
list all the overviews that are there:

gdalinfo first_level.vrt 
GNM: GNMRegisterAllInternal
GNM: RegisterGNMFile
GNM: RegisterGNMdatabase
GDAL: GDALOpen(first_level.vrt, this=0xac2d20) succeeds as VRT.
Driver: VRT/Virtual Raster
GDAL: GDALDefaultOverviews::OverviewScan()
Files: first_level.vrt
       test.tif
Size is 1000, 1000
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (10.000000000000000,10.000000000000000)
Pixel Size = (0.010000000000000,-0.010000000000000)
OGRCT: PROJ >= 4.8.0 features enabled
OGRCT: Using locale-safe proj version
OGRCT: Source: +proj=longlat +datum=WGS84 +no_defs
OGRCT: Target: +proj=longlat +datum=WGS84 +no_defs
Corner Coordinates:
Upper Left  (  10.0000000,  10.0000000) ( 10d 0' 0.00"E, 10d 0' 0.00"N)
Lower Left  (  10.0000000,   0.0000000) ( 10d 0' 0.00"E,  0d 0' 0.01"N)
Upper Right (  20.0000000,  10.0000000) ( 20d 0' 0.00"E, 10d 0' 0.00"N)
Lower Right (  20.0000000,   0.0000000) ( 20d 0' 0.00"E,  0d 0' 0.01"N)
Center      (  15.0000000,   5.0000000) ( 15d 0' 0.00"E,  5d 0' 0.00"N)
Band 1 Block=128x128 Type=Float64, ColorInterp=Gray
GDAL: GDALOpen(test.tif, this=0xa83930) succeeds as GTiff.
GTiff: ScanDirectories()
GTiff: Opened 500x500 overview.
GTiff: Opened 250x250 overview.
GTiff: Opened 125x125 overview.
GTiff: Opened 63x63 overview.
  Overviews: 500x500, 250x250
GDAL: GDALClose(first_level.vrt, this=0xac2d20)
GDAL: GDALClose(test.tif, this=0xa83930)
GDAL: In GDALDestroy - unloading GDAL shared library.

All the best,
Christoph

-- 

-------------------------------------------------------
Christoph Paulik
VanderSat // Satellite observed water data. Globally. Daily.
Wilhelminastraat 43a, 2011 VK, Haarlem (NL), The Netherlands
W: www.vandersat.com
M: +31 (0) 6 1827 1928
PGP: 8CFC D7DF 2867 B2DC 749B  1B0A 6E3B A262 5186 A0AC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20180803/d58727b8/attachment-0001.sig>