[gdal-dev] gdaladdo creates very large files in 1.8 but not trunk
Eli Adam
eadam at co.lincoln.or.us
Fri Apr 15 05:01:12 EDT 2011
>> I'm not aware of any recent change in the GTiff driver that could have
>> "solved"
>> the issue you've seen. This is a bit surprising.
>>
>> Could you also try with the -stable builds (based on 1.8 branch) from
>> http://vbkto.dyndns.org/sdk/ to compare ?
>
> I am trying it with,
> http://vbkto.dyndns.org/sdk/PackageList.aspx?file=release-1500-gdal-1-8-mapserver-5-
> 6.zip
>
> Although not finished, based on progress so far, it looks to be working
> properly and on track to create a 3G ovr. That is consistent with trunk.
> I'll provide more information when it finishes.
Yes, this produced the expected results. Here is some information on that, (comments of creation included too)
E:\Orthos>dir mosaic_ycbcr_big*
04/06/2011 11:24 AM 7,780,934,363 mosaic_ycbcr_big.tif
04/11/2011 08:43 PM 3,262,697,247 mosaic_ycbcr_big.tif.ovr (created from svn a few days ago on ubuntu)
04/06/2011 11:24 AM 7,780,934,363 mosaic_ycbcr_big_1_8.tif
04/14/2011 09:47 PM 3,260,246,941 mosaic_ycbcr_big_1_8.tif.ovr (created from http://vbkto.dyndns.org/sdk/PackageList.aspx?file=release-1500-gdal-1-8-mapserver-5-6.zip downloaded today or yesterday)
04/06/2011 11:24 AM 7,780,934,363 mosaic_ycbcr_big_addo_v2.tif
04/07/2011 03:55 AM 385,580,179,214 mosaic_ycbcr_big_addo_v2.tif.ovr (Created from OSGEO4W 1.8.0)
04/06/2011 11:24 AM 7,780,934,363 mosaic_ycbcr_big_win.tif
04/12/2011 08:17 PM 3,260,246,941 mosaic_ycbcr_big_win.tif.ovr (created from a trunk version download from Tamas's site from a few days ago)
gdalinfo for mosaic_ycbcr_big.tif and mosaic_ycbcr_big_addo_v2.tif produces identical results (diff'ed). Here is one for reference.
Driver: GTiff/GeoTIFF
Files: mosaic_ycbcr_big_addo_v2.tif
mosaic_ycbcr_big_addo_v2.tif.ovr
Size is 141400, 280500
Coordinate System is:
PROJCS["NAD83(HARN) / Oregon North (ft)",
GEOGCS["NAD83(HARN)",
DATUM["NAD83_High_Accuracy_Regional_Network",
SPHEROID["GRS 1980",6378137,298.2572221010002,
AUTHORITY["EPSG","7019"]],
AUTHORITY["EPSG","6152"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4152"]],
PROJECTION["Lambert_Conformal_Conic_2SP"],
PARAMETER["standard_parallel_1",46],
PARAMETER["standard_parallel_2",44.33333333333334],
PARAMETER["latitude_of_origin",43.66666666666666],
PARAMETER["central_meridian",-120.5],
PARAMETER["false_easting",8202099.738],
PARAMETER["false_northing",0],
UNIT["foot",0.3048,
AUTHORITY["EPSG","9002"]],
AUTHORITY["EPSG","2913"]]
Origin = (7255000.000467186800000,522299.999999974850000)
Pixel Size = (1.000000000000018,-1.000000000000018)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
SOURCE_COLOR_SPACE=YCbCr
COMPRESSION=YCbCr JPEG
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 7255000.000, 522300.000) (124d 9'55.56"W, 45d 2'25.33"N)
Lower Left ( 7255000.000, 241800.000) (124d 7' 0.72"W, 44d16'18.24"N)
Upper Right ( 7396400.000, 522300.000) (123d37' 7.62"W, 45d 3'23.93"N)
Lower Right ( 7396400.000, 241800.000) (123d34'38.80"W, 44d17'16.07"N)
Center ( 7325700.000, 382050.000) (123d52'10.27"W, 44d39'52.08"N)
Band 1 Block=256x256 Type=Byte, ColorInterp=Red
Overviews: 70700x140250, 35350x70125, 17675x35063, 8838x17532, 4419x8766, 2210
x4383, 1105x2192
Band 2 Block=256x256 Type=Byte, ColorInterp=Green
Overviews: 70700x140250, 35350x70125, 17675x35063, 8838x17532, 4419x8766, 2210
x4383, 1105x2192
Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
Overviews: 70700x140250, 35350x70125, 17675x35063, 8838x17532, 4419x8766, 2210
x4383, 1105x2192
gdalinfo for mosaic_ycbcr_big.tif.ovr and mosaic_ycbcr_big_addo_v2.tif.ovr produces identical results. Here for reference too.
Driver: GTiff/GeoTIFF
Files: mosaic_ycbcr_big_addo_v2.tif.ovr
Size is 70700, 140250
Coordinate System is `'
Image Structure Metadata:
SOURCE_COLOR_SPACE=YCbCr
COMPRESSION=YCbCr JPEG
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 0.0, 0.0)
Lower Left ( 0.0,140250.0)
Upper Right (70700.0, 0.0)
Lower Right (70700.0,140250.0)
Center (35350.0,70125.0)
Band 1 Block=128x128 Type=Byte, ColorInterp=Red
Overviews: 35350x70125, 17675x35063, 8838x17532, 4419x8766, 2210x4383, 1105x21
92
Band 2 Block=128x128 Type=Byte, ColorInterp=Green
Overviews: 35350x70125, 17675x35063, 8838x17532, 4419x8766, 2210x4383, 1105x21
92
Band 3 Block=128x128 Type=Byte, ColorInterp=Blue
Overviews: 35350x70125, 17675x35063, 8838x17532, 4419x8766, 2210x4383, 1105x21
92
I can run more tests, although for results you will have to wait since it takes a long time to run these. Wait, this is easier, it is reproducible with OSGEO4W 1.8.0 and http://svn.osgeo.org/gdal/trunk/autotest/warp/data/test3658.tif (I don't know what this tif is designed to test, so hopefully it isn't causing problems.)
C:\Documents and Settings\eadam\Desktop>gdaladdo -ro test3658.tif 2
0...10...20...30...40...50...60...70...80...90...100 - done.
dir test*
04/15/2011 01:32 AM 8,071 test3658.tif
04/15/2011 01:32 AM 49,568 test3658.tif.ovr
Once again, I get normal sized results from trunk. I only get consistently unexpectedly large files in OSGEO4W 1.8. I do not get consistently expected or unexpected results from a recent 1.8 from Tamas's site. That may depend on whether it is a BIG_TIFF or not and whether the designation of BIG_TIFF was correct. I'll try looking into that more later.
Preliminarily it seems that with recent 1.8 from Tamas's site, the results are as expected when BIG_TIFF is specified and correct and otherwise, the results are unexpectedly large. I have some long running operations right now and need to reboot before looking into this more. It may be a few days for a reboot opportunity.
Bests, Eli
>
> Eli
>
>
>>
>>> This is an informational report only, there are not problems in trunk.
>>>
>>> Using 1.8.0 from OSGeo4W on XP, I made a mosaic of about 1200 uncompressed
>>> tifs totally 80G.
>>>
>>> gdalbuildvrt mosaic.vrt *.tif
>>>
>>> Then gdal_translated that to a compressed tif of about 8G
>>>
>>> gdal_translate mosaic.vrt mosaic_ycbcr_big.tif -co COMPRESS=JPEG -co
>>> PHOTOMETRIC=YCBCR -co TILED=YES -co BIGTIFF=YES --config GDAL_CACHEMAX 800
>>>
>>> Last step was to add some overviews:
>>>
>>> gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW
>>> YCBCR --config INTERLEAVE_OVERVIEW PIXEL --config BIGTIFF_OVERVIEW YES
>>> --config GDAL_CACHEMAX 800 -r average -ro mosaic_ycbcr_big.tif 2 4 8 16 32
>>> 64 128
>>>
>>> It works (at least in OpenEV it displays quickly like it is using
>>> overviews), except the file sizes for the overviews seem odd and too
>>> large.
>>>
>>> The tiles are about 80G, the result of gdal_translate is 8G, the result of
>>> gdaladdo, mosaic_ycbcr_big.tif.ovr, is 375G! That was using 1.8.0.
>>>
>>> I tested on trunk on ubuntu and XP from here, http://vbkto.dyndns.org/sdk/
>>> which produce reasonable results. The ovr is about 3G.
>>>
>>> A dif of gdalinfo on the .tif and .ovr are the same.
>>>
>>> (I hope that is not the result of a typo when I initially typed the
>>> command. I unfortunately don't have the shell around any more but took
>>> the above listed commands from some notes I took.)
>>>
>>> I have tried making the overviews internal and external with similar
>>> results. I also tried making the overviews on the uncompressed files (the
>>> vrt) with the same results. Then I made the overviews 1 level at a time
>>> and those results seemed reasonable.
>>>
>>> I didn't find any tickets about this in trac, but don't experience this in
>>> trunk so apparently it is fixed. Just reporting in case it is useful.
>>>
>>> Do you want any other information?
>>>
>>> Thanks, Eli
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> gdal-dev mailing list
>>> gdal-dev at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
More information about the gdal-dev
mailing list