[gdal-dev] gdal create tif, errors with gdaladdo

Stephen Woodbridge woodbri at swoodbridge.com
Sat Mar 21 12:28:53 EDT 2009 

Even,

It belatedly dawned on me that just rebuilding the gdal package to use 
the internal libtiff would be MUCH easier the messing around with the 
libtiff packages.

In case anyone else needs to do this on debian:

cd /usr/src
apt-src install libgdal1-1.5.0
cd gdal-1.5.2
vi debian/rules
# added to configure:
#    --with-libtiff=internal
#    --with-ecw=no
#    --with-mrsid=no
#    --with-jp2mrsid=no
dpkg-buildpackage -rfakeroot
cd ..
sudo dpkg -i gdal-bin_1.5.2-3_amd64.deb libgdal-doc_1.5.2-3_all.deb 
libgdal-perl_1.5.2-3_amd64.deb libgdal1-1.5.0_1.5.2-3_amd64.deb 
libgdal1-dev_1.5.2-3_amd64.deb

I had to disable ecw and mrsid packages because I have them built as 
loadable modules using Alan Boudreault packages from mapgears.com 
otherwise the dpkg-buildpackage errored out because it found the 
libNCSUtil... which is not part of a package.

Anyway this builds and installs great and ldd /usr/bin/gdalinfo does not 
pull in the system libtiff. I'll test this as soon as I can delete anc 
copy, and run gdaladdo on about 51 GB of files. Hopefully I will not 
have issues to report :)

Thank you,
   -Steve

Even Rouault wrote:
> Steve,
> 
> yes you're looking at the correct site. None of those versions have been 
> released yet. But they are both in a state close to being released.
> 
> For the test, I didn't use the .tar.gz but the CVS version. 
> To get libtiff 3.9 CVS head (Extract from 
> http://www.remotesensing.org/libtiff/) :
> 
> export CVSROOT=:pserver:cvsanon at cvs.maptools.org:/cvs/maptools/cvsroot
> cvs login
> (use empty password)
> cvs checkout -r branch-3-9 libtiff
> 
> Note that you cannot use directly libtiff 4.0 as a replacement of the libtiff 
> 3.X series as there have been changes to the API/ABI that make it 
> incompatible with a GDAL compiled against libtiff 3.X. If you want to use 
> libtiff 4.0 (mainly to get BigTIFF support), you'd need to compile GDAL from 
> the source against it (or just build GDAL with internal libtiff support, 
> which has been sync'ed with libtiff 4.0 development since GDAL 1.5).
> 
> Even
> 
> Le Saturday 21 March 2009 04:10:18 Stephen Woodbridge, vous avez écrit :
>> Even,
>>
>> Thank you for all the extra work you did, I very much appreciate it.
>> Looks like I get to spend my weekend trying to rebuild the Lenny Source
>> package for libtiff based on 3.9.0.
>>
>> This is probably not the right list, but looking at:
>>
>> ftp://ftp.remotesensing.org/pub/libtiff/
>>
>> 3.9.0 looks to be beta since 7/13/2007
>> 4.0.0 looks to be beta since 1/21/2009
>>
>> Did these ever get released? Am I looking at the correct site?
>>
>> Thanks again for you assistance.
>>
>> -Steve
>>
>> Even Rouault wrote:
>>> Stephen,
>>>
>>> I've managed to reproduce your issue with GDAL 1.5 compiled with libtiff
>>> 3.8.2 (and I'm not so surprised, as I remember that it was a known
>>> issue). But when built with internal libtiff or against external libtiff
>>> 3.9.0 (CVS head version), it works fine. So it is due to some bug in
>>> libtiff 3.8.2. So you could either compile GDAL with internal libtiff
>>> support, or just build libtiff 3.9.0 and trick GDAL to link against it by
>>> using LD_LIBRARY_PATH. You'll need to create a symlink from the generated
>>> libtiff.so to libtiff.so.4 which is expected by the Debian libgdal.so.
>>> I've tried that and it works fine.
>>>
>>> Best regards,
>>>
>>> Even
>>>
>>> Le Friday 20 March 2009 04:43:39 Stephen Woodbridge, vous avez écrit :
>>>> Even,
>>>>
>>>> So I just happened to have made a backup copy of the tif files before
>>>> adding the overviews. And as a said, all the files without overviews did
>>>> not report errors with gdalinfo -checksum file.tif.  I also check all
>>>> the files with the overviews and some of them, not all, do report
>>>> errors. I'm seeing reported:
>>>>
>>>> Warning 1: TIFFReadDirectory:n-41-35_0-0.tif: Wrong "StripByteCounts"
>>>> field, ignoring and calculating from imagelength
>>>>
>>>> on some of the files that have overviews added. I went back and checked
>>>> output from the gdaladdo and it shows up in places there also.
>>>>
>>>> This is the file without overviews added:
>>>>
>>>> woodbri at mappy:/u/leaddog-200903013/af-bck$ gdalinfo -checksum
>>>> n-41-35_0-0.tif
>>>> Driver: GTiff/GeoTIFF
>>>> Files: n-41-35_0-0.tif
>>>> Size is 26688, 16853
>>>> Coordinate System is:
>>>> GEOGCS["WGS 84",
>>>>      DATUM["WGS_1984",
>>>>          SPHEROID["WGS 84",6378137,298.2572235630016,
>>>>              AUTHORITY["EPSG","7030"]],
>>>>          AUTHORITY["EPSG","6326"]],
>>>>      PRIMEM["Greenwich",0],
>>>>      UNIT["degree",0.0174532925199433],
>>>>      AUTHORITY["EPSG","4326"]]
>>>> Origin = (59.025339626823701,37.469898080618897)
>>>> Pixel Size = (0.000148923698721,-0.000148923698721)
>>>> Metadata:
>>>>    AREA_OR_POINT=Area
>>>> Image Structure Metadata:
>>>>    INTERLEAVE=PIXEL
>>>> Corner Coordinates:
>>>> Upper Left  (  59.0253396,  37.4698981) ( 59d 1'31.22"E, 37d28'11.63"N)
>>>> Lower Left  (  59.0253396,  34.9600870) ( 59d 1'31.22"E, 34d57'36.31"N)
>>>> Upper Right (  62.9998153,  37.4698981) ( 62d59'59.34"E, 37d28'11.63"N)
>>>> Lower Right (  62.9998153,  34.9600870) ( 62d59'59.34"E, 34d57'36.31"N)
>>>> Center      (  61.0125775,  36.2149925) ( 61d 0'45.28"E, 36d12'53.97"N)
>>>> Band 1 Block=256x256 Type=Byte, ColorInterp=Red
>>>>    Checksum=4400
>>>>    NoData Value=0
>>>> Band 2 Block=256x256 Type=Byte, ColorInterp=Green
>>>>    Checksum=18333
>>>>    NoData Value=0
>>>> Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
>>>>    Checksum=64262
>>>>    NoData Value=0
>>>>
>>>> And here is the samefile after adding the overviews:
>>>>
>>>> woodbri at mappy:/u/leaddog-200903013/af-bck$ cd ../af
>>>> woodbri at mappy:/u/leaddog-200903013/af$ gdalinfo -checksum
>>>> n-41-35_0-0.tif Warning 1: TIFFReadDirectory:n-41-35_0-0.tif: Wrong
>>>> "StripByteCounts" field, ignoring and calculating from imagelength
>>>> Driver: GTiff/GeoTIFF
>>>> Files: n-41-35_0-0.tif
>>>> Size is 26688, 16853
>>>> Coordinate System is:
>>>> GEOGCS["WGS 84",
>>>>      DATUM["WGS_1984",
>>>>          SPHEROID["WGS 84",6378137,298.2572235630016,
>>>>              AUTHORITY["EPSG","7030"]],
>>>>          AUTHORITY["EPSG","6326"]],
>>>>      PRIMEM["Greenwich",0],
>>>>      UNIT["degree",0.0174532925199433],
>>>>      AUTHORITY["EPSG","4326"]]
>>>> Origin = (59.025339626823701,37.469898080618897)
>>>> Pixel Size = (0.000148923698721,-0.000148923698721)
>>>> Metadata:
>>>>    AREA_OR_POINT=Area
>>>> Image Structure Metadata:
>>>>    INTERLEAVE=PIXEL
>>>> Corner Coordinates:
>>>> Upper Left  (  59.0253396,  37.4698981) ( 59d 1'31.22"E, 37d28'11.63"N)
>>>> Lower Left  (  59.0253396,  34.9600870) ( 59d 1'31.22"E, 34d57'36.31"N)
>>>> Upper Right (  62.9998153,  37.4698981) ( 62d59'59.34"E, 37d28'11.63"N)
>>>> Lower Right (  62.9998153,  34.9600870) ( 62d59'59.34"E, 34d57'36.31"N)
>>>> Center      (  61.0125775,  36.2149925) ( 61d 0'45.28"E, 36d12'53.97"N)
>>>> Band 1 Block=256x256 Type=Byte, ColorInterp=Red
>>>>    Checksum=4400
>>>>    NoData Value=0
>>>>    Overviews: 13344x8427, 6672x4214, 3336x2107, 1668x1054, 834x527,
>>>> 417x264, 209x132, 105x66
>>>> Band 2 Block=256x256 Type=Byte, ColorInterp=Green
>>>>    Checksum=18333
>>>>    NoData Value=0
>>>>    Overviews: 13344x8427, 6672x4214, 3336x2107, 1668x1054, 834x527,
>>>> 417x264, 209x132, 105x66
>>>> Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
>>>>    Checksum=64262
>>>>    NoData Value=0
>>>>    Overviews: 13344x8427, 6672x4214, 3336x2107, 1668x1054, 834x527,
>>>> 417x264, 209x132, 105x66
>>>> Warning 1: TIFFReadDirectory:n-41-35_0-0.tif: Wrong "StripByteCounts"
>>>> field, ignoring and calculating from imagelength
>>>>
>>>> Stephen Woodbridge wrote:
>>>>> Even,
>>>>>
>>>>> I ran gdalinfo -checksum on all 51GB of images and none of them
>>>>> reported any errors. And none of the images are compressed.
>>>>>
>>>>> I'm using Debian Lenny packages. There is an ldd output below.
>>>>>
>>>>> $ uname -a
>>>>> Linux mappy 2.6.23-1-amd64 #1 SMP Fri Oct 12 23:45:48 UTC 2007 x86_64
>>>>> GNU/Linux
>>>>>
>>>>> $ gdalinfo --version
>>>>> GDAL 1.5.2, released 2008/05/29
>>>>>
>>>>> $ apt-cache show libtiff4
>>>>> Package: libtiff4
>>>>> Priority: optional
>>>>> Section: libs
>>>>> Installed-Size: 484
>>>>> Maintainer: Jay Berkenbilt <qjb at debian.org>
>>>>> Architecture: amd64
>>>>> Source: tiff
>>>>> Version: 3.8.2-11
>>>>> Depends: libc6 (>= 2.7-1), libjpeg62, zlib1g (>= 1:1.1.4)
>>>>> Filename: pool/main/t/tiff/libtiff4_3.8.2-11_amd64.deb
>>>>> Size: 169372
>>>>> MD5sum: 8c0019eef3a752760dca8f14c80c3820
>>>>> SHA1: 3af8b1abde23f8b55eb840ff874cd0839d340e0f
>>>>> SHA256:
>>>>> e6d69f5e619e97bea1513f080750dc847eab04ef7f3eb9dc26985b2e2f3bcb56
>>>>> Description: Tag Image File Format (TIFF) library
>>>>>  libtiff is a library providing support for the Tag Image File Format
>>>>>  (TIFF), a widely used format for storing image data.  This package
>>>>>  includes the shared library.
>>>>> Homepage: http://libtiff.maptools.org
>>>>> Tag: role::shared-lib, works-with::image:raster
>>>>>
>>>>> A typical gdalinfo on one of the files is:
>>>>>
>>>>> $ gdalinfo -checksum n-41-25_0-0.tif
>>>>> Files: n-41-25_0-0.tif
>>>>> Size is 27192, 18091
>>>>> Coordinate System is:
>>>>> GEOGCS["WGS 84",
>>>>>     DATUM["WGS_1984",
>>>>>         SPHEROID["WGS 84",6378137,298.2572235630016,
>>>>>             AUTHORITY["EPSG","7030"]],
>>>>>         AUTHORITY["EPSG","6326"]],
>>>>>     PRIMEM["Greenwich",0],
>>>>>     UNIT["degree",0.0174532925199433],
>>>>>     AUTHORITY["EPSG","4326"]]
>>>>> Origin = (59.228401534356898,27.479955940770701)
>>>>> Pixel Size = (0.000138699063483,-0.000138699063483)
>>>>> Metadata:
>>>>>   AREA_OR_POINT=Area
>>>>> Image Structure Metadata:
>>>>>   INTERLEAVE=PIXEL
>>>>> Corner Coordinates:
>>>>> Upper Left  (  59.2284015,  27.4799559) ( 59d13'42.25"E, 27d28'47.84"N)
>>>>> Lower Left  (  59.2284015,  24.9707512) ( 59d13'42.25"E, 24d58'14.70"N)
>>>>> Upper Right (  62.9999065,  27.4799559) ( 62d59'59.66"E, 27d28'47.84"N)
>>>>> Lower Right (  62.9999065,  24.9707512) ( 62d59'59.66"E, 24d58'14.70"N)
>>>>> Center      (  61.1141540,  26.2253536) ( 61d 6'50.95"E, 26d13'31.27"N)
>>>>> Band 1 Block=256x256 Type=Byte, ColorInterp=Red
>>>>>   Checksum=29566
>>>>>   NoData Value=0
>>>>> Band 2 Block=256x256 Type=Byte, ColorInterp=Green
>>>>>   Checksum=34905
>>>>>   NoData Value=0
>>>>> Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
>>>>>   Checksum=24337
>>>>>   NoData Value=0
>>>>>
>>>>>
>>>>> woodbri at mappy:/u/leaddog-200903013$ ldd /usr/bin/gdalinfo
>>>>>         linux-vdso.so.1 =>  (0x00007fff937fe000)
>>>>>         libgdal1.5.0.so.1 => /usr/lib/libgdal1.5.0.so.1
>>>>> (0x00002ac017605000)
>>>>>         libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00002ac017dd4000)
>>>>>         libm.so.6 => /lib/libm.so.6 (0x00002ac0180e0000)
>>>>>         libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00002ac018364000)
>>>>>         libc.so.6 => /lib/libc.so.6 (0x00002ac01857b000)
>>>>>         libgeos_c.so.1 => /usr/lib/libgeos_c.so.1 (0x00002ac0188ce000)
>>>>>         libsqlite3.so.0 => /usr/lib/libsqlite3.so.0
>>>>> (0x00002ac018adf000) libodbc.so.1 => /usr/lib/libodbc.so.1
>>>>> (0x00002ac018d55000) libodbcinst.so.1 => /usr/lib/libodbcinst.so.1
>>>>> (0x00002ac018fb1000) libexpat.so.1 => /usr/lib/libexpat.so.1
>>>>> (0x00002ac0191be000) libxerces-c.so.28 => /usr/lib/libxerces-c.so.28
>>>>> (0x00002ac0193e7000)
>>>>>         libjasper.so.1 => /usr/lib/libjasper.so.1 (0x00002ac0199e0000)
>>>>>         libhdf5-1.6.6.so.0 => /usr/lib/libhdf5-1.6.6.so.0
>>>>> (0x00002ac019c3a000)
>>>>>         libmfhdf.so.4 => /usr/lib/libmfhdf.so.4 (0x00002ac019f61000)
>>>>>         libdf.so.4 => /usr/lib/libdf.so.4 (0x00002ac01a188000)
>>>>>         libogdi.so.3.2 => /usr/lib/libogdi.so.3.2 (0x00002ac01a43a000)
>>>>>         libgif.so.4 => /usr/lib/libgif.so.4 (0x00002ac01a65a000)
>>>>>         libjpeg.so.62 => /usr/lib/libjpeg.so.62 (0x00002ac01a862000)
>>>>>         libtiff.so.4 => /usr/lib/libtiff.so.4 (0x00002ac01aa85000)
>>>>>         libpng12.so.0 => /usr/lib/libpng12.so.0 (0x00002ac01ace1000)
>>>>>         libnetcdf.so.4 => /usr/lib/libnetcdf.so.4 (0x00002ac01af06000)
>>>>>         libpq.so.5 => /usr/lib/libpq.so.5 (0x00002ac01b13c000)
>>>>>         libz.so.1 => /usr/lib/libz.so.1 (0x00002ac01b360000)
>>>>>         libpthread.so.0 => /lib/libpthread.so.0 (0x00002ac01b577000)
>>>>>         librt.so.1 => /lib/librt.so.1 (0x00002ac01b794000)
>>>>>         libdl.so.2 => /lib/libdl.so.2 (0x00002ac01b99d000)
>>>>>         libcurl-gnutls.so.4 => /usr/lib/libcurl-gnutls.so.4
>>>>> (0x00002ac01bba1000)
>>>>>         libmysqlclient.so.15 => /usr/lib/libmysqlclient.so.15
>>>>> (0x00002ac01bde0000)
>>>>>         /lib64/ld-linux-x86-64.so.2 (0x00002ac0173e8000)
>>>>>         libgeos-3.0.0.so => /usr/lib/libgeos-3.0.0.so
>>>>> (0x00002ac01c1eb000) libltdl.so.3 => /usr/lib/libltdl.so.3
>>>>> (0x00002ac01c53a000) libicuuc.so.38 => /usr/lib/libicuuc.so.38
>>>>> (0x00002ac01c741000) libicudata.so.38 => /usr/lib/libicudata.so.38
>>>>> (0x00002ac01ca82000) libproj.so.0 => /usr/lib/libproj.so.0
>>>>> (0x00002ac01d759000) libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8
>>>>> (0x00002ac01d99a000) libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8
>>>>> (0x00002ac01dbec000)
>>>>>         libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x00002ac01df87000)
>>>>>         libcom_err.so.2 => /lib/libcom_err.so.2 (0x00002ac01e228000)
>>>>>         libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2
>>>>> (0x00002ac01e42c000)
>>>>>         libcrypt.so.1 => /lib/libcrypt.so.1 (0x00002ac01e658000)
>>>>>         libldap_r-2.4.so.2 => /usr/lib/libldap_r-2.4.so.2
>>>>> (0x00002ac01e890000)
>>>>>         libidn.so.11 => /usr/lib/libidn.so.11 (0x00002ac01eada000)
>>>>>         liblber-2.4.so.2 => /usr/lib/liblber-2.4.so.2
>>>>> (0x00002ac01ed0c000) libgnutls.so.26 => /usr/lib/libgnutls.so.26
>>>>> (0x00002ac01ef1b000) libnsl.so.1 => /lib/libnsl.so.1
>>>>> (0x00002ac01f1ce000) libk5crypto.so.3 => /usr/lib/libk5crypto.so.3
>>>>> (0x00002ac01f3e6000) libkrb5support.so.0 =>
>>>>> /usr/lib/libkrb5support.so.0 (0x00002ac01f60d000)
>>>>>         libkeyutils.so.1 => /lib/libkeyutils.so.1 (0x00002ac01f815000)
>>>>>         libresolv.so.2 => /lib/libresolv.so.2 (0x00002ac01fa17000)
>>>>>         libsasl2.so.2 => /usr/lib/libsasl2.so.2 (0x00002ac01fc2c000)
>>>>>         libtasn1.so.3 => /usr/lib/libtasn1.so.3 (0x00002ac01fe46000)
>>>>>         libgpg-error.so.0 => /usr/lib/libgpg-error.so.0
>>>>> (0x00002ac020056000)
>>>>>         libgcrypt.so.11 => /usr/lib/libgcrypt.so.11
>>>>> (0x00002ac02015a000)
>>>>>
>>>>> Even Rouault wrote:
>>>>>> Hi,
>>>>>>
>>>>>> no those warnings don't sound good at all ! The number of overviews
>>>>>> should be fine too.
>>>>>>
>>>>>> Could you precise the version of GDAL you're using and if you have
>>>>>> built with internal libtiff or external libtiff ?
>>>>>> (Such warnings could happen with older libtiff, especially with TIFF
>>>>>> compression)
>>>>>> Could you just check gdalinfo -checksum file.tif on your original file
>>>>>> before running gdaladdo and check that it doesn't throw similar errors
>>>>>> ?
>>>>>>
>>>>>> Best regards,
>>>>>> Even
>>>>>>
>>>>>> Le Thursday 19 March 2009 22:41:53 Stephen Woodbridge, vous avez 
> écrit :
>>>>>>> Hi,
>>>>>>>
>>>>>>> I just created a bunch of tif files from a mrsid file and when I use
>>>>>>> gdaladdo file.tif 2 4 8 16 32 64 128 256
>>>>>>>
>>>>>>> I get over a 1000 errors like:
>>>>>>>
>>>>>>> ERROR 1: n-41-25_0-0.tif:DumpModeDecode: Not enough data for scanline
>>>>>>> 1536
>>>>>>> ERROR 1: TIFFReadEncodedTile() failed.
>>>>>>> ERROR 1: IReadBlock failed at X offset 12, Y offset 7
>>>>>>> ERROR 1: GetBlockRef failed at X block offset 12, Y block offset 7
>>>>>>> ERROR 1: n-41-25_0-0.tif:DumpModeDecode: Not enough data for scanline
>>>>>>> 1664
>>>>>>> ERROR 1: TIFFReadEncodedTile() failed.
>>>>>>> ERROR 1: IReadBlock failed at X offset 13, Y offset 7
>>>>>>> ERROR 1: GetBlockRef failed at X block offset 13, Y block offset 7
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Is this because of the number of overviews I requested? ot something
>>>>>>> else? Is it safe to ignore these?
>>>>>>>
>>>>>>> -Steve
>>>>>>> _______________________________________________
>>>>>>> gdal-dev mailing list
>>>>>>> gdal-dev at lists.osgeo.org
>>>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>>> _______________________________________________
>>>>> gdal-dev mailing list
>>>>> gdal-dev at lists.osgeo.org
>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> 
>

More information about the gdal-dev mailing list