[gdal-dev] gdal create tif, errors with gdaladdo
Stephen Woodbridge
woodbri at swoodbridge.com
Sat Mar 21 20:33:55 EDT 2009
Just to report back the fix below solved my problem. Reprocessed all the
imagery and no errors reported, and no nasty artifacts on the images.
Many thank to Even for his copious help and to the whole GDAL team.
-Steve
Stephen Woodbridge wrote:
> Even,
>
> It belatedly dawned on me that just rebuilding the gdal package to use
> the internal libtiff would be MUCH easier the messing around with the
> libtiff packages.
>
> In case anyone else needs to do this on debian:
>
> cd /usr/src
> apt-src install libgdal1-1.5.0
> cd gdal-1.5.2
> vi debian/rules
> # added to configure:
> # --with-libtiff=internal
> # --with-ecw=no
> # --with-mrsid=no
> # --with-jp2mrsid=no
> dpkg-buildpackage -rfakeroot
> cd ..
> sudo dpkg -i gdal-bin_1.5.2-3_amd64.deb libgdal-doc_1.5.2-3_all.deb
> libgdal-perl_1.5.2-3_amd64.deb libgdal1-1.5.0_1.5.2-3_amd64.deb
> libgdal1-dev_1.5.2-3_amd64.deb
>
> I had to disable ecw and mrsid packages because I have them built as
> loadable modules using Alan Boudreault packages from mapgears.com
> otherwise the dpkg-buildpackage errored out because it found the
> libNCSUtil... which is not part of a package.
>
> Anyway this builds and installs great and ldd /usr/bin/gdalinfo does not
> pull in the system libtiff. I'll test this as soon as I can delete anc
> copy, and run gdaladdo on about 51 GB of files. Hopefully I will not
> have issues to report :)
>
> Thank you,
> -Steve
>
> Even Rouault wrote:
>> Steve,
>>
>> yes you're looking at the correct site. None of those versions have
>> been released yet. But they are both in a state close to being released.
>>
>> For the test, I didn't use the .tar.gz but the CVS version. To get
>> libtiff 3.9 CVS head (Extract from
>> http://www.remotesensing.org/libtiff/) :
>>
>> export CVSROOT=:pserver:cvsanon at cvs.maptools.org:/cvs/maptools/cvsroot
>> cvs login
>> (use empty password)
>> cvs checkout -r branch-3-9 libtiff
>>
>> Note that you cannot use directly libtiff 4.0 as a replacement of the
>> libtiff 3.X series as there have been changes to the API/ABI that make
>> it incompatible with a GDAL compiled against libtiff 3.X. If you want
>> to use libtiff 4.0 (mainly to get BigTIFF support), you'd need to
>> compile GDAL from the source against it (or just build GDAL with
>> internal libtiff support, which has been sync'ed with libtiff 4.0
>> development since GDAL 1.5).
>>
>> Even
>>
>> Le Saturday 21 March 2009 04:10:18 Stephen Woodbridge, vous avez écrit :
>>> Even,
>>>
>>> Thank you for all the extra work you did, I very much appreciate it.
>>> Looks like I get to spend my weekend trying to rebuild the Lenny Source
>>> package for libtiff based on 3.9.0.
>>>
>>> This is probably not the right list, but looking at:
>>>
>>> ftp://ftp.remotesensing.org/pub/libtiff/
>>>
>>> 3.9.0 looks to be beta since 7/13/2007
>>> 4.0.0 looks to be beta since 1/21/2009
>>>
>>> Did these ever get released? Am I looking at the correct site?
>>>
>>> Thanks again for you assistance.
>>>
>>> -Steve
>>>
>>> Even Rouault wrote:
>>>> Stephen,
>>>>
>>>> I've managed to reproduce your issue with GDAL 1.5 compiled with
>>>> libtiff
>>>> 3.8.2 (and I'm not so surprised, as I remember that it was a known
>>>> issue). But when built with internal libtiff or against external
>>>> libtiff
>>>> 3.9.0 (CVS head version), it works fine. So it is due to some bug in
>>>> libtiff 3.8.2. So you could either compile GDAL with internal libtiff
>>>> support, or just build libtiff 3.9.0 and trick GDAL to link against
>>>> it by
>>>> using LD_LIBRARY_PATH. You'll need to create a symlink from the
>>>> generated
>>>> libtiff.so to libtiff.so.4 which is expected by the Debian libgdal.so.
>>>> I've tried that and it works fine.
>>>>
>>>> Best regards,
>>>>
>>>> Even
>>>>
>>>> Le Friday 20 March 2009 04:43:39 Stephen Woodbridge, vous avez écrit :
>>>>> Even,
>>>>>
>>>>> So I just happened to have made a backup copy of the tif files before
>>>>> adding the overviews. And as a said, all the files without
>>>>> overviews did
>>>>> not report errors with gdalinfo -checksum file.tif. I also check all
>>>>> the files with the overviews and some of them, not all, do report
>>>>> errors. I'm seeing reported:
>>>>>
>>>>> Warning 1: TIFFReadDirectory:n-41-35_0-0.tif: Wrong "StripByteCounts"
>>>>> field, ignoring and calculating from imagelength
>>>>>
>>>>> on some of the files that have overviews added. I went back and
>>>>> checked
>>>>> output from the gdaladdo and it shows up in places there also.
>>>>>
>>>>> This is the file without overviews added:
>>>>>
>>>>> woodbri at mappy:/u/leaddog-200903013/af-bck$ gdalinfo -checksum
>>>>> n-41-35_0-0.tif
>>>>> Driver: GTiff/GeoTIFF
>>>>> Files: n-41-35_0-0.tif
>>>>> Size is 26688, 16853
>>>>> Coordinate System is:
>>>>> GEOGCS["WGS 84",
>>>>> DATUM["WGS_1984",
>>>>> SPHEROID["WGS 84",6378137,298.2572235630016,
>>>>> AUTHORITY["EPSG","7030"]],
>>>>> AUTHORITY["EPSG","6326"]],
>>>>> PRIMEM["Greenwich",0],
>>>>> UNIT["degree",0.0174532925199433],
>>>>> AUTHORITY["EPSG","4326"]]
>>>>> Origin = (59.025339626823701,37.469898080618897)
>>>>> Pixel Size = (0.000148923698721,-0.000148923698721)
>>>>> Metadata:
>>>>> AREA_OR_POINT=Area
>>>>> Image Structure Metadata:
>>>>> INTERLEAVE=PIXEL
>>>>> Corner Coordinates:
>>>>> Upper Left ( 59.0253396, 37.4698981) ( 59d 1'31.22"E,
>>>>> 37d28'11.63"N)
>>>>> Lower Left ( 59.0253396, 34.9600870) ( 59d 1'31.22"E,
>>>>> 34d57'36.31"N)
>>>>> Upper Right ( 62.9998153, 37.4698981) ( 62d59'59.34"E,
>>>>> 37d28'11.63"N)
>>>>> Lower Right ( 62.9998153, 34.9600870) ( 62d59'59.34"E,
>>>>> 34d57'36.31"N)
>>>>> Center ( 61.0125775, 36.2149925) ( 61d 0'45.28"E,
>>>>> 36d12'53.97"N)
>>>>> Band 1 Block=256x256 Type=Byte, ColorInterp=Red
>>>>> Checksum=4400
>>>>> NoData Value=0
>>>>> Band 2 Block=256x256 Type=Byte, ColorInterp=Green
>>>>> Checksum=18333
>>>>> NoData Value=0
>>>>> Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
>>>>> Checksum=64262
>>>>> NoData Value=0
>>>>>
>>>>> And here is the samefile after adding the overviews:
>>>>>
>>>>> woodbri at mappy:/u/leaddog-200903013/af-bck$ cd ../af
>>>>> woodbri at mappy:/u/leaddog-200903013/af$ gdalinfo -checksum
>>>>> n-41-35_0-0.tif Warning 1: TIFFReadDirectory:n-41-35_0-0.tif: Wrong
>>>>> "StripByteCounts" field, ignoring and calculating from imagelength
>>>>> Driver: GTiff/GeoTIFF
>>>>> Files: n-41-35_0-0.tif
>>>>> Size is 26688, 16853
>>>>> Coordinate System is:
>>>>> GEOGCS["WGS 84",
>>>>> DATUM["WGS_1984",
>>>>> SPHEROID["WGS 84",6378137,298.2572235630016,
>>>>> AUTHORITY["EPSG","7030"]],
>>>>> AUTHORITY["EPSG","6326"]],
>>>>> PRIMEM["Greenwich",0],
>>>>> UNIT["degree",0.0174532925199433],
>>>>> AUTHORITY["EPSG","4326"]]
>>>>> Origin = (59.025339626823701,37.469898080618897)
>>>>> Pixel Size = (0.000148923698721,-0.000148923698721)
>>>>> Metadata:
>>>>> AREA_OR_POINT=Area
>>>>> Image Structure Metadata:
>>>>> INTERLEAVE=PIXEL
>>>>> Corner Coordinates:
>>>>> Upper Left ( 59.0253396, 37.4698981) ( 59d 1'31.22"E,
>>>>> 37d28'11.63"N)
>>>>> Lower Left ( 59.0253396, 34.9600870) ( 59d 1'31.22"E,
>>>>> 34d57'36.31"N)
>>>>> Upper Right ( 62.9998153, 37.4698981) ( 62d59'59.34"E,
>>>>> 37d28'11.63"N)
>>>>> Lower Right ( 62.9998153, 34.9600870) ( 62d59'59.34"E,
>>>>> 34d57'36.31"N)
>>>>> Center ( 61.0125775, 36.2149925) ( 61d 0'45.28"E,
>>>>> 36d12'53.97"N)
>>>>> Band 1 Block=256x256 Type=Byte, ColorInterp=Red
>>>>> Checksum=4400
>>>>> NoData Value=0
>>>>> Overviews: 13344x8427, 6672x4214, 3336x2107, 1668x1054, 834x527,
>>>>> 417x264, 209x132, 105x66
>>>>> Band 2 Block=256x256 Type=Byte, ColorInterp=Green
>>>>> Checksum=18333
>>>>> NoData Value=0
>>>>> Overviews: 13344x8427, 6672x4214, 3336x2107, 1668x1054, 834x527,
>>>>> 417x264, 209x132, 105x66
>>>>> Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
>>>>> Checksum=64262
>>>>> NoData Value=0
>>>>> Overviews: 13344x8427, 6672x4214, 3336x2107, 1668x1054, 834x527,
>>>>> 417x264, 209x132, 105x66
>>>>> Warning 1: TIFFReadDirectory:n-41-35_0-0.tif: Wrong "StripByteCounts"
>>>>> field, ignoring and calculating from imagelength
>>>>>
>>>>> Stephen Woodbridge wrote:
>>>>>> Even,
>>>>>>
>>>>>> I ran gdalinfo -checksum on all 51GB of images and none of them
>>>>>> reported any errors. And none of the images are compressed.
>>>>>>
>>>>>> I'm using Debian Lenny packages. There is an ldd output below.
>>>>>>
>>>>>> $ uname -a
>>>>>> Linux mappy 2.6.23-1-amd64 #1 SMP Fri Oct 12 23:45:48 UTC 2007 x86_64
>>>>>> GNU/Linux
>>>>>>
>>>>>> $ gdalinfo --version
>>>>>> GDAL 1.5.2, released 2008/05/29
>>>>>>
>>>>>> $ apt-cache show libtiff4
>>>>>> Package: libtiff4
>>>>>> Priority: optional
>>>>>> Section: libs
>>>>>> Installed-Size: 484
>>>>>> Maintainer: Jay Berkenbilt <qjb at debian.org>
>>>>>> Architecture: amd64
>>>>>> Source: tiff
>>>>>> Version: 3.8.2-11
>>>>>> Depends: libc6 (>= 2.7-1), libjpeg62, zlib1g (>= 1:1.1.4)
>>>>>> Filename: pool/main/t/tiff/libtiff4_3.8.2-11_amd64.deb
>>>>>> Size: 169372
>>>>>> MD5sum: 8c0019eef3a752760dca8f14c80c3820
>>>>>> SHA1: 3af8b1abde23f8b55eb840ff874cd0839d340e0f
>>>>>> SHA256:
>>>>>> e6d69f5e619e97bea1513f080750dc847eab04ef7f3eb9dc26985b2e2f3bcb56
>>>>>> Description: Tag Image File Format (TIFF) library
>>>>>> libtiff is a library providing support for the Tag Image File Format
>>>>>> (TIFF), a widely used format for storing image data. This package
>>>>>> includes the shared library.
>>>>>> Homepage: http://libtiff.maptools.org
>>>>>> Tag: role::shared-lib, works-with::image:raster
>>>>>>
>>>>>> A typical gdalinfo on one of the files is:
>>>>>>
>>>>>> $ gdalinfo -checksum n-41-25_0-0.tif
>>>>>> Files: n-41-25_0-0.tif
>>>>>> Size is 27192, 18091
>>>>>> Coordinate System is:
>>>>>> GEOGCS["WGS 84",
>>>>>> DATUM["WGS_1984",
>>>>>> SPHEROID["WGS 84",6378137,298.2572235630016,
>>>>>> AUTHORITY["EPSG","7030"]],
>>>>>> AUTHORITY["EPSG","6326"]],
>>>>>> PRIMEM["Greenwich",0],
>>>>>> UNIT["degree",0.0174532925199433],
>>>>>> AUTHORITY["EPSG","4326"]]
>>>>>> Origin = (59.228401534356898,27.479955940770701)
>>>>>> Pixel Size = (0.000138699063483,-0.000138699063483)
>>>>>> Metadata:
>>>>>> AREA_OR_POINT=Area
>>>>>> Image Structure Metadata:
>>>>>> INTERLEAVE=PIXEL
>>>>>> Corner Coordinates:
>>>>>> Upper Left ( 59.2284015, 27.4799559) ( 59d13'42.25"E,
>>>>>> 27d28'47.84"N)
>>>>>> Lower Left ( 59.2284015, 24.9707512) ( 59d13'42.25"E,
>>>>>> 24d58'14.70"N)
>>>>>> Upper Right ( 62.9999065, 27.4799559) ( 62d59'59.66"E,
>>>>>> 27d28'47.84"N)
>>>>>> Lower Right ( 62.9999065, 24.9707512) ( 62d59'59.66"E,
>>>>>> 24d58'14.70"N)
>>>>>> Center ( 61.1141540, 26.2253536) ( 61d 6'50.95"E,
>>>>>> 26d13'31.27"N)
>>>>>> Band 1 Block=256x256 Type=Byte, ColorInterp=Red
>>>>>> Checksum=29566
>>>>>> NoData Value=0
>>>>>> Band 2 Block=256x256 Type=Byte, ColorInterp=Green
>>>>>> Checksum=34905
>>>>>> NoData Value=0
>>>>>> Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
>>>>>> Checksum=24337
>>>>>> NoData Value=0
>>>>>>
>>>>>>
>>>>>> woodbri at mappy:/u/leaddog-200903013$ ldd /usr/bin/gdalinfo
>>>>>> linux-vdso.so.1 => (0x00007fff937fe000)
>>>>>> libgdal1.5.0.so.1 => /usr/lib/libgdal1.5.0.so.1
>>>>>> (0x00002ac017605000)
>>>>>> libstdc++.so.6 => /usr/lib/libstdc++.so.6
>>>>>> (0x00002ac017dd4000)
>>>>>> libm.so.6 => /lib/libm.so.6 (0x00002ac0180e0000)
>>>>>> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00002ac018364000)
>>>>>> libc.so.6 => /lib/libc.so.6 (0x00002ac01857b000)
>>>>>> libgeos_c.so.1 => /usr/lib/libgeos_c.so.1
>>>>>> (0x00002ac0188ce000)
>>>>>> libsqlite3.so.0 => /usr/lib/libsqlite3.so.0
>>>>>> (0x00002ac018adf000) libodbc.so.1 => /usr/lib/libodbc.so.1
>>>>>> (0x00002ac018d55000) libodbcinst.so.1 => /usr/lib/libodbcinst.so.1
>>>>>> (0x00002ac018fb1000) libexpat.so.1 => /usr/lib/libexpat.so.1
>>>>>> (0x00002ac0191be000) libxerces-c.so.28 => /usr/lib/libxerces-c.so.28
>>>>>> (0x00002ac0193e7000)
>>>>>> libjasper.so.1 => /usr/lib/libjasper.so.1
>>>>>> (0x00002ac0199e0000)
>>>>>> libhdf5-1.6.6.so.0 => /usr/lib/libhdf5-1.6.6.so.0
>>>>>> (0x00002ac019c3a000)
>>>>>> libmfhdf.so.4 => /usr/lib/libmfhdf.so.4 (0x00002ac019f61000)
>>>>>> libdf.so.4 => /usr/lib/libdf.so.4 (0x00002ac01a188000)
>>>>>> libogdi.so.3.2 => /usr/lib/libogdi.so.3.2
>>>>>> (0x00002ac01a43a000)
>>>>>> libgif.so.4 => /usr/lib/libgif.so.4 (0x00002ac01a65a000)
>>>>>> libjpeg.so.62 => /usr/lib/libjpeg.so.62 (0x00002ac01a862000)
>>>>>> libtiff.so.4 => /usr/lib/libtiff.so.4 (0x00002ac01aa85000)
>>>>>> libpng12.so.0 => /usr/lib/libpng12.so.0 (0x00002ac01ace1000)
>>>>>> libnetcdf.so.4 => /usr/lib/libnetcdf.so.4
>>>>>> (0x00002ac01af06000)
>>>>>> libpq.so.5 => /usr/lib/libpq.so.5 (0x00002ac01b13c000)
>>>>>> libz.so.1 => /usr/lib/libz.so.1 (0x00002ac01b360000)
>>>>>> libpthread.so.0 => /lib/libpthread.so.0 (0x00002ac01b577000)
>>>>>> librt.so.1 => /lib/librt.so.1 (0x00002ac01b794000)
>>>>>> libdl.so.2 => /lib/libdl.so.2 (0x00002ac01b99d000)
>>>>>> libcurl-gnutls.so.4 => /usr/lib/libcurl-gnutls.so.4
>>>>>> (0x00002ac01bba1000)
>>>>>> libmysqlclient.so.15 => /usr/lib/libmysqlclient.so.15
>>>>>> (0x00002ac01bde0000)
>>>>>> /lib64/ld-linux-x86-64.so.2 (0x00002ac0173e8000)
>>>>>> libgeos-3.0.0.so => /usr/lib/libgeos-3.0.0.so
>>>>>> (0x00002ac01c1eb000) libltdl.so.3 => /usr/lib/libltdl.so.3
>>>>>> (0x00002ac01c53a000) libicuuc.so.38 => /usr/lib/libicuuc.so.38
>>>>>> (0x00002ac01c741000) libicudata.so.38 => /usr/lib/libicudata.so.38
>>>>>> (0x00002ac01ca82000) libproj.so.0 => /usr/lib/libproj.so.0
>>>>>> (0x00002ac01d759000) libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8
>>>>>> (0x00002ac01d99a000) libcrypto.so.0.9.8 =>
>>>>>> /usr/lib/libcrypto.so.0.9.8
>>>>>> (0x00002ac01dbec000)
>>>>>> libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x00002ac01df87000)
>>>>>> libcom_err.so.2 => /lib/libcom_err.so.2 (0x00002ac01e228000)
>>>>>> libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2
>>>>>> (0x00002ac01e42c000)
>>>>>> libcrypt.so.1 => /lib/libcrypt.so.1 (0x00002ac01e658000)
>>>>>> libldap_r-2.4.so.2 => /usr/lib/libldap_r-2.4.so.2
>>>>>> (0x00002ac01e890000)
>>>>>> libidn.so.11 => /usr/lib/libidn.so.11 (0x00002ac01eada000)
>>>>>> liblber-2.4.so.2 => /usr/lib/liblber-2.4.so.2
>>>>>> (0x00002ac01ed0c000) libgnutls.so.26 => /usr/lib/libgnutls.so.26
>>>>>> (0x00002ac01ef1b000) libnsl.so.1 => /lib/libnsl.so.1
>>>>>> (0x00002ac01f1ce000) libk5crypto.so.3 => /usr/lib/libk5crypto.so.3
>>>>>> (0x00002ac01f3e6000) libkrb5support.so.0 =>
>>>>>> /usr/lib/libkrb5support.so.0 (0x00002ac01f60d000)
>>>>>> libkeyutils.so.1 => /lib/libkeyutils.so.1
>>>>>> (0x00002ac01f815000)
>>>>>> libresolv.so.2 => /lib/libresolv.so.2 (0x00002ac01fa17000)
>>>>>> libsasl2.so.2 => /usr/lib/libsasl2.so.2 (0x00002ac01fc2c000)
>>>>>> libtasn1.so.3 => /usr/lib/libtasn1.so.3 (0x00002ac01fe46000)
>>>>>> libgpg-error.so.0 => /usr/lib/libgpg-error.so.0
>>>>>> (0x00002ac020056000)
>>>>>> libgcrypt.so.11 => /usr/lib/libgcrypt.so.11
>>>>>> (0x00002ac02015a000)
>>>>>>
>>>>>> Even Rouault wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> no those warnings don't sound good at all ! The number of overviews
>>>>>>> should be fine too.
>>>>>>>
>>>>>>> Could you precise the version of GDAL you're using and if you have
>>>>>>> built with internal libtiff or external libtiff ?
>>>>>>> (Such warnings could happen with older libtiff, especially with TIFF
>>>>>>> compression)
>>>>>>> Could you just check gdalinfo -checksum file.tif on your original
>>>>>>> file
>>>>>>> before running gdaladdo and check that it doesn't throw similar
>>>>>>> errors
>>>>>>> ?
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Even
>>>>>>>
>>>>>>> Le Thursday 19 March 2009 22:41:53 Stephen Woodbridge, vous avez
>> écrit :
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I just created a bunch of tif files from a mrsid file and when I
>>>>>>>> use
>>>>>>>> gdaladdo file.tif 2 4 8 16 32 64 128 256
>>>>>>>>
>>>>>>>> I get over a 1000 errors like:
>>>>>>>>
>>>>>>>> ERROR 1: n-41-25_0-0.tif:DumpModeDecode: Not enough data for
>>>>>>>> scanline
>>>>>>>> 1536
>>>>>>>> ERROR 1: TIFFReadEncodedTile() failed.
>>>>>>>> ERROR 1: IReadBlock failed at X offset 12, Y offset 7
>>>>>>>> ERROR 1: GetBlockRef failed at X block offset 12, Y block offset 7
>>>>>>>> ERROR 1: n-41-25_0-0.tif:DumpModeDecode: Not enough data for
>>>>>>>> scanline
>>>>>>>> 1664
>>>>>>>> ERROR 1: TIFFReadEncodedTile() failed.
>>>>>>>> ERROR 1: IReadBlock failed at X offset 13, Y offset 7
>>>>>>>> ERROR 1: GetBlockRef failed at X block offset 13, Y block offset 7
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Is this because of the number of overviews I requested? ot
>>>>>>>> something
>>>>>>>> else? Is it safe to ignore these?
>>>>>>>>
>>>>>>>> -Steve
>>>>>>>> _______________________________________________
>>>>>>>> gdal-dev mailing list
>>>>>>>> gdal-dev at lists.osgeo.org
>>>>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>>>> _______________________________________________
>>>>>> gdal-dev mailing list
>>>>>> gdal-dev at lists.osgeo.org
>>>>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
More information about the gdal-dev
mailing list