[Gdal-dev] gdaladdo problems on an Mac OS X Panther

Gross, Dorit (SDRN) Dorit.Gross at fao.org
Tue Mar 21 09:35:33 EST 2006


William, 

Thank you very much for the many tests!

I now removed the LZW compression and limited the overview levels to 2 and 4.
I receive the StripByteCounts warnings as you do, but the process runs till
the end anyway.

In reference to our system: We had made a backup of the previous GIS and
Graphic Libs (1.5.2) and I tested with those ones on the updated system. In
this setup the old Libs generate exactly the same errors as the new ones.
Unfortunately, we cannot move back to the previous Mac OS X system version
(also not for testing). So for the time being we can calculate overvies up to
level 4, however a solution for the new Mac version would just be great.

Looking forward to hearing from you,
Dorit




-----Original Message-----
From: William Kyngesburye [mailto:woklist at kyngchaos.com] 
Sent: 21 March 2006 01:03
To: gdal-dev at lists.maptools.org
Cc: Frank Warmerdam; Gross, Dorit (SDRN)
Subject: Re: [Gdal-dev] gdaladdo problems on an Mac OS X Panther


First, as far as libtiff or gdal+libtiff goes, I think LZW  
compression is OK.  I use gdal_translate to create LZW compressed  
TIFFs a lot, including as I write this.  So there is something going  
on in either gdaladdo or the overview functions in libgdal.

Here is a Mac OS X crash report for part of the gdaladdo problem.

One odd thing I notice in this thread trail: it looks like it's  
coming out of the JPEG 2000 driver.


Date/Time:      2006-03-16 12:39:37.551 -0600
OS Version:     10.4.5 (Build 8H14)
Report Version: 4

Command: gdaladdo
Path:    /usr/local/gislibs/bin/gdaladdo
Parent:  bash [15756]

Version: ??? (???)

PID:    15882
Thread: 0

Exception:  EXC_BAD_ACCESS (0x0001)
Codes:      KERN_PROTECTION_FAILURE (0x0002) at 0x00000810

Thread 0 Crashed:
0   <<00000000>> 	0xffff862c __bzero + 44 (cpu_capabilities.h:187)
1   libtiff.3.dylib 	0x0072f30c LZWPreDecode + 284 (icplusplus.c:28)
2   libtiff.3.dylib 	0x00737f38 TIFFReadEncodedTile + 140  
(icplusplus.c:28)
3   libgdal.1.dylib 	0x01059f18 GTiffRasterBand::IReadBlock(int, int,  
void*) + 296 (icplusplus.c:28)
4   libgdal.1.dylib 	0x010f7824 GDALRasterBand::GetLockedBlockRef 
(int, int, int) + 256 (jpeg2000dataset.cpp:1188)
5   libgdal.1.dylib 	0x010fbb14 GDALRasterBand::IRasterIO(GDALRWFlag,  
int, int, int, int, void*, int, int, GDALDataType, int, int) + 984  
(jpeg2000dataset.cpp:1188)
6   libgdal.1.dylib 	0x010f9cb0 GDALRegenerateOverviews + 1036  
(jpeg2000dataset.cpp:1188)
7   libgdal.1.dylib 	0x010f9b24 GDALRegenerateOverviews + 640  
(jpeg2000dataset.cpp:1188)
8   libgdal.1.dylib 	0x0105c400 GTiffDataset::IBuildOverviews(char  
const*, int, int*, int, int*, int (*)(double, char const*, void*),  
void*) + 1748 (icplusplus.c:28)
9   libgdal.1.dylib 	0x010ec6d8 GDALDataset::BuildOverviews(char  
const*, int, int*, int, int*, int (*)(double, char const*, void*),  
void*) + 188 (jpeg2000dataset.cpp:1188)
10  gdaladdo        	0x00002830 main + 332 (crt.c:355)
11  gdaladdo        	0x00001f10 _start + 340 (crt.c:272)
12  gdaladdo        	0x00001db8 start + 60


With average_mp resampling I get a seg fault and the thread trail now  
ends with

0   libtiff.3.dylib 	0x0072fd94 cl_hash + 52 (icplusplus.c:28)
1   libtiff.3.dylib 	0x0072fe78 LZWPreEncode + 152 (icplusplus.c:28)
2   libtiff.3.dylib 	0x0073a50c TIFFWriteEncodedTile + 404  
(icplusplus.c:28)
3   libgdal.1.dylib 	0x01059cf8 GTiffRasterBand::IWriteBlock(int,  
int, void*) + 156 (icplusplus.c:28)
4   libgdal.1.dylib 	0x010f71bc GDALRasterBand::FlushBlock(int, int)  
+ 300 (jpeg2000dataset.cpp:1188)
5   libgdal.1.dylib 	0x010f7250 GDALRasterBand::FlushCache() + 80  
(jpeg2000dataset.cpp:1188)
6   libgdal.1.dylib 	0x010fa650 GDALRegenerateOverviews + 3500  
(jpeg2000dataset.cpp:1188)
...


Here is a table of tests I've done so far:

(* Dorit's used to work here)
the 10.3.9 tests are without any security updates

                    orig               compress             levels
TIFF       color  compress  Mac OS X     OV     resample     OK      
error
------------------------------------------
SPOT       grey    none
                              10.3.6/GDAL 1.2.6/TIFF 3.7.3
                                        default   nearest    2..128
                                         (none)   average    2..128
                                        LZW(?)    nearest    2..128    
but 'LZW' compression ignored
                            * 10.3.6/GDAL 1.3.0/TIFF 3.7.3
                                        default   average    2..128
                              10.3.9/GDAL 1.3.0/TIFF 3.7.3
                                        default   average    2..128
                              10.3.9/GDAL 1.3.1/TIFF 3.8.0
                                        default   nearest    2,4       
StripByteCounts warnings always
                                                                       b 
lock errors at lvl 8
                                                  average    2,4       
StripByteCounts warnings always
                                                                       b 
lock errors at lvl 8
                              10.4.5/GDAL 1.3.0/TIFF 3.7.3
                                don't have a clean Tiger system  
available yet to test
                              10.4.5/GDAL 1.3.1/TIFF 3.8.1
                                        default   nearest    2,4       
StripByteCounts warnings always
                                                                       b 
lock errors at lvl 8
                                                  average    2,4       
StripByteCounts warnings always
                                                                       b 
lock errors at lvl 8

small      color   LZW
                              10.3.6/GDAL 1.2.6/TIFF 3.7.3
                                        default   nearest    2..128
                                         (LZW)    average    2         
bus error at lvl 4
                                        none(?)   nearest    2..128    
but 'none' compression is ignored
                                        none(?)   average    2         
bus error at lvl 4
                            * 10.3.6/GDAL 1.3.0/TIFF 3.7.3
                                        default   nearest    2..128
                                         (LZW)    average    2         
bus error at lvl 4
                              10.3.9/GDAL 1.3.0/TIFF 3.7.3
                                        default   nearest    2..128
                                         (LZW)    average    2         
bus error at lvl 4
                              10.3.9/GDAL 1.3.1/TIFF 3.8.0
                                        default   nearest    2..128
                                         (LZW)    average    2         
bus error at lvl 4
                              10.4.5/GDAL 1.3.0/TIFF 3.7.3
                                don't have a clean Tiger system  
available yet to test
                              10.4.5/GDAL 1.3.1/TIFF 3.8.1
                                        default   nearest    2..128
                                         (LZW)    average    2         
bus error at lvl 4
                                                  average_mp  
2         seg fault at lvl 4
                                        none(?)   nearest    2..128    
but 'none' compression is ignored


So, I see a couple problems going on.  These bus errors and seg  
faults seem to have something to do with the resampling method used.   
Maybe combined with LZW compression, average resampling starts to choke.

Then the StripByteCounts warnings and block errors start happening  
with libtiff 3.8.0.

gdaladdo seems to ignore the compress overview option and use the  
same compression as the original file always.  So this option is  
pointless.


Dorit, are you sure that the errors are the same when you went back  
to GIS/Graphics Libs 1.5.2?  Was that on the same old system?  Or the  
new 10.3.9 system?  If you want to install the older packages over  
newer ones, I'm not sure the installers overwrite a newer  
installation correctly.  To be sure, try deleting the /Library/ 
Receipts for them, and then deleting /usr/local/baselibs, /usr/local/ 
gislibs and /usr/local/graphicslibs, then reinstall the old versions.


On Mar 20, 2006, at 4:17 AM, Gross, Dorit (SDRN) wrote:

> Hi William,
>
> Our previous versions were GIS Libs 1.5.2 and Graphic Libs 1.5.2 but 
> strangely they gernerate the same error now.
>
> Might it be also related to the update of our system? We updated
> last week
> and made the last security updates today. We run now system version  
> Mac OS X
> Server 10.3.9 (7W98). I tested today another time ... trying to add  
> overviews
> with the old and the new GIS Libs with the result that I receive  
> the same
> error message for both tests.
>
> Does it help?
>
> Dorit
>
>
>
>
>
> -----Original Message-----
> From: William Kyngesburye [mailto:kyngchaos at kyngchaos.com]
> Sent: 16 March 2006 19:54
> To: Gross, Dorit (SDRN)
> Cc: Frank Warmerdam
> Subject: Re: [Gdal-dev] gdaladdo problems on an Mac OS X Panther
>
>
> I get the same problem with your geotiff, on both Panther and Tiger. A 
> Panther build problem was one possibility.
>
> What were the previous GIS Libs and Graphics Libs versions that 
> worked?  At least between this version and the next recent release, 
> updates include libtiff 3.8.0 from 3.7.4 and geotiff CVS 2006/1/8 from 
> 1.2.2.  I'm in the middle of some more updates, and tried the tests 
> below with libtiff 3.8.1/geotiff 1.2.3 also with the same
> results.
>
> I also have the problem with one of my own old greyscale geotiffs 
> converted from TIFF with geotifcp.  And with a color plain TIFF, not a 
> geotiff.
>
> More problems with other options and files used:
>
> - a smaller color TIFF which is cropped (5000x5000) from the color 
> tiff above: same problem, but it makes more progress.
>
> - same cropped tiff, but only level 2 overview: OK
>
> - same cropped tiff, level 2 & 4 overviews: bus error at 30%
>
> - full size color tiff, only level 2 overview: makes it to 100% - 
> done, but give a final Wrong "StripByteCounts" field warning at the 
> end.  Photoshop opens it, but I don't know how I can verify the 
> overview.
>
> - your spot tiff, only level 2 & 4 overviews: initial Wrong 
> "StripByteCounts" field warning, but completes.  Opens in Photoshop. I 
> get the block errors when I add the level 8 overview.
>
>
> So, it seems the Wrong "StripByteCounts" field warnings are harmless, 
> it's the block errors that are the problem.
>
>
> On Mar 16, 2006, at 9:35 AM, Gross, Dorit (SDRN) wrote:
>
>> Just tried with a GeoTiff generated long time ago from a MODIS HDF 
>> (used the MODIS Reprojection Tool then) and it causes the same error. 
>> Will provide you
>> and Frank with a data set to test.
>>
>> Regards,
>> Dorit
>>
>> -----Original Message-----
>> From: William Kyngesburye [mailto:woklist at kyngchaos.com]
>> Sent: 16 March 2006 15:50
>> To: gdal-dev at lists.maptools.org
>> Cc: Gross, Dorit (SDRN)
>> Subject: Re: [Gdal-dev] gdaladdo problems on an Mac OS X Panther
>>
>>
>> What about other TIFFs?
>>
>> I'd like to get the image also so I can check my GIS Libs builds.
>> I'll
>> try some of my own TIFFs also.  There may a problem with my
>> distribution.
>>
>>
>> On Mar 16, 2006, at 2:30 AM, Gross, Dorit (SDRN) wrote:
>>
>>> Hi Frank,
>>>
>>> The original file is a SPOT VGT HDF that was converted into GeoTiff 
>>> using gdal_translate. The overviews are added just in the next step. 
>>> The script was running before the upgrade of GIS Libs. I tried the 
>>> script for different SPOT
>>> files after the upgrade but receive now the same error message for
>>> all of
>>> them.
>>>
>>> I can provide you an image. Please let me know where it can be 
>>> uploaded to.
>>>
>>> Regards,
>>> Dorit
>>>
>>>
>>> -----Original Message-----
>>> From: Frank Warmerdam [mailto:fwarmerdam at gmail.com] On Behalf Of 
>>> Frank Warmerdam
>>> Sent: 15 March 2006 17:32
>>> To: Gross, Dorit (SDRN)
>>> Cc: gdal-dev at lists.maptools.org
>>> Subject: Re: [Gdal-dev] gdaladdo problems on an Mac OS X Panther
>>>
>>>
>>> Gross, Dorit (SDRN) wrote:
>>>> Dear list,
>>>>
>>>> I use gdaladdo to add overviews to a GeoTiff and receive following 
>>>> error
>>>> message:
>>>>
>>>> /usr/local/gislibs/bin/gdaladdo -r average --config 
>>>> COMPRESS_OVERVIEW LZW 
>>>> /Artemis/public/spot-veg/ndv/2006/dv06031_wrld.tif 2 4 8 16 32 64
>>>> 128 0.Warning 1:
>>>> TIFFReadDirectory:/Artemis/public/spot-veg/ndv/2006/
>>>> dv06031_wrld.tif:
>>>> Wrong "StripByteCounts" field, ignoring and calculating from
>>>> imagelength Warning 1:
>>>> TIFFReadDirectory:/Artemis/public/spot-veg/ndv/2006/
>>>> dv06031_wrld.tif: Wrong
>>>> "StripByteCounts" field, ignoring and calculating from imagelength
>>>> ..10...20...30...40...50...60...70...80...90..Warning 1:
>>>> TIFFReadDirectory:/Artemis/public/spot-veg/ndv/2006/
>>>> dv06031_wrld.tif: Wrong
>>>> "StripByteCounts" field, ignoring and calculating from imagelength
>>>> Warning
>>> 1:
>>>> TIFFReadDirectory:/Artemis/public/spot-veg/ndv/2006/
>>>> dv06031_wrld.tif: Wrong
>>>> "StripByteCounts" field, ignoring and calculating from imagelength 
>>>> ERROR 1:
>>>> /Artemis/public/spot-veg/ndv/2006/dv06031_wrld.tif:DumpModeDecode:
>>>> Not
>>> enough
>>>> data for scanline 0 ERROR 1: TIFFReadEncodedTile() failed.
>>>>
>>>> ERROR 1: IReadBlock failed at X offset 0, Y offset 4
>>>> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 4 
>>>> ERROR 1:
>>>> /Artemis/public/spot-veg/ndv/2006/dv06031_wrld.tif:DumpModeDecode:
>>>> Not
>>> enough
>>>> data for scanline 0 ERROR 1: TIFFReadEncodedTile() failed.
>>>>
>>>> ...
>>>>
>>>> Any ideas what could be the problem and how it could be solved?
>>>>
>>>> We use the latest GIS Libs for Mac OS X Panther (2.2.1) including 
>>>> GDAL/OGR v1.3.1 downloaded from Kyng Chaos.
>>>
>>> Dorit,
>>>
>>> It *sounds* like the input file may be corrupt in some sense.   
>>> Was it
>>> generated with libtiff?  Can you make the original file available
>>> for
>>> me to look at?
>>>
>>> Best regards,
>>> --
>>> ---------------------------------------
>>> +--------------------------------
>>> ---------------------------------------+------
>>> I set the clouds in motion - turn up   | Frank Warmerdam,
>>> warmerdam at pobox.com
>>> light and sound - activate the windows | http://pobox.com/~warmerdam
>>> and watch the world go round - Rush    | President OSGF, http://
>>> osgeo.org
>>>
>>> _______________________________________________
>>> Gdal-dev mailing list
>>> Gdal-dev at lists.maptools.org 
>>> http://lists.maptools.org/mailman/listinfo/gdal-dev
>>
>> -----
>> William Kyngesburye <kyngchaos at kyngchaos.com> http:// 
>> www.kyngchaos.com/
>>
>> "This is a question about the past, is it? ... How can I tell that
>> the
>> past isn't a fiction designed to account for the discrepancy between
>> my immediate physical sensations and my state of mind?"
>>
>> - The Ruler of the Universe
>>
>> _______________________________________________
>> Gdal-dev mailing list
>> Gdal-dev at lists.maptools.org 
>> http://lists.maptools.org/mailman/listinfo/gdal-dev
>
> -----
> William Kyngesburye <kyngchaos at kyngchaos.com> http://
> www.kyngchaos.com/
>
> First Pogril: Why is life like sticking your head in a bucket filled 
> with hyena offal? Second Pogril: I don't know.  Why IS life like 
> sticking your head in a bucket filled with hyena offal?
> First Pogril: I don't know either.  Wretched, isn't it?
>
> -HitchHiker's Guide to the Galaxy
>

-----
William Kyngesburye <kyngchaos at kyngchaos.com> http://www.kyngchaos.com/

First Pogril: Why is life like sticking your head in a bucket filled  
with hyena offal?
Second Pogril: I don't know.  Why IS life like sticking your head in  
a bucket filled with hyena offal?
First Pogril: I don't know either.  Wretched, isn't it?

-HitchHiker's Guide to the Galaxy




More information about the Gdal-dev mailing list