[gdal-dev] ERROR 1: LZWDecode:Wrong length of decoded string: data probably corrupted

[Hermann Peifer]

Hi,

Over the weekend, I tried to gdal_rasterize a 350+ MB shapefile with 
149671 polygon features into a 64116 x 61850 GeoTIFF.

I first had some trouble while trying to rasterize everything at once, 
because after some time, the gdal_rasterize process got killed by the 
kerned (that's how I understood it).

I then tried to loop through the shapefile's CODE values, something like 
the bash code [1] below. I ended up with different errors, see [2]

Any idea what is going on and how to avoid these errors? (I am using 
GDAL v1.8dev from trunk)

Thanks in advance, Hermann

PS I tried without LZW compression, but this generates a 3+GB GeoTIFF, 
which in return seems to lead to other (performance) problems.


[1]

for file in *.shp ; do
   for code in {0..20} ; do
     if [[ $code = 0 ]] ; then
       gdal_rasterize --debug on --config GDAL_CACHEMAX 1000 \
         -ot byte -co compress=lzw -tr 2 2 -tap -l ${file%.shp} \
         -burn $code -where CODE=\'$code\' \
         $file ${file%.shp}_2m.tif >  ${file%.shp}_2m.log 2>&1
     else
       gdal_rasterize --debug on --config GDAL_CACHEMAX 1000 \
         -l ${file%.shp} -burn $code -where CODE=\'$code\' \
         $file ${file%.shp}_2m.tif >> ${file%.shp}_2m.log 2>&1
     fi
   done
done


[2]

$ head -90 uk001l_london_2m.log
OGR: OGROpen(shpfiles/uk001l_london.shp/0x764610) succeeded as ESRI 
Shapefile.
GDAL: 
GDALDriver::Create(GTiff,uk001l_london_2m.tif,64116,61850,1,Byte,0x764ee0)
Shape: 149671 features read on layer 'uk001l_london'.
GDAL: GDALClose(uk001l_london_2m.tif, this=0x782e00)
GDAL: GDALDeregister_GTiff() called.
OGR: OGROpen(shpfiles/uk001l_london.shp/0x2080610) succeeded as ESRI 
Shapefile.
GDAL: GDALOpen(uk001l_london_2m.tif, this=0x209feb0) succeeds as GTiff.
0...10...20...30...40...50...60.OGR: 
OGROpen(shpfiles/uk001l_london.shp/0x10c1610) succeeded as ESRI Shapefile.
GDAL: GDALOpen(uk001l_london_2m.tif, this=0x10e0eb0) succeeds as GTiff.
.0...70..10.....2080.....30.90......Shape: 149671 features read on layer 
'uk001l_london'.
40.GDAL: GDALClose(uk001l_london_2m.tif, this=0x209feb0)
GDAL: GDALDeregister_GTiff() called.
100 - done.
..50..OGR: OGROpen(shpfiles/uk001l_london.shp/0x225e610) succeeded as 
ESRI Shapefile.
GDAL: GDALOpen(uk001l_london_2m.tif, this=0x227deb0) succeeds as GTiff.
0....6010....20.....3070.....OGR: 
OGROpen(shpfiles/uk001l_london.shp/0x1543610) succeeded as ESRI Shapefile.
GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1562eb0) succeeds as GTiff.
0...10...2040.....3080.....50..40....90...ERROR 1: LZWDecode:Wrong 
length of decoded string: data probably corrupted at scanline 36908
ERROR 1: TIFFReadEncodedStrip() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 36908
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 36908
Shape: 149671 features read on layer 'uk001l_london'.
..GDAL: GDALClose(uk001l_london_2m.tif, this=0x227deb0)
GDAL: GDALDeregister_GTiff() called.
50...Shape: 149671 features read on layer 'uk001l_london'.
.ERROR 1: LZWDecode:Wrong length of decoded string: data probably 
corrupted at scanline 36908
ERROR 1: TIFFReadEncodedStrip() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 36908
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 36908
Shape: 149671 features read on layer 'uk001l_london'.
GDAL: GDALClose(uk001l_london_2m.tif, this=0x10e0eb0)
GDAL: GDALDeregister_GTiff() called.
100 - done.
GDAL: GDALClose(uk001l_london_2m.tif, this=0x1562eb0)
GDAL: GDALDeregister_GTiff() called.
OGR: OGROpen(shpfiles/uk001l_london.shp/0xa21610) succeeded as ESRI 
Shapefile.
GDAL: GDALOpen(uk001l_london_2m.tif, this=0xa40eb0) succeeds as GTiff.
0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
ERROR 1: TIFFReadEncodedStrip() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 863
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
Shape: 149671 features read on layer 'uk001l_london'.
GDAL: GDALClose(uk001l_london_2m.tif, this=0xa40eb0)
GDAL: GDALDeregister_GTiff() called.
OGR: OGROpen(shpfiles/uk001l_london.shp/0x1cd3610) succeeded as ESRI 
Shapefile.
GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1cf2eb0) succeeds as GTiff.
0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
ERROR 1: TIFFReadEncodedStrip() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 863
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
Shape: 149671 features read on layer 'uk001l_london'.
GDAL: GDALClose(uk001l_london_2m.tif, this=0x1cf2eb0)
GDAL: GDALDeregister_GTiff() called.
OGR: OGROpen(shpfiles/uk001l_london.shp/0x1cc5610) succeeded as ESRI 
Shapefile.
GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1ce4eb0) succeeds as GTiff.
0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
ERROR 1: TIFFReadEncodedStrip() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 863
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
Shape: 149671 features read on layer 'uk001l_london'.
GDAL: GDALClose(uk001l_london_2m.tif, this=0x1ce4eb0)
GDAL: GDALDeregister_GTiff() called.
OGR: OGROpen(shpfiles/uk001l_london.shp/0x172e610) succeeded as ESRI 
Shapefile.
GDAL: GDALOpen(uk001l_london_2m.tif, this=0x174deb0) succeeds as GTiff.
0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
ERROR 1: TIFFReadEncodedStrip() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 863
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
Shape: 149671 features read on layer 'uk001l_london'.
GDAL: GDALClose(uk001l_london_2m.tif, this=0x174deb0)
GDAL: GDALDeregister_GTiff() called.
OGR: OGROpen(shpfiles/uk001l_london.shp/0x1276610) succeeded as ESRI 
Shapefile.
GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1295eb0) succeeds as GTiff.
0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
ERROR 1: TIFFReadEncodedStrip() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 863
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
Shape: 149671 features read on layer 'uk001l_london'.
GDAL: GDALClose(uk001l_london_2m.tif, this=0x1295eb0)
GDAL: GDALDeregister_GTiff() called.
OGR: OGROpen(shpfiles/uk001l_london.shp/0x1c0c610) succeeded as ESRI 
Shapefile.
GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1c2beb0) succeeds as GTiff.
0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
ERROR 1: TIFFReadEncodedStrip() failed.