[gdal-dev] ERROR 1: LZWDecode:Wrong length of decoded string: data probably corrupted

[Even Rouault]

Selon Hermann Peifer <peifer at gmx.eu>:

Herman,

About your attempt to try to rasterize everything at once, it is likely that you
ran out of virtual memory as gdal_rasterize will load all the geometries in
memory before proceeding to the rasterization (it requires probably much more
memory than the on-disk size of the shapefile). If you can debug, you could set
up a breakpoint before the call to GDALRasterizeGeometries() at line 293 of
gdal_rasterize.cpp. If my hypothesis is right, then the process will be killed
before reaching the breakpoint. (There's a GDALRasterizeLayers() API that would
avoid loading all the OGR layer at once, but it is unused by gdal_rasterize
application, apparently because there are some options that couldn't be done
with it)

As far as the error about LZW is concerned, do you use internal libtiff or your
system libtiff ? If using external libtiff, you could try using internal libtiff
instead. And if you are already using internal libtiff, then it's likely an
evidence there's a remaining bug in how it deals with updating... But anyway I
would recommand you against using a compression scheme when doing multiple
updates of the file. Due to the way how libtiff manages strips/tiles allocation,
it will very likely to suboptimal file size because strips/tiles will be
rewritten many times. I'd rather use uncompressed TIFF for intermediate work and
as a final step do a gdal_translate -of LZW.

Best regards,

Even

>
> Hi,
>
> Over the weekend, I tried to gdal_rasterize a 350+ MB shapefile with
> 149671 polygon features into a 64116 x 61850 GeoTIFF.
>
> I first had some trouble while trying to rasterize everything at once,
> because after some time, the gdal_rasterize process got killed by the
> kerned (that's how I understood it).
>
> I then tried to loop through the shapefile's CODE values, something like
> the bash code [1] below. I ended up with different errors, see [2]
>
> Any idea what is going on and how to avoid these errors? (I am using
> GDAL v1.8dev from trunk)
>
> Thanks in advance, Hermann
>
> PS I tried without LZW compression, but this generates a 3+GB GeoTIFF,
> which in return seems to lead to other (performance) problems.
>
>
> [1]
>
> for file in *.shp ; do
>    for code in {0..20} ; do
>      if [[ $code = 0 ]] ; then
>        gdal_rasterize --debug on --config GDAL_CACHEMAX 1000 \
>          -ot byte -co compress=lzw -tr 2 2 -tap -l ${file%.shp} \
>          -burn $code -where CODE=\'$code\' \
>          $file ${file%.shp}_2m.tif >  ${file%.shp}_2m.log 2>&1
>      else
>        gdal_rasterize --debug on --config GDAL_CACHEMAX 1000 \
>          -l ${file%.shp} -burn $code -where CODE=\'$code\' \
>          $file ${file%.shp}_2m.tif >> ${file%.shp}_2m.log 2>&1
>      fi
>    done
> done
>
>
> [2]
>
> $ head -90 uk001l_london_2m.log
> OGR: OGROpen(shpfiles/uk001l_london.shp/0x764610) succeeded as ESRI
> Shapefile.
> GDAL:
> GDALDriver::Create(GTiff,uk001l_london_2m.tif,64116,61850,1,Byte,0x764ee0)
> Shape: 149671 features read on layer 'uk001l_london'.
> GDAL: GDALClose(uk001l_london_2m.tif, this=0x782e00)
> GDAL: GDALDeregister_GTiff() called.
> OGR: OGROpen(shpfiles/uk001l_london.shp/0x2080610) succeeded as ESRI
> Shapefile.
> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x209feb0) succeeds as GTiff.
> 0...10...20...30...40...50...60.OGR:
> OGROpen(shpfiles/uk001l_london.shp/0x10c1610) succeeded as ESRI Shapefile.
> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x10e0eb0) succeeds as GTiff.
> .0...70..10.....2080.....30.90......Shape: 149671 features read on layer
> 'uk001l_london'.
> 40.GDAL: GDALClose(uk001l_london_2m.tif, this=0x209feb0)
> GDAL: GDALDeregister_GTiff() called.
> 100 - done.
> ..50..OGR: OGROpen(shpfiles/uk001l_london.shp/0x225e610) succeeded as
> ESRI Shapefile.
> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x227deb0) succeeds as GTiff.
> 0....6010....20.....3070.....OGR:
> OGROpen(shpfiles/uk001l_london.shp/0x1543610) succeeded as ESRI Shapefile.
> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1562eb0) succeeds as GTiff.
> 0...10...2040.....3080.....50..40....90...ERROR 1: LZWDecode:Wrong
> length of decoded string: data probably corrupted at scanline 36908
> ERROR 1: TIFFReadEncodedStrip() failed.
>
> ERROR 1: IReadBlock failed at X offset 0, Y offset 36908
> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 36908
> Shape: 149671 features read on layer 'uk001l_london'.
> ..GDAL: GDALClose(uk001l_london_2m.tif, this=0x227deb0)
> GDAL: GDALDeregister_GTiff() called.
> 50...Shape: 149671 features read on layer 'uk001l_london'.
> .ERROR 1: LZWDecode:Wrong length of decoded string: data probably
> corrupted at scanline 36908
> ERROR 1: TIFFReadEncodedStrip() failed.
>
> ERROR 1: IReadBlock failed at X offset 0, Y offset 36908
> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 36908
> Shape: 149671 features read on layer 'uk001l_london'.
> GDAL: GDALClose(uk001l_london_2m.tif, this=0x10e0eb0)
> GDAL: GDALDeregister_GTiff() called.
> 100 - done.
> GDAL: GDALClose(uk001l_london_2m.tif, this=0x1562eb0)
> GDAL: GDALDeregister_GTiff() called.
> OGR: OGROpen(shpfiles/uk001l_london.shp/0xa21610) succeeded as ESRI
> Shapefile.
> GDAL: GDALOpen(uk001l_london_2m.tif, this=0xa40eb0) succeeds as GTiff.
> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
> ERROR 1: TIFFReadEncodedStrip() failed.
>
> ERROR 1: IReadBlock failed at X offset 0, Y offset 863
> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
> Shape: 149671 features read on layer 'uk001l_london'.
> GDAL: GDALClose(uk001l_london_2m.tif, this=0xa40eb0)
> GDAL: GDALDeregister_GTiff() called.
> OGR: OGROpen(shpfiles/uk001l_london.shp/0x1cd3610) succeeded as ESRI
> Shapefile.
> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1cf2eb0) succeeds as GTiff.
> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
> ERROR 1: TIFFReadEncodedStrip() failed.
>
> ERROR 1: IReadBlock failed at X offset 0, Y offset 863
> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
> Shape: 149671 features read on layer 'uk001l_london'.
> GDAL: GDALClose(uk001l_london_2m.tif, this=0x1cf2eb0)
> GDAL: GDALDeregister_GTiff() called.
> OGR: OGROpen(shpfiles/uk001l_london.shp/0x1cc5610) succeeded as ESRI
> Shapefile.
> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1ce4eb0) succeeds as GTiff.
> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
> ERROR 1: TIFFReadEncodedStrip() failed.
>
> ERROR 1: IReadBlock failed at X offset 0, Y offset 863
> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
> Shape: 149671 features read on layer 'uk001l_london'.
> GDAL: GDALClose(uk001l_london_2m.tif, this=0x1ce4eb0)
> GDAL: GDALDeregister_GTiff() called.
> OGR: OGROpen(shpfiles/uk001l_london.shp/0x172e610) succeeded as ESRI
> Shapefile.
> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x174deb0) succeeds as GTiff.
> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
> ERROR 1: TIFFReadEncodedStrip() failed.
>
> ERROR 1: IReadBlock failed at X offset 0, Y offset 863
> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
> Shape: 149671 features read on layer 'uk001l_london'.
> GDAL: GDALClose(uk001l_london_2m.tif, this=0x174deb0)
> GDAL: GDALDeregister_GTiff() called.
> OGR: OGROpen(shpfiles/uk001l_london.shp/0x1276610) succeeded as ESRI
> Shapefile.
> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1295eb0) succeeds as GTiff.
> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
> ERROR 1: TIFFReadEncodedStrip() failed.
>
> ERROR 1: IReadBlock failed at X offset 0, Y offset 863
> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
> Shape: 149671 features read on layer 'uk001l_london'.
> GDAL: GDALClose(uk001l_london_2m.tif, this=0x1295eb0)
> GDAL: GDALDeregister_GTiff() called.
> OGR: OGROpen(shpfiles/uk001l_london.shp/0x1c0c610) succeeded as ESRI
> Shapefile.
> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1c2beb0) succeeds as GTiff.
> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
> ERROR 1: TIFFReadEncodedStrip() failed.
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>