[gdal-dev] ERROR 1: LZWDecode:Wrong length of decoded string: data probably corrupted

[Even Rouault]

Selon Hermann Peifer <peifer at gmx.eu>:

> Even,
>
> Thanks for the quick feedback.
>
> Just to clarify: last week, I compiled revision 21190 from
> http://svn.osgeo.org/gdal/trunk/gdal, configured
> --with-libtiff=internal.

Hum OK so there's definitely a bug somewhere. Would be worth opening a ticket if
you had a way of reproducing those errors. Ideally with a smaller shapefile.

Would be interesting to see if it works better with another compression scheme,
let's say DEFLATE.

I have 8G of memory, which is about 20 times
> the size of the shapefile. However, I might have been processing other
> shapefiles in parallel when running into the out-of-memory error. I will
> double-check.

Is your OS a 64 bit OS ? Just to be sure you aren't limited to 2 GB per process
if it is a 32 bit OS.

>
> About using an uncompressed TIFF for intermediate work:
> Doesn't this lead to performance issues, as the uncompressed TIFF (say:
> 3G) has be written to disc so many times (?)

As often, the performance impact is hard to predict. It is a trade-off between
bigger I/O amount for uncompressed versus bigger CPU time for compressed.

>
> Thanks again, Hermann
>
>
> On 06/12/2010 11:19, Even Rouault wrote:
> > Selon Hermann Peifer<peifer at gmx.eu>:
> >
> > Herman,
> >
> > About your attempt to try to rasterize everything at once, it is likely
> that you
> > ran out of virtual memory as gdal_rasterize will load all the geometries in
> > memory before proceeding to the rasterization (it requires probably much
> more
> > memory than the on-disk size of the shapefile). If you can debug, you could
> set
> > up a breakpoint before the call to GDALRasterizeGeometries() at line 293 of
> > gdal_rasterize.cpp. If my hypothesis is right, then the process will be
> killed
> > before reaching the breakpoint. (There's a GDALRasterizeLayers() API that
> would
> > avoid loading all the OGR layer at once, but it is unused by gdal_rasterize
> > application, apparently because there are some options that couldn't be
> done
> > with it)
> >
> > As far as the error about LZW is concerned, do you use internal libtiff or
> your
> > system libtiff ? If using external libtiff, you could try using internal
> libtiff
> > instead. And if you are already using internal libtiff, then it's likely an
> > evidence there's a remaining bug in how it deals with updating... But
> anyway I
> > would recommand you against using a compression scheme when doing multiple
> > updates of the file. Due to the way how libtiff manages strips/tiles
> allocation,
> > it will very likely to suboptimal file size because strips/tiles will be
> > rewritten many times. I'd rather use uncompressed TIFF for intermediate
> work and
> > as a final step do a gdal_translate -of LZW.
> >
> > Best regards,
> >
> > Even
> >
> >> Hi,
> >>
> >> Over the weekend, I tried to gdal_rasterize a 350+ MB shapefile with
> >> 149671 polygon features into a 64116 x 61850 GeoTIFF.
> >>
> >> I first had some trouble while trying to rasterize everything at once,
> >> because after some time, the gdal_rasterize process got killed by the
> >> kerned (that's how I understood it).
> >>
> >> I then tried to loop through the shapefile's CODE values, something like
> >> the bash code [1] below. I ended up with different errors, see [2]
> >>
> >> Any idea what is going on and how to avoid these errors? (I am using
> >> GDAL v1.8dev from trunk)
> >>
> >> Thanks in advance, Hermann
> >>
> >> PS I tried without LZW compression, but this generates a 3+GB GeoTIFF,
> >> which in return seems to lead to other (performance) problems.
> >>
> >>
> >> [1]
> >>
> >> for file in *.shp ; do
> >>     for code in {0..20} ; do
> >>       if [[ $code = 0 ]] ; then
> >>         gdal_rasterize --debug on --config GDAL_CACHEMAX 1000 \
> >>           -ot byte -co compress=lzw -tr 2 2 -tap -l ${file%.shp} \
> >>           -burn $code -where CODE=\'$code\' \
> >>           $file ${file%.shp}_2m.tif>   ${file%.shp}_2m.log 2>&1
> >>       else
> >>         gdal_rasterize --debug on --config GDAL_CACHEMAX 1000 \
> >>           -l ${file%.shp} -burn $code -where CODE=\'$code\' \
> >>           $file ${file%.shp}_2m.tif>>  ${file%.shp}_2m.log 2>&1
> >>       fi
> >>     done
> >> done
> >>
> >>
> >> [2]
> >>
> >> $ head -90 uk001l_london_2m.log
> >> OGR: OGROpen(shpfiles/uk001l_london.shp/0x764610) succeeded as ESRI
> >> Shapefile.
> >> GDAL:
> >> GDALDriver::Create(GTiff,uk001l_london_2m.tif,64116,61850,1,Byte,0x764ee0)
> >> Shape: 149671 features read on layer 'uk001l_london'.
> >> GDAL: GDALClose(uk001l_london_2m.tif, this=0x782e00)
> >> GDAL: GDALDeregister_GTiff() called.
> >> OGR: OGROpen(shpfiles/uk001l_london.shp/0x2080610) succeeded as ESRI
> >> Shapefile.
> >> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x209feb0) succeeds as GTiff.
> >> 0...10...20...30...40...50...60.OGR:
> >> OGROpen(shpfiles/uk001l_london.shp/0x10c1610) succeeded as ESRI Shapefile.
> >> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x10e0eb0) succeeds as GTiff.
> >> .0...70..10.....2080.....30.90......Shape: 149671 features read on layer
> >> 'uk001l_london'.
> >> 40.GDAL: GDALClose(uk001l_london_2m.tif, this=0x209feb0)
> >> GDAL: GDALDeregister_GTiff() called.
> >> 100 - done.
> >> ..50..OGR: OGROpen(shpfiles/uk001l_london.shp/0x225e610) succeeded as
> >> ESRI Shapefile.
> >> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x227deb0) succeeds as GTiff.
> >> 0....6010....20.....3070.....OGR:
> >> OGROpen(shpfiles/uk001l_london.shp/0x1543610) succeeded as ESRI Shapefile.
> >> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1562eb0) succeeds as GTiff.
> >> 0...10...2040.....3080.....50..40....90...ERROR 1: LZWDecode:Wrong
> >> length of decoded string: data probably corrupted at scanline 36908
> >> ERROR 1: TIFFReadEncodedStrip() failed.
> >>
> >> ERROR 1: IReadBlock failed at X offset 0, Y offset 36908
> >> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 36908
> >> Shape: 149671 features read on layer 'uk001l_london'.
> >> ..GDAL: GDALClose(uk001l_london_2m.tif, this=0x227deb0)
> >> GDAL: GDALDeregister_GTiff() called.
> >> 50...Shape: 149671 features read on layer 'uk001l_london'.
> >> .ERROR 1: LZWDecode:Wrong length of decoded string: data probably
> >> corrupted at scanline 36908
> >> ERROR 1: TIFFReadEncodedStrip() failed.
> >>
> >> ERROR 1: IReadBlock failed at X offset 0, Y offset 36908
> >> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 36908
> >> Shape: 149671 features read on layer 'uk001l_london'.
> >> GDAL: GDALClose(uk001l_london_2m.tif, this=0x10e0eb0)
> >> GDAL: GDALDeregister_GTiff() called.
> >> 100 - done.
> >> GDAL: GDALClose(uk001l_london_2m.tif, this=0x1562eb0)
> >> GDAL: GDALDeregister_GTiff() called.
> >> OGR: OGROpen(shpfiles/uk001l_london.shp/0xa21610) succeeded as ESRI
> >> Shapefile.
> >> GDAL: GDALOpen(uk001l_london_2m.tif, this=0xa40eb0) succeeds as GTiff.
> >> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
> >> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
> >> ERROR 1: TIFFReadEncodedStrip() failed.
> >>
> >> ERROR 1: IReadBlock failed at X offset 0, Y offset 863
> >> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
> >> Shape: 149671 features read on layer 'uk001l_london'.
> >> GDAL: GDALClose(uk001l_london_2m.tif, this=0xa40eb0)
> >> GDAL: GDALDeregister_GTiff() called.
> >> OGR: OGROpen(shpfiles/uk001l_london.shp/0x1cd3610) succeeded as ESRI
> >> Shapefile.
> >> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1cf2eb0) succeeds as GTiff.
> >> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
> >> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
> >> ERROR 1: TIFFReadEncodedStrip() failed.
> >>
> >> ERROR 1: IReadBlock failed at X offset 0, Y offset 863
> >> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
> >> Shape: 149671 features read on layer 'uk001l_london'.
> >> GDAL: GDALClose(uk001l_london_2m.tif, this=0x1cf2eb0)
> >> GDAL: GDALDeregister_GTiff() called.
> >> OGR: OGROpen(shpfiles/uk001l_london.shp/0x1cc5610) succeeded as ESRI
> >> Shapefile.
> >> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1ce4eb0) succeeds as GTiff.
> >> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
> >> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
> >> ERROR 1: TIFFReadEncodedStrip() failed.
> >>
> >> ERROR 1: IReadBlock failed at X offset 0, Y offset 863
> >> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
> >> Shape: 149671 features read on layer 'uk001l_london'.
> >> GDAL: GDALClose(uk001l_london_2m.tif, this=0x1ce4eb0)
> >> GDAL: GDALDeregister_GTiff() called.
> >> OGR: OGROpen(shpfiles/uk001l_london.shp/0x172e610) succeeded as ESRI
> >> Shapefile.
> >> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x174deb0) succeeds as GTiff.
> >> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
> >> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
> >> ERROR 1: TIFFReadEncodedStrip() failed.
> >>
> >> ERROR 1: IReadBlock failed at X offset 0, Y offset 863
> >> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
> >> Shape: 149671 features read on layer 'uk001l_london'.
> >> GDAL: GDALClose(uk001l_london_2m.tif, this=0x174deb0)
> >> GDAL: GDALDeregister_GTiff() called.
> >> OGR: OGROpen(shpfiles/uk001l_london.shp/0x1276610) succeeded as ESRI
> >> Shapefile.
> >> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1295eb0) succeeds as GTiff.
> >> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
> >> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
> >> ERROR 1: TIFFReadEncodedStrip() failed.
> >>
> >> ERROR 1: IReadBlock failed at X offset 0, Y offset 863
> >> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
> >> Shape: 149671 features read on layer 'uk001l_london'.
> >> GDAL: GDALClose(uk001l_london_2m.tif, this=0x1295eb0)
> >> GDAL: GDALDeregister_GTiff() called.
> >> OGR: OGROpen(shpfiles/uk001l_london.shp/0x1c0c610) succeeded as ESRI
> >> Shapefile.
> >> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1c2beb0) succeeds as GTiff.
> >> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
> >> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
> >> ERROR 1: TIFFReadEncodedStrip() failed.
> >>
> >> _______________________________________________
> >> gdal-dev mailing list
> >> gdal-dev at lists.osgeo.org
> >> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> >>
> >
>