[gdal-dev] ERROR 1: LZWDecode:Wrong length of decoded string:
data probably corrupted
Hermann Peifer
peifer at gmx.eu
Mon Dec 6 05:43:13 EST 2010
Even,
Thanks for the quick feedback.
Just to clarify: last week, I compiled revision 21190 from
http://svn.osgeo.org/gdal/trunk/gdal, configured
--with-libtiff=internal. I have 8G of memory, which is about 20 times
the size of the shapefile. However, I might have been processing other
shapefiles in parallel when running into the out-of-memory error. I will
double-check.
About using an uncompressed TIFF for intermediate work:
Doesn't this lead to performance issues, as the uncompressed TIFF (say:
3G) has be written to disc so many times (?)
Thanks again, Hermann
On 06/12/2010 11:19, Even Rouault wrote:
> Selon Hermann Peifer<peifer at gmx.eu>:
>
> Herman,
>
> About your attempt to try to rasterize everything at once, it is likely that you
> ran out of virtual memory as gdal_rasterize will load all the geometries in
> memory before proceeding to the rasterization (it requires probably much more
> memory than the on-disk size of the shapefile). If you can debug, you could set
> up a breakpoint before the call to GDALRasterizeGeometries() at line 293 of
> gdal_rasterize.cpp. If my hypothesis is right, then the process will be killed
> before reaching the breakpoint. (There's a GDALRasterizeLayers() API that would
> avoid loading all the OGR layer at once, but it is unused by gdal_rasterize
> application, apparently because there are some options that couldn't be done
> with it)
>
> As far as the error about LZW is concerned, do you use internal libtiff or your
> system libtiff ? If using external libtiff, you could try using internal libtiff
> instead. And if you are already using internal libtiff, then it's likely an
> evidence there's a remaining bug in how it deals with updating... But anyway I
> would recommand you against using a compression scheme when doing multiple
> updates of the file. Due to the way how libtiff manages strips/tiles allocation,
> it will very likely to suboptimal file size because strips/tiles will be
> rewritten many times. I'd rather use uncompressed TIFF for intermediate work and
> as a final step do a gdal_translate -of LZW.
>
> Best regards,
>
> Even
>
>> Hi,
>>
>> Over the weekend, I tried to gdal_rasterize a 350+ MB shapefile with
>> 149671 polygon features into a 64116 x 61850 GeoTIFF.
>>
>> I first had some trouble while trying to rasterize everything at once,
>> because after some time, the gdal_rasterize process got killed by the
>> kerned (that's how I understood it).
>>
>> I then tried to loop through the shapefile's CODE values, something like
>> the bash code [1] below. I ended up with different errors, see [2]
>>
>> Any idea what is going on and how to avoid these errors? (I am using
>> GDAL v1.8dev from trunk)
>>
>> Thanks in advance, Hermann
>>
>> PS I tried without LZW compression, but this generates a 3+GB GeoTIFF,
>> which in return seems to lead to other (performance) problems.
>>
>>
>> [1]
>>
>> for file in *.shp ; do
>> for code in {0..20} ; do
>> if [[ $code = 0 ]] ; then
>> gdal_rasterize --debug on --config GDAL_CACHEMAX 1000 \
>> -ot byte -co compress=lzw -tr 2 2 -tap -l ${file%.shp} \
>> -burn $code -where CODE=\'$code\' \
>> $file ${file%.shp}_2m.tif> ${file%.shp}_2m.log 2>&1
>> else
>> gdal_rasterize --debug on --config GDAL_CACHEMAX 1000 \
>> -l ${file%.shp} -burn $code -where CODE=\'$code\' \
>> $file ${file%.shp}_2m.tif>> ${file%.shp}_2m.log 2>&1
>> fi
>> done
>> done
>>
>>
>> [2]
>>
>> $ head -90 uk001l_london_2m.log
>> OGR: OGROpen(shpfiles/uk001l_london.shp/0x764610) succeeded as ESRI
>> Shapefile.
>> GDAL:
>> GDALDriver::Create(GTiff,uk001l_london_2m.tif,64116,61850,1,Byte,0x764ee0)
>> Shape: 149671 features read on layer 'uk001l_london'.
>> GDAL: GDALClose(uk001l_london_2m.tif, this=0x782e00)
>> GDAL: GDALDeregister_GTiff() called.
>> OGR: OGROpen(shpfiles/uk001l_london.shp/0x2080610) succeeded as ESRI
>> Shapefile.
>> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x209feb0) succeeds as GTiff.
>> 0...10...20...30...40...50...60.OGR:
>> OGROpen(shpfiles/uk001l_london.shp/0x10c1610) succeeded as ESRI Shapefile.
>> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x10e0eb0) succeeds as GTiff.
>> .0...70..10.....2080.....30.90......Shape: 149671 features read on layer
>> 'uk001l_london'.
>> 40.GDAL: GDALClose(uk001l_london_2m.tif, this=0x209feb0)
>> GDAL: GDALDeregister_GTiff() called.
>> 100 - done.
>> ..50..OGR: OGROpen(shpfiles/uk001l_london.shp/0x225e610) succeeded as
>> ESRI Shapefile.
>> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x227deb0) succeeds as GTiff.
>> 0....6010....20.....3070.....OGR:
>> OGROpen(shpfiles/uk001l_london.shp/0x1543610) succeeded as ESRI Shapefile.
>> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1562eb0) succeeds as GTiff.
>> 0...10...2040.....3080.....50..40....90...ERROR 1: LZWDecode:Wrong
>> length of decoded string: data probably corrupted at scanline 36908
>> ERROR 1: TIFFReadEncodedStrip() failed.
>>
>> ERROR 1: IReadBlock failed at X offset 0, Y offset 36908
>> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 36908
>> Shape: 149671 features read on layer 'uk001l_london'.
>> ..GDAL: GDALClose(uk001l_london_2m.tif, this=0x227deb0)
>> GDAL: GDALDeregister_GTiff() called.
>> 50...Shape: 149671 features read on layer 'uk001l_london'.
>> .ERROR 1: LZWDecode:Wrong length of decoded string: data probably
>> corrupted at scanline 36908
>> ERROR 1: TIFFReadEncodedStrip() failed.
>>
>> ERROR 1: IReadBlock failed at X offset 0, Y offset 36908
>> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 36908
>> Shape: 149671 features read on layer 'uk001l_london'.
>> GDAL: GDALClose(uk001l_london_2m.tif, this=0x10e0eb0)
>> GDAL: GDALDeregister_GTiff() called.
>> 100 - done.
>> GDAL: GDALClose(uk001l_london_2m.tif, this=0x1562eb0)
>> GDAL: GDALDeregister_GTiff() called.
>> OGR: OGROpen(shpfiles/uk001l_london.shp/0xa21610) succeeded as ESRI
>> Shapefile.
>> GDAL: GDALOpen(uk001l_london_2m.tif, this=0xa40eb0) succeeds as GTiff.
>> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
>> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
>> ERROR 1: TIFFReadEncodedStrip() failed.
>>
>> ERROR 1: IReadBlock failed at X offset 0, Y offset 863
>> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
>> Shape: 149671 features read on layer 'uk001l_london'.
>> GDAL: GDALClose(uk001l_london_2m.tif, this=0xa40eb0)
>> GDAL: GDALDeregister_GTiff() called.
>> OGR: OGROpen(shpfiles/uk001l_london.shp/0x1cd3610) succeeded as ESRI
>> Shapefile.
>> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1cf2eb0) succeeds as GTiff.
>> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
>> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
>> ERROR 1: TIFFReadEncodedStrip() failed.
>>
>> ERROR 1: IReadBlock failed at X offset 0, Y offset 863
>> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
>> Shape: 149671 features read on layer 'uk001l_london'.
>> GDAL: GDALClose(uk001l_london_2m.tif, this=0x1cf2eb0)
>> GDAL: GDALDeregister_GTiff() called.
>> OGR: OGROpen(shpfiles/uk001l_london.shp/0x1cc5610) succeeded as ESRI
>> Shapefile.
>> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1ce4eb0) succeeds as GTiff.
>> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
>> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
>> ERROR 1: TIFFReadEncodedStrip() failed.
>>
>> ERROR 1: IReadBlock failed at X offset 0, Y offset 863
>> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
>> Shape: 149671 features read on layer 'uk001l_london'.
>> GDAL: GDALClose(uk001l_london_2m.tif, this=0x1ce4eb0)
>> GDAL: GDALDeregister_GTiff() called.
>> OGR: OGROpen(shpfiles/uk001l_london.shp/0x172e610) succeeded as ESRI
>> Shapefile.
>> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x174deb0) succeeds as GTiff.
>> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
>> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
>> ERROR 1: TIFFReadEncodedStrip() failed.
>>
>> ERROR 1: IReadBlock failed at X offset 0, Y offset 863
>> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
>> Shape: 149671 features read on layer 'uk001l_london'.
>> GDAL: GDALClose(uk001l_london_2m.tif, this=0x174deb0)
>> GDAL: GDALDeregister_GTiff() called.
>> OGR: OGROpen(shpfiles/uk001l_london.shp/0x1276610) succeeded as ESRI
>> Shapefile.
>> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1295eb0) succeeds as GTiff.
>> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
>> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
>> ERROR 1: TIFFReadEncodedStrip() failed.
>>
>> ERROR 1: IReadBlock failed at X offset 0, Y offset 863
>> ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 863
>> Shape: 149671 features read on layer 'uk001l_london'.
>> GDAL: GDALClose(uk001l_london_2m.tif, this=0x1295eb0)
>> GDAL: GDALDeregister_GTiff() called.
>> OGR: OGROpen(shpfiles/uk001l_london.shp/0x1c0c610) succeeded as ESRI
>> Shapefile.
>> GDAL: GDALOpen(uk001l_london_2m.tif, this=0x1c2beb0) succeeds as GTiff.
>> 0Warning 1: LZWDecode:LZWDecode: Strip 863 not terminated with EOI code
>> ERROR 1: LZWDecode:Not enough data at scanline 863 (short 17574 bytes)
>> ERROR 1: TIFFReadEncodedStrip() failed.
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
More information about the gdal-dev
mailing list