[gdal-dev] Re: TiffAppendToStrip errors with uncompressed output

[Armin Burger]

Jukka

I downloaded and converted your test VRT file to a single Geotiff 
without any problems - on Linux... Conversion took a few minutes on a 
fast server.

Just a guess, but in my opinion this problem is related to the limited 
memory handling capabilities of Windows. It does not seem to be a 
shortage of total memory but more a sort of memory "fragmentation". 
Windows is not able to provide a contiguous block of memory that is 
required by some software in some cases. Some time ago a colleague had 
this type of problem when working with IDL scripts, it worked fine on 
Linux or Mac OS, never on Windows with larger datasets.

On Windows, working with large datasets and gdal (especially compressed 
and VRT) or allowing gdal_translate to use more memory via GDAL_CACHEMAX 
config, caused always problems for me. So I stopped trying this... On 
Linux I can easily work with VRT files that reference several 100's of 
GB. There on Linux however, higher settings for GDAL_CACHEMAX were 
speeding up conversions noticeably when working with lots of tiles, 
especially when they were compressed Tiffs.

So I can just recommend, if feasible, to try out Linux or any type of 
Unix for your processing and see if the problem disappears.

Greetings
Armin


On 08/05/2012 14:32, Jukka Rahkonen wrote:
> Jukka Rahkonen<jukka.rahkonen<at>  mmmtike.fi>  writes:
>
>>
>> Hi,
>>
>> I have some LZW compressed paletted 8-bit tiff files which I look
> through a .vrt
>> file. I have been taking some timings with different gdal_translate
> parameters
>> but for some reason the most common command fails always for me. The
> result is
>>
>> gdal_translate -of GTiff test.vrt uncompressed.tif
>>
>> Input file size is 96000, 48000
>> 0...10...20...30ERROR 1: TIFFAppendToStrip:Write error at scanline 14650
>> ERROR 1: TIFFAppendToStrip:Write error at scanline 14650
>> ERROR 1: An error occured while writing a dirty block
>> ERROR 1: TIFFAppendToStrip:Write error at scanline 14664
>> ERROR 1: TIFFAppendToStrip:Write error at scanline 14664
>
> I can repeat the error with another computer (Vista 32-bit). This time
> the error occurs at scanline 44731 when progress bar had advanced to
> value of 90.
> Obviously there is a lack of some resource but I cannot imagine what it is and
> how I could give more of it for gdal_translate. Both computers have 2 GB of
> memory but resource manager does not show signs about running out of memory.
>
> There is now a 460 MB zip file at
> http://latuviitta.org/documents/TiffAppendToStrip_error.zip
>
> It contains the source tiffs and .vrt file "UM52.vrt" and after unzipping it
> should be ready for a test with
> gdal_translate -of GTiff UM542.vrt output.tif
>
> SRS is EPSG:3067 if it matters.
>
> -Jukka Rahkonen-
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>