[gdal-dev] Problems generating a COG file

Wed Aug 20 13:39:16 PDT 2025

On Wed, 20 Aug 2025 at 21:41, Javier Jimenez Shaw <j1 at jimenezshaw.com>
wrote:

>
>
> On Wed, 20 Aug 2025 at 20:31, Even Rouault <even.rouault at spatialys.com>
> wrote:
>
>> Javier,
>>
>> - which GDAL version it is ?
>>
> $ gdalinfo --version
> GDAL 3.11.3 "Eganville", released 2025/07/12
>
> (I installed conda and qgis today in that machine)
>
>> - the "corrupted double-linked list" is definitely a proof of either a
>> bug, or a mis-configuration. This is a message we see sometimes when GDAL
>> accidentally links against 2 different libproj. Can you check "ldd
>> /path/to/libgdal.so" to see if there is not 2 libproj appearing ?
>>
> $ ldd anaconda3/envs/qgis/lib/libgdal.so | grep proj
> libproj.so.25 => /home/jshaw/anaconda3/envs/qgis/lib/./libproj.so.25
> (0x00007f2af442e000)
>
>
>> - can you reproduce the issue with a fully blank file with the same
>> characterists. For example try "gdal_create -if web_mercator.tif
>> web_mercator_blank.tif -co SPARSE_OK=YES -co TILED=YES", and then run the
>> gdal_translate to COG on that web_mercator_blank.tif
>>
> It says it takes 2 hours... but it was only 30 min
> The final file was 1.8 GB
>
>> - does running in single threaded mode with GDAL_NUM_THREADS=1 make a
>> difference regarding the error messages?
>>
> I started right after the first email a session without
> defining  GDAL_NUM_THREADS. It said 12 hours... let's see tomorrow.
> (after connecting to check) So far it goes to 40%, much more than before
> that crashed about 13% (and only 3 hours left)
>
>
It was faster than expected. It is done in 2:44 hours.
The source file was the one compressed with LZW, so the LZWDecode message
is not in the data. It looks like something in the multithreading (that you
already guessed).
The output works in QGIS. 3.7 GB.

Can I do other useful tests tonight?

>
> BTW, htop shows 16 cores and 64 GB of RAM.
>
> Even
>> Le 20/08/2025 à 19:17, Javier Jimenez Shaw via gdal-dev a écrit :
>>
>> Hi
>>
>> I am warping a file to EPSG:3857 and later generating a COG with JPEG
>> compression.
>> The input tif file has also JPEG compression. (I learned that I need the -
>> dstalpha to keep it transparent)
>> I am doing it in Ubuntu 22.04 using conda.
>>
>> First I tried this, using LZW as output of the warp
>>
>> GDAL_CACHEMAX=16GB GDAL_NUM_THREADS=ALL_CPUS gdalwarp -t_srs EPSG:3857
>> -ct "+proj=pipeline +step +inv +proj=tmerc +lat_0=49 +lon_0=-2
>> +k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +step +proj=push +v_3
>> +step +proj=cart +ellps=airy +step +proj=helmert +x=446.448 +y=-125.157
>> +z=542.06 +rx=0.15 +ry=0.247 +rz=0.842 +s=-20.489
>> +convention=position_vector +step +inv +proj=cart +ellps=WGS84 +step
>> +proj=pop +v_3 +step +proj=webmerc +lat_0=0 +lon_0=0 +x_0=0 +y_0=0
>> +ellps=WGS84" -co COMPRESS=LZW -co BIGTIFF=YES -co TILED=YES -dstalpha
>> orthomosaic.tiff web_mercator.tif
>>
>> Creating output file that is 677881P x 303664L.
>> Processing orthomosaic.tiff [1/1] :
>> 0...10...20...30...40...50...60...70...80...90...100 - done in 00:27:04.
>>
>>
>> then I try to generate the COG
>> GDAL_CACHEMAX=16GB  GDAL_NUM_THREADS=ALL_CPUS gdal_translate
>> web_mercator.tif web_mercator_cog.tif -co COMPRESS=JPEG -co BIGTIFF=YES -of
>> COG
>> Input file size is 677881, 303664
>> 0ERROR 1: LZWDecode:Not enough data at scanline 0 (short 40448 bytes)
>> 0.                                                   - estimated
>> remaining time: 00:14:34
>>
>> and failed
>>
>> As it is an LZW problem, I tried with DEFLATE:
>>
>> GDAL_CACHEMAX=16GB GDAL_NUM_THREADS=ALL_CPUS gdalwarp -t_srs EPSG:3857
>> -ct "+proj=pipeline +step +inv +proj=tmerc +lat_0=49 +lon_0=-2
>> +k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +step +proj=push +v_3
>> +step +proj=cart +ellps=airy +step +proj=helmert +x=446.448 +y=-125.157
>> +z=542.06 +rx=0.15 +ry=0.247 +rz=0.842 +s=-20.489
>> +convention=position_vector +step +inv +proj=cart +ellps=WGS84 +step
>> +proj=pop +v_3 +step +proj=webmerc +lat_0=0 +lon_0=0 +x_0=0 +y_0=0
>> +ellps=WGS84" -co COMPRESS=DEFLATE -co BIGTIFF=YES -co TILED=YES -dstalpha
>> orthomosaic.tiff web_mercator_d.tif
>>
>> Creating output file that is 677881P x 303664L.
>> Processing orthomosaic.tiff [1/1] :
>> 0...10...20...30...40...50...60...70...80...90...100 - done in 00:27:34.
>>
>>
>> And then the same
>> GDAL_CACHEMAX=16GB  GDAL_NUM_THREADS=ALL_CPUS gdal_translate
>> web_mercator_d.tif web_mercator_cog.tif -co COMPRESS=JPEG -co BIGTIFF=YES
>> -of COG
>> Input file size is 677881, 303664
>> 0..                                                  - estimated
>> remaining time: 02:29:49ERROR 1: ZIPDecode:Decoding error at scanline 0
>> 0...10...                                            - estimated
>> remaining time: 00:42:12
>>
>> and failed :(
>>
>> Trying again I got this output at a similar point (but nothing about ZIPDecode:Decoding
>> error at scanline 0).
>> corrupted double-linked list
>> Aborted (core dumped)
>>
>> What can it be? Is there any workaround?
>> The file is big in pixels, but a big part is transparent.
>> web_mercator_d.tif is "only" 14GB
>>
>> Thank you.
>>
>> _______________________________________________
>> gdal-dev mailing listgdal-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>> -- http://www.spatialys.com
>> My software is free, but my time generally not.
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250820/3566f2d4/attachment.htm>