[gdal-dev] Problems generating a COG file

Wed Aug 20 12:41:34 PDT 2025

On Wed, 20 Aug 2025 at 20:31, Even Rouault <even.rouault at spatialys.com>
wrote:

> Javier,
>
> - which GDAL version it is ?
>
$ gdalinfo --version
GDAL 3.11.3 "Eganville", released 2025/07/12

(I installed conda and qgis today in that machine)

> - the "corrupted double-linked list" is definitely a proof of either a
> bug, or a mis-configuration. This is a message we see sometimes when GDAL
> accidentally links against 2 different libproj. Can you check "ldd
> /path/to/libgdal.so" to see if there is not 2 libproj appearing ?
>
$ ldd anaconda3/envs/qgis/lib/libgdal.so | grep proj
libproj.so.25 => /home/jshaw/anaconda3/envs/qgis/lib/./libproj.so.25
(0x00007f2af442e000)

> - can you reproduce the issue with a fully blank file with the same
> characterists. For example try "gdal_create -if web_mercator.tif
> web_mercator_blank.tif -co SPARSE_OK=YES -co TILED=YES", and then run the
> gdal_translate to COG on that web_mercator_blank.tif
>
It says it takes 2 hours... but it was only 30 min
The final file was 1.8 GB

> - does running in single threaded mode with GDAL_NUM_THREADS=1 make a
> difference regarding the error messages?
>
I started right after the first email a session without
defining  GDAL_NUM_THREADS. It said 12 hours... let's see tomorrow.
(after connecting to check) So far it goes to 40%, much more than before
that crashed about 13% (and only 3 hours left)

BTW, htop shows 16 cores and 64 GB of RAM.

Even
> Le 20/08/2025 à 19:17, Javier Jimenez Shaw via gdal-dev a écrit :
>
> Hi
>
> I am warping a file to EPSG:3857 and later generating a COG with JPEG
> compression.
> The input tif file has also JPEG compression. (I learned that I need the -
> dstalpha to keep it transparent)
> I am doing it in Ubuntu 22.04 using conda.
>
> First I tried this, using LZW as output of the warp
>
> GDAL_CACHEMAX=16GB GDAL_NUM_THREADS=ALL_CPUS gdalwarp -t_srs EPSG:3857 -ct
> "+proj=pipeline +step +inv +proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717
> +x_0=400000 +y_0=-100000 +ellps=airy +step +proj=push +v_3 +step +proj=cart
> +ellps=airy +step +proj=helmert +x=446.448 +y=-125.157 +z=542.06 +rx=0.15
> +ry=0.247 +rz=0.842 +s=-20.489 +convention=position_vector +step +inv
> +proj=cart +ellps=WGS84 +step +proj=pop +v_3 +step +proj=webmerc +lat_0=0
> +lon_0=0 +x_0=0 +y_0=0 +ellps=WGS84" -co COMPRESS=LZW -co BIGTIFF=YES -co
> TILED=YES -dstalpha orthomosaic.tiff web_mercator.tif
>
> Creating output file that is 677881P x 303664L.
> Processing orthomosaic.tiff [1/1] :
> 0...10...20...30...40...50...60...70...80...90...100 - done in 00:27:04.
>
>
> then I try to generate the COG
> GDAL_CACHEMAX=16GB  GDAL_NUM_THREADS=ALL_CPUS gdal_translate
> web_mercator.tif web_mercator_cog.tif -co COMPRESS=JPEG -co BIGTIFF=YES -of
> COG
> Input file size is 677881, 303664
> 0ERROR 1: LZWDecode:Not enough data at scanline 0 (short 40448 bytes)
> 0.                                                   - estimated remaining
> time: 00:14:34
>
> and failed
>
> As it is an LZW problem, I tried with DEFLATE:
>
> GDAL_CACHEMAX=16GB GDAL_NUM_THREADS=ALL_CPUS gdalwarp -t_srs EPSG:3857 -ct
> "+proj=pipeline +step +inv +proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717
> +x_0=400000 +y_0=-100000 +ellps=airy +step +proj=push +v_3 +step +proj=cart
> +ellps=airy +step +proj=helmert +x=446.448 +y=-125.157 +z=542.06 +rx=0.15
> +ry=0.247 +rz=0.842 +s=-20.489 +convention=position_vector +step +inv
> +proj=cart +ellps=WGS84 +step +proj=pop +v_3 +step +proj=webmerc +lat_0=0
> +lon_0=0 +x_0=0 +y_0=0 +ellps=WGS84" -co COMPRESS=DEFLATE -co BIGTIFF=YES
> -co TILED=YES -dstalpha orthomosaic.tiff web_mercator_d.tif
>
> Creating output file that is 677881P x 303664L.
> Processing orthomosaic.tiff [1/1] :
> 0...10...20...30...40...50...60...70...80...90...100 - done in 00:27:34.
>
>
> And then the same
> GDAL_CACHEMAX=16GB  GDAL_NUM_THREADS=ALL_CPUS gdal_translate
> web_mercator_d.tif web_mercator_cog.tif -co COMPRESS=JPEG -co BIGTIFF=YES
> -of COG
> Input file size is 677881, 303664
> 0..                                                  - estimated remaining
> time: 02:29:49ERROR 1: ZIPDecode:Decoding error at scanline 0
> 0...10...                                            - estimated remaining
> time: 00:42:12
>
> and failed :(
>
> Trying again I got this output at a similar point (but nothing about ZIPDecode:Decoding
> error at scanline 0).
> corrupted double-linked list
> Aborted (core dumped)
>
> What can it be? Is there any workaround?
> The file is big in pixels, but a big part is transparent.
> web_mercator_d.tif is "only" 14GB
>
> Thank you.
>
> _______________________________________________
> gdal-dev mailing listgdal-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250820/07b6256c/attachment.htm>