[gdal-dev] Problems generating a COG file

Wed Aug 20 13:43:14 PDT 2025

> Can I do other useful tests tonight?

How long does it take to fall asleep?

-Jukka Rahkonen-

________________________________________
Lähettäjä: gdal-dev <gdal-dev-bounces at lists.osgeo.org> käyttäjän Javier Jimenez Shaw via gdal-dev <gdal-dev at lists.osgeo.org> puolesta
Lähetetty: Keskiviikko 20. elokuuta 2025 23.39
Vastaanottaja: Even Rouault <even.rouault at spatialys.com>
Kopio: gdal dev <gdal-dev at lists.osgeo.org>
Aihe: Re: [gdal-dev] Problems generating a COG file

On Wed, 20 Aug 2025 at 21:41, Javier Jimenez Shaw <j1 at jimenezshaw.com> wrote:On Wed, 20 Aug 2025 at 20:31, Even Rouault <even.rouault at spatialys.com> wrote:Javier,- which GDAL version it is ?$ gdalinfo --versionGDAL 3.11.3 "Eganville", released 2025/07/12(I installed conda and qgis today in that machine) - the "corrupted double-linked list" is definitely a proof of either a bug, or a mis-configuration. This is a message we see sometimes when GDAL accidentally links against 2 different libproj. Can you check "ldd /path/to/libgdal.so" to see if there is not 2 libproj appearing ?$ ldd anaconda3/envs/qgis/lib/libgdal.so | grep projlibproj.so.25 => /home/jshaw/anaconda3/envs/qgis/lib/./libproj.so.25 (0x00007f2af442e000) - can you reproduce the issue with a fully blank file with the same characterists. For example try "gdal_create -if web_mercator.tif web_mercator_blank.tif -co SPARSE_OK=YES -co TILED=YES", and then run the gdal_translate to COG on that web_mercator_blank.tifIt says it takes 2 hours... but it was only 30 minThe final file was 1.8 GB- does running in single threaded mode with GDAL_NUM_THREADS=1 make a difference regarding the error messages?I started right after the first email a session without defining  GDAL_NUM_THREADS. It said 12 hours... let's see tomorrow. (after connecting to check) So far it goes to 40%, much more than before that crashed about 13% (and only 3 hours left)It was faster than expected. It is done in 2:44 hours.The source file was the one compressed with LZW, so the LZWDecode message is not in the data. It looks like something in the multithreading (that you already guessed).The output works in QGIS. 3.7 GB.Can I do other useful tests tonight? BTW, htop shows 16 cores and 64 GB of RAM.EvenLe 20/08/2025 à 19:17, Javier Jimenez Shaw via gdal-dev a écrit :HiI am warping a file to EPSG:3857 and later generating a COG with JPEG compression.The input tif file has also JPEG compression. (I learned that I need the -dstalpha to keep it transparent)I am doing it in Ubuntu 22.04 using conda.First I tried this, using LZW as output of the warpGDAL_CACHEMAX=16GB GDAL_NUM_THREADS=ALL_CPUS gdalwarp -t_srs EPSG:3857 -ct "+proj=pipeline +step +inv +proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +step +proj=push +v_3 +step +proj=cart +ellps=airy +step +proj=helmert +x=446.448 +y=-125.157 +z=542.06 +rx=0.15 +ry=0.247 +rz=0.842 +s=-20.489 +convention=position_vector +step +inv +proj=cart +ellps=WGS84 +step +proj=pop +v_3 +step +proj=webmerc +lat_0=0 +lon_0=0 +x_0=0 +y_0=0 +ellps=WGS84" -co COMPRESS=LZW -co BIGTIFF=YES -co TILED=YES -dstalpha orthomosaic.tiff web_mercator.tifCreating output file that is 677881P x 303664L.Processing orthomosaic.tiff [1/1] : 0...10...20...30...40...50...60...70...80...90...100 - done in 00:27:04.                 then I try to generate the COGGDAL_CACHEMAX=16GB  GDAL_NUM_THREADS=ALL_CPUS gdal_translate web_mercator.tif web_mercator_cog.tif -co COMPRESS=JPEG -co BIGTIFF=YES -of COGInput file size is 677881, 3036640ERROR 1: LZWDecode:Not enough data at scanline 0 (short 40448 bytes)0.                                                   - estimated remaining time: 00:14:34and failedAs it is an LZW problem, I tried with DEFLATE:GDAL_CACHEMAX=16GB GDAL_NUM_THREADS=ALL_CPUS gdalwarp -t_srs EPSG:3857 -ct "+proj=pipeline +step +inv +proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +step +proj=push +v_3 +step +proj=cart +ellps=airy +step +proj=helmert +x=446.448 +y=-125.157 +z=542.06 +rx=0.15 +ry=0.247 +rz=0.842 +s=-20.489 +convention=position_vector +step +inv +proj=cart +ellps=WGS84 +step +proj=pop +v_3 +step +proj=webmerc +lat_0=0 +lon_0=0 +x_0=0 +y_0=0 +ellps=WGS84" -co COMPRESS=DEFLATE -co BIGTIFF=YES -co TILED=YES -dstalpha orthomosaic.tiff web_mercator_d.tifCreating output file that is 677881P x 303664L.Processing orthomosaic.tiff [1/1] : 0...10...20...30...40...50...60...70...80...90...100 - done in 00:27:34.                 And then the sameGDAL_CACHEMAX=16GB  GDAL_NUM_THREADS=ALL_CPUS gdal_translate web_mercator_d.tif web_mercator_cog.tif -co COMPRESS=JPEG -co BIGTIFF=YES -of COGInput file size is 677881, 3036640..                                                  - estimated remaining time: 02:29:49ERROR 1: ZIPDecode:Decoding error at scanline 00...10...                                            - estimated remaining time: 00:42:12and failed :(Trying again I got this output at a similar point (but nothing about ZIPDecode:Decoding error at scanline 0).corrupted double-linked listAborted (core dumped)What can it be? Is there any workaround?The file is big in pixels, but a big part is transparent. web_mercator_d.tif is "only" 14GBThank you. _______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
-- 
http://www.spatialys.com
My software is free, but my time generally not.