<div dir="ltr"><div dir="ltr"></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Wed, 20 Aug 2025 at 21:41, Javier Jimenez Shaw <<a href="mailto:j1@jimenezshaw.com">j1@jimenezshaw.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 20 Aug 2025 at 20:31, Even Rouault <<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div>
<p>Javier,</p>
<p>- which GDAL version it is ?</p></div></blockquote><div>$ gdalinfo --version<br>GDAL 3.11.3 "Eganville", released 2025/07/12</div><div><br></div><div>(I installed conda and qgis today in that machine) </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>- the "corrupted double-linked list" is definitely a proof of
either a bug, or a mis-configuration. This is a message we see
sometimes when GDAL accidentally links against 2 different
libproj. Can you check "ldd /path/to/libgdal.so" to see if there
is not 2 libproj appearing ?</p></div></blockquote><div>$ ldd anaconda3/envs/qgis/lib/libgdal.so | grep proj<br> libproj.so.25 => /home/jshaw/anaconda3/envs/qgis/lib/./libproj.so.25 (0x00007f2af442e000)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>- can you reproduce the issue with a fully blank file with the
same characterists. For example try "gdal_create -if
web_mercator.tif web_mercator_blank.tif -co SPARSE_OK=YES -co
TILED=YES", and then run the gdal_translate to COG on that
web_mercator_blank.tif</p></div></blockquote><div>It says it takes 2 hours... but it was only 30 min</div><div>The final file was 1.8 GB</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>- does running in single threaded mode with GDAL_NUM_THREADS=1
make a difference regarding the error messages?</p></div></blockquote><div>I started right after the first email a session without defining GDAL_NUM_THREADS. It said 12 hours... let's see tomorrow. </div><div>(after connecting to check) So far it goes to 40%, much more than before that crashed about 13% (and only 3 hours left)</div><div><br></div></div></div></blockquote><div><br></div><div>It was faster than expected. It is done in 2:44 hours.</div><div>The source file was the one compressed with LZW, so the LZWDecode message is not in the data. It looks like something in the multithreading (that you already guessed).</div><div>The output works in QGIS. 3.7 GB.</div><div><br></div><div>Can I do other useful tests tonight?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div></div><div><br></div><div>BTW, htop shows 16 cores and 64 GB of RAM.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>Even<br>
</p>
<div>Le 20/08/2025 à 19:17, Javier Jimenez
Shaw via gdal-dev a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hi</div>
<div><br>
</div>
<div>I am warping a file to EPSG:3857 and later generating a COG
with JPEG compression.</div>
<div>The input tif file has also JPEG compression. (I learned
that I need the -<span style="font-family:monospace">dstalpha</span> to
keep it transparent)</div>
<div>I am doing it in Ubuntu 22.04 using conda.</div>
<div><br>
</div>
<div>First I tried this, using LZW as output of the warp</div>
<div><br>
</div>
<div><span style="font-family:monospace">GDAL_CACHEMAX=16GB
GDAL_NUM_THREADS=ALL_CPUS gdalwarp -t_srs EPSG:3857 -ct
"+proj=pipeline +step +inv +proj=tmerc +lat_0=49 +lon_0=-2
+k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +step
+proj=push +v_3 +step +proj=cart +ellps=airy +step
+proj=helmert +x=446.448 +y=-125.157 +z=542.06 +rx=0.15
+ry=0.247 +rz=0.842 +s=-20.489 +convention=position_vector
+step +inv +proj=cart +ellps=WGS84 +step +proj=pop +v_3
+step +proj=webmerc +lat_0=0 +lon_0=0 +x_0=0 +y_0=0
+ellps=WGS84" -co COMPRESS=LZW -co BIGTIFF=YES -co TILED=YES
-dstalpha orthomosaic.tiff web_mercator.tif<br>
</span></div>
<div><br>
</div>
<div><span style="font-family:monospace">Creating output file
that is 677881P x 303664L.<br>
Processing orthomosaic.tiff [1/1] :
0...10...20...30...40...50...60...70...80...90...100 - done
in 00:27:04. </span> </div>
<div><br>
</div>
<div>then I try to generate the COG</div>
<div><span style="font-family:monospace">GDAL_CACHEMAX=16GB
GDAL_NUM_THREADS=ALL_CPUS gdal_translate web_mercator.tif
web_mercator_cog.tif -co COMPRESS=JPEG -co BIGTIFF=YES -of
COG</span></div>
<div><span style="font-family:monospace">Input file size is
677881, 303664<br>
0ERROR 1: LZWDecode:Not enough data at scanline 0 (short
40448 bytes)<br>
0. -
estimated remaining time: 00:14:34</span></div>
<div><br>
</div>
<div>and failed</div>
<div><br>
</div>
<div>As it is an LZW problem, I tried with DEFLATE:</div>
<div><br>
</div>
<div><span style="font-family:monospace">GDAL_CACHEMAX=16GB
GDAL_NUM_THREADS=ALL_CPUS gdalwarp -t_srs EPSG:3857 -ct
"+proj=pipeline +step +inv +proj=tmerc +lat_0=49 +lon_0=-2
+k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +step
+proj=push +v_3 +step +proj=cart +ellps=airy +step
+proj=helmert +x=446.448 +y=-125.157 +z=542.06 +rx=0.15
+ry=0.247 +rz=0.842 +s=-20.489 +convention=position_vector
+step +inv +proj=cart +ellps=WGS84 +step +proj=pop +v_3
+step +proj=webmerc +lat_0=0 +lon_0=0 +x_0=0 +y_0=0
+ellps=WGS84" -co COMPRESS=DEFLATE -co BIGTIFF=YES -co
TILED=YES -dstalpha orthomosaic.tiff web_mercator_d.tif<br>
<br>
Creating output file that is 677881P x 303664L.<br>
Processing orthomosaic.tiff [1/1] :
0...10...20...30...40...50...60...70...80...90...100 - done
in 00:27:34. </span><br>
</div>
<div><br>
</div>
<div>And then the same</div>
<div><span style="font-family:monospace">GDAL_CACHEMAX=16GB
GDAL_NUM_THREADS=ALL_CPUS gdal_translate web_mercator_d.tif
web_mercator_cog.tif -co COMPRESS=JPEG -co BIGTIFF=YES -of
COG</span></div>
<div><span style="font-family:monospace">Input file size is
677881, 303664<br>
0.. -
estimated remaining time: 02:29:49ERROR 1:
ZIPDecode:Decoding error at scanline 0</span></div>
<div><span style="font-family:monospace">0...10...
- estimated remaining time:
00:42:12</span></div>
<div><br>
</div>
<div>and failed :(</div>
<div><br>
</div>
<div>Trying again I got this output at a similar point (but
nothing about <span style="font-family:monospace">ZIPDecode:Decoding
error at scanline 0)</span>.</div>
<div><span style="font-family:monospace">corrupted double-linked
list<br>
Aborted (core dumped)</span></div>
<div><br>
</div>
<div>What can it be? Is there any workaround?</div>
<div>The file is big in pixels, but a big part is transparent. <span style="font-family:monospace">web_mercator_d.tif </span>is
"only" 14GB</div>
<div><br>
</div>
<div>Thank you.</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote></div></div>
</blockquote></div></div>