<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi Jeremy and Even,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Sorry I should’ve run more tests to clarify the situation re BIGTIFFs. It looks like gdal_translate honors -co BIGTIFF=NO for the raster but not the mask.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If I omit the mask, then I don’t see any bigtiff messages with -co BIGTIFF=NO. If I include the mask band (change -b 4 to -mask 4 with lzw-compressed) then with –debug on, then in the first few seconds of running I see the following lines:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">COG: Generating overviews of the mask<o:p></o:p></p>
<p class="MsoNormal">GTIFF: File being created as a BIGTIFF.<o:p></o:p></p>
<p class="MsoNormal">GTIFF: Using 24 threads for compression<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Incidentally, when I kill the process with ctrl-C (on a windoze machine) GDAL fails to exit gracefully (2 of 2 times this run) with the following as the final debug message<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">GDAL: Flushing dirty blocks: 0GTIFF: Waiting for worker job to finish handling block 0<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My cmd:<o:p></o:p></p>
<p class="MsoNormal">gdal_translate <infile.tif> <outfile.tif> -b 1 -b 2 -b 3 -mask 4 -of cog -co COMPRESS=LZW -co PREDICTOR=2 -co NUM_THREADS=ALL_CPUs -co RESAMPLING=AVERAGE -co BIGTIFF=NO –config GDAL_TIF_OVR_BLOCKSIZE 128 –debug ON<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Jeremy – to clarify, I have confirmed that if I wait long enough, the COG will finish, so generating in the background is feasible if slow. I was just surprised that including a transparency mask increases the processing time so much. It’s
necessary to reduce the file size using jpeg or webp compression and still provide transparency I guess, but it’s a huge performance penalty to pay. I don’t have enough programming experience (or time) to do profiling and figure out what the bottleneck is,
and don’t get me wrong – I <span style="font-family:"Segoe UI Emoji",sans-serif">
❤</span> gdal x 10^10, but I thought this was worth mentioning because of the increase in time (which is so long I initially thought it was actually a hang).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">As far as the steps to generate a COG – I output tiled tiffs, then create a VRT, then create a RGBA LZW cog, preview, and generate a JPEG COG. I only added the RGBA LZW cog because of the issues I was having generating the JPG cog – it’s
actually a good point to delete the tiles in my workflow because I can go back to the LZW cog again and again if I need to since it’s lossless.<u><o:p></o:p></u></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Andy<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>From:</b> Jeremy Palmer <palmerjnz@gmail.com> <br>
<b>Sent:</b> Tuesday, April 21, 2020 2:57 PM<br>
<b>To:</b> Even Rouault <even.rouault@spatialys.com><br>
<b>Cc:</b> Ritchie, Andrew C <aritchie@usgs.gov>; gdal-dev@lists.osgeo.org<br>
<b>Subject:</b> [EXTERNAL] Re: [gdal-dev] gdal_translate (3.1.0dev) "never" finishes on large jpeg cogs... REALLLLLY long time to unload.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Hi Andrew,<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Apr 22, 2020 at 6:11 AM Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"Courier New"">Andrew,<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"Courier New"">> When I create a mask band in a large lzw-compressed or jpeg-compressed tif<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"Courier New"">> using the COG driver it dramatically increases processing time over writing<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"Courier New"">> RGBA (hours instead of minutes), so the issue is not jpeg compression, it's<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"Courier New"">> the creation of the mask band. Steps to reproduce:<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"Courier New"">There's clearly some complications & overhead internally to be able to deal with non-alpha mask bands. I wouldn't have expected it to be as large as you observed
though. Would require deeper investigation to understand why that causes such performance difference.<o:p></o:p></span></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I would also be interested to see what steps you have taken to process the COG. We have used the driver to create both WEBP, and JPEG (including the 1bit transparency mask) COGs up to 200GB in size. It did take a couple of days on a mid-sized
AWS EC2 instance but completed OK. Note the most of the time is taken up with the overview generation on large COGs. This will hopefully improve soon with multi-threaded overview computation likely coming to GDAL.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Warm Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Jeremy<o:p></o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>