<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:508952619;
mso-list-type:hybrid;
mso-list-template-ids:1140470704 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Just wanted to follow up on this because I didn’t want to leave the impression that the issue was resolved, or limited to jpeg compression. It seems to be an issue with writing the internal mask band.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">When I create a mask band in a large lzw-compressed or jpeg-compressed tif using the COG driver it dramatically increases processing time over writing RGBA (hours instead of minutes), so the issue is not jpeg compression, it’s the creation
of the mask band. Steps to reproduce:<o:p></o:p></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Take a decent-sized RGBA LZW tiff<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Generate a LZW COG with -b 1 -b 2 -b 3 -b 4 –config GDAL_TIFF_INTERNAL_MASK YES and time it<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Generate a LZW COG with -b 1 -b 2 -b 3 -mask 4 –config GDAL_TIFF_INTERNAL_MASK YES and time it<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Compare times. When I do this with a sourcefile that’s RGBA COG 102600x91100 my time doing (1) is about 2 minutes and my time doing (2) is about 120 minutes<o:p></o:p></li></ol>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I also noticed that the -co BIGTIFF=NO option appears to be ignored in the COG driver. I can share a file if that’s helpful (can not provide a link on the listserv)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Is there a faster way to generate an external nodata mask then add it? From reading the GTIFF and COG format notes on internal masks it wasn’t clear but I didn’t see a way to specify copying masks in the COG driver.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Ritchie, Andrew C <br>
<b>Sent:</b> Wednesday, April 15, 2020 12:26 PM<br>
<b>To:</b> Even Rouault <even.rouault@spatialys.com>; gdal-dev@lists.osgeo.org<br>
<b>Subject:</b> RE: [EXTERNAL] Re: [gdal-dev] gdal_translate (3.1.0dev) "never" finishes on large jpeg cogs... REALLLLLY long time to unload.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Even,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for the quick response! The source dataset is a LZW cog with RGBA, and I confirmed (I think) that the issue was the mask layer by playing with the switches I used to generate the LZW cog – I didn’t even have to do a JPEG COG. I can
cause the same, or very similar behavior, by changing from:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-b 1 -b 2 -b 3 -b 4 <o:p></o:p></p>
<p class="MsoNormal">to:<o:p></o:p></p>
<p class="MsoNormal">-b 1 -b 2 -b 3 -mask 4 <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">with GDAL_TIFF_INTERNAL_MASK YES. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">With the -b 4 switch (or omitting all -b and -mask switches), I get LZW cogs in 2 minutes. With -mask 4 I get hung up at 20% with directory thrashing messages in debug for at least 30 minutes, and I’m guessing I’ll get the same behavior
at the “done” message if I care to wait.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Below are the two configurations that show such a difference in performance for me. I didn’t play around with CACHEMAX or MAX_DATASET_POOL_SIZE, was trying to keep it simple.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">2 minute TIFFs:<o:p></o:p></p>
<p class="MsoNormal">gdal_translate <infile> <outfile> -b 1 -b 2 -b 3 -b 4 -of COG -co COMPRESS=LZW -co PREDICTOR=2 -co NUM_THREADS=ALL_CPUS -co RESAMPLING=AVERAGE –config GDAL_TIFF_INTERNAL_MASK YES –config GDAL_TIF_OVR_BLOCKSIZE 128<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">A couple orders of magnitude longer:<o:p></o:p></p>
<p class="MsoNormal">gdal_translate <infile> <outfile> -b 1 -b 2 -b 3 -mask 4 -of COG -co COMPRESS=LZW -co PREDICTOR=2 -co NUM_THREADS=ALL_CPUS -co RESAMPLING=AVERAGE –config GDAL_TIFF_INTERNAL_MASK YES –config GDAL_TIF_OVR_BLOCKSIZE 128<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>>
<br>
<b>Sent:</b> Wednesday, April 15, 2020 4:38 AM<br>
<b>To:</b> <a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
<b>Cc:</b> Ritchie, Andrew C <<a href="mailto:aritchie@usgs.gov">aritchie@usgs.gov</a>><br>
<b>Subject:</b> [EXTERNAL] Re: [gdal-dev] gdal_translate (3.1.0dev) "never" finishes on large jpeg cogs... REALLLLLY long time to unload.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"Courier New"">Andrew,<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0">
<span style="font-size:9.0pt;font-family:"Courier New"">Has your source raster an alpha band ? That could explain the difference since it isn't possible to directly create a YCbCrA JPEG compressed file, but internally a mask band must be created. However I
wouldn't anticipate such a huge difference in performance between compression schemes. I would suggest not setting GDAL_CACHEMAX at all and letting it at its 5% default (increasing it is not always a good idea), in case it would be a performance issue at de-allocating
cached blocks. <o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0">
<span style="font-size:9.0pt;font-family:"Courier New"">Even<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0">
<span style="font-size:9.0pt;font-family:"Courier New"">-- <o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">Spatialys - Geospatial professional services<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><a href="http://www.spatialys.com"><span style="font-size:9.0pt;font-family:"Courier New"">http://www.spatialys.com</span></a><span style="font-size:9.0pt;font-family:"Courier New""><o:p></o:p></span></p>
</div>
</body>
</html>