<div dir="ltr"><div>Hi Andy,</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 22, 2020 at 8:33 AM Ritchie, Andrew C <<a href="mailto:aritchie@usgs.gov">aritchie@usgs.gov</a>> wrote:</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_875883053731227198WordSection1">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Sorry I should’ve run more tests to clarify the situation re BIGTIFFs. It looks like gdal_translate honors -co BIGTIFF=NO for the raster but not the mask.</p></div></div></blockquote><div><br></div><div>What's the output size of your COG when it successful completes?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_875883053731227198WordSection1">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Incidentally, when I kill the process with ctrl-C (on a windoze machine) GDAL fails to exit gracefully (2 of 2 times this run) with the following as the final debug message<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">GDAL: Flushing dirty blocks: 0GTIFF: Waiting for worker job to finish handling block 0</p></div></div></blockquote><div><br></div><div>In my experience, the progress reporting in GDAL is not very good and can spend a lot of time in the flushing dirty blocks process. It might be that you can't interrupt GDAL at this point. I would wait a little longer. Even will be able to comment further on this. <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_875883053731227198WordSection1"><p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">My cmd:<u></u><u></u></p>
<p class="MsoNormal">gdal_translate <infile.tif> <outfile.tif> -b 1 -b 2 -b 3 -mask 4 -of cog -co COMPRESS=LZW -co PREDICTOR=2 -co NUM_THREADS=ALL_CPUs -co RESAMPLING=AVERAGE -co BIGTIFF=NO –config GDAL_TIF_OVR_BLOCKSIZE 128 –debug ON</p></div></div></blockquote><div><br></div><div>Seems ok to me. For our processing of aerial RGB photos COGs, when we are interested in web mapping use and a good balance between storage size and quality, we go for something like:</div><div><br></div><div>gdalbuildvrt \<br> -addalpha -hidenodata \<br> $PWD/$TIF_FOLDER.vrt \<br> $PWD/$TIF_FOLDER/*.tif<br></div><div><br></div><div>gdal_translate \<br> -of COG \<br> -co COMPRESS=WebP \<br> -co NUM_THREADS=ALL_CPUS \<br> -co BIGTIFF=YES \<br> -co TILING_SCHEME=GoogleMapsCompatible \<br> --config BIGTIFF_OVERVIEW YES \<br> -co ALIGNED_LEVELS=3 \<br> -co ADD_ALPHA=YES \<br> -co BLOCKSIZE=512 \<br> -co RESAMPLING=CUBIC \<br> $PWD/$TIF_FOLDER.vrt $PWD/$TIF_FOLDER.webp.google.aligned.cog.tif</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_875883053731227198WordSection1"><p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Jeremy – to clarify, I have confirmed that if I wait long enough, the COG will finish, so generating in the background is feasible if slow. I was just surprised that including a transparency mask increases the processing time so much. It’s
necessary to reduce the file size using jpeg or webp compression and still provide transparency I guess, but it’s a huge performance penalty to pay. I don’t have enough programming experience (or time) to do profiling and figure out what the bottleneck is,
and don’t get me wrong – I <span style="font-family:"Segoe UI Emoji",sans-serif">
❤</span> gdal x 10^10, but I thought this was worth mentioning because of the increase in time (which is so long I initially thought it was actually a hang). </p></div></div></blockquote><div><br>First, I would consider using WebP if you think your users can handle that. It's way better than JPEG+Mask. Note I'm surprised that adding the mask to the tiff is adding heaps of additional time. Can you generate your dataset with and without the mask to see the time difference? As mentioned before, most of the processing time is taken up in the overview generation (especially when compared to the data compression stage, which can use all of your CPU cores). Hopefully, some upcoming GDAL improvements can improve this situation.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_875883053731227198WordSection1"><p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">As far as the steps to generate a COG – I output tiled tiffs, then create a VRT, then create a RGBA LZW cog, preview, and generate a JPEG COG. I only added the RGBA LZW cog because of the issues I was having generating the JPG cog – it’s
actually a good point to delete the tiles in my workflow because I can go back to the LZW cog again and again if I need to since it’s lossless.</p></div></div></blockquote><div><br></div><div>What was the issue you were having with JPEG compression? Just time to process? I would try the above command to see if that gives a good result (remove warping to GoogleMap projection if you don't need that as that adds a lot to processing times)</div><div><br></div><div>Cheers,<br>Jeremy</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_875883053731227198WordSection1"><div><div><div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div></div>