[gdal-dev] [EXTERNAL] Re: gdal_translate (3.1.0dev) "never" finishes on large jpeg cogs... REALLLLLY long time to unload.

[Even Rouault]

> If I omit the mask, then I don’t see any bigtiff messages with -co 
> BIGTIFF=NO. If I include the mask band (change -b 4 to -mask 4 with lzw-
> compressed) then with –debug on, then in the first few seconds of running I
> see the following lines:
> COG: Generating overviews of the mask
> GTIFF: File being created as a BIGTIFF.

Ah, yes, that's expected and you shouldn't matter. As the file that contains the
overview of the mask is temporary, it is systematically created with BIGTIFF to
avoid any potential issue with size. This has no influence on the final size.

> I’ve definitely identified that it’s the mask generation that takes more
> time and not the jpeg compression. 

Does it take more time than the following step which is "COG: Generating overviews of the imagery" ?

I've tried to replicate on a small VRT of size 10000x9259 made of 2 TIFF RGB+mask

Here's what I get (the "COG: Generating final product" has just been added in master)
with non relevant traces removed

$ gdal_translate in.vrt cog.tif -of COG --debug COG,GDAL --config CPL_TIMESTAMP YES -co COMPRESS=LZW -co PREDICTOR=2 -co NUM_THREADS=ALL_CPUs -co RESAMPLING=AVERAGE -co BIGTIFF=NO --config GDAL_TIF_OVR_BLOCKSIZE 128
[Wed Apr 22 13:17:20 2020].209431: COG: Generating overviews of the mask
[Wed Apr 22 13:17:23 2020].517241: COG: Generating overviews of the imagery
[Wed Apr 22 13:17:33 2020].255700: COG: Generating final product
[Wed Apr 22 13:17:36 2020].264771: GDAL: In GDALDestroy - unloading GDAL shared library.

So basically, generation of the overviews of the mask took 3 seconds, generation of the
overviews of imagery took 10 seconds, and final COG generation 3 seconds

Given the work actually needed to generate a COG, this is what I expect.
 
> I just modified my command to make webp at the same quality setting and it
> looks great in QGIS, and shrinks my test COG from 287MB to 195MB, but
> ArcMap hates it and so does GlobalMapper. Unfortunately as far as I can
> tell the only one that all three of them like is the LZW COGs but those are
> huge. I’m working with GlobalMapper on COGs right now, and I’ll see if I
> can get the ear of our people who talk with ESRI.

ESRI already use (some of) GDAL: they just need to enable more of it.
And GlobalMapper last time I looked (several years ago) didn't seem to use GDAL.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20200422/f87355df/attachment.html>