[gdal-dev] ZSTD compression level 17 smaller than 18+?

Mon Mar 15 02:35:34 PDT 2021

I don’t think it’s particularly unusual for any compression algorithm to have this sort of behaviour once it’s “maxed out”.

Running the in-built zstd benchmarks from the command line without any custom parameters – so entirely separate of how GDAL uses it – it shows your observed behaviour to an even greater extent, with most levels doing worse than level 1!

Level
Size
% Smaller than Level 1
1
3154783
0.00%
2
3130807
0.76%
3
3231415
-2.43%
4
3346308
-6.07%
5
3335646
-5.73%
6
3340284
-5.88%
7
3324298
-5.37%
8
3315170
-5.08%
9
3321085
-5.27%
10
3355473
-6.36%
11
3355932
-6.38%
12
3355857
-6.37%
13
3355401
-6.36%
14
3354678
-6.34%
15
3353801
-6.31%
16
3083731
2.25%
17
3142331
0.39%
18
3147466
0.23%
19
3146084
0.28%
20
3146548
0.26%
21
3145580
0.29%
22
3142972
0.37%

Table generated from output of `zstd -b1 -e22`, “zstd command line interface 64-bits v1.4.7”.

Dr Daniel Evans
Software Developer

From: gdal-dev <gdal-dev-bounces at lists.osgeo.org> On Behalf Of Matt.Wilkie at yukon.ca
Sent: 15 March 2021 04:04
To: gdal-dev at lists.osgeo.org
Subject: [gdal-dev] ZSTD compression level 17 smaller than 18+?

So this is curious: output to Cloud optimized Geotiff with ZSTD compression to Level 17 is smaller than levels 18-22. Or at least that's my results when testing compression results with Gdal 3.1 via Qgis 3.18 on Windows 10. I expected compression levels to get smaller with higher levels, at expense of longer and longer computation time. Levels 1 to 16 do that. Even more curious than 17 being the smallest of the lot is that each step 18 to 22 is progressively larger, albeit by a miniscule amount.

Here's my results:

% smaller than biggest
Bytes
Filename
Compression
Predictor
Level
24.2%
200,968,821
cog-zstd-pyes-lev17.tif
zstd
yes
17
22.8%
204,567,155
cog-zstd-pyes-lev18.tif
zstd
yes
18
22.8%
204,594,678
cog-zstd-pyes-lev19.tif
zstd
yes
19
22.8%
204,594,678
cog-zstd-pyes-lev20.tif
zstd
yes
20
22.8%
204,626,722
cog-zstd-pyes-lev21.tif
zstd
yes
21
22.8%
204,635,985
cog-zstd-pyes-lev22.tif
zstd
yes
22
22.6%
205,026,301
cog-zstd-pyes-lev16.tif
zstd
yes
16
21.5%
208,039,012
cog-zstd-pyes-lev15.tif
zstd
yes
15
21.5%
208,094,559
cog-zstd-pyes-lev13.tif
zstd
yes
13
21.5%
208,094,560
cog-zstd-pyes-lev14.tif
zstd
yes
14
21.3%
208,714,731
cog-zstd-pyes-lev12.tif
zstd
yes
12
21.2%
208,989,301
cog-zstd-pyes-lev11.tif
zstd
yes
11
21.2%
208,989,994
cog-zstd-pyes-lev10.tif
zstd
yes
10
21.1%
209,071,549
cog-zstd-pyes-lev9.tif
zstd
yes
9
21.0%
209,453,858
cog-zstd-pyes-lev8.tif
zstd
yes
8
20.9%
209,634,360
cog-zstd-pyes-lev7.tif
zstd
yes
7
20.7%
210,182,178
cog-zstd-pyes-lev6.tif
zstd
yes
6
20.4%
210,992,074
cog-zstd-pyes-lev5.tif
zstd
yes
5
18.9%
214,894,175
cog-zstd-pyes-lev4.tif
zstd
yes
4
17.4%
218,911,762
cog-zstd-pyes-lev3.tif
lzw
2
3
16.6%
220,991,236
cog-dflt-p2.tif
deflate
2
13.0%
230,553,902
cog-zstd-pyes-lev2.tif
zstd
yes
2
11.6%
234,277,147
cog-zstd-pyes-lev1.tif
zstd
yes
1
5.7%
249,979,858
cog-dflt-p1.tif
deflate
1
0.0%
265,062,854
cog-lzw-p2.tif
lzw
2

Command line pattern:

gdal_translate ^

  -co compress=zstd ^

  -co predictor=yes ^

  -co level=3 ^

  -co num_threads=all_cpus ^

  -of cog ^

  mtn-view.tif ^

  mtn-view-cog-zstd-pyes-lev3.tif

Source image:

Name

AerialPhotos/AerialPhotos

URL

https://mapservices.gov.yk.ca/imagery/rest/services/AerialPhotos/AerialPhotos/ImageServer

Source

crs='EPSG:3578' format='JPGPNG' layer='' url='https://mapservices.gov.yk.ca/imagery/rest/services/AerialPhotos/AerialPhotos/ImageServer'

CRS

EPSG:3578 - NAD83 / Yukon Albers - Projected

Export extent

Upper Left, Lower Left, Upper Right, Lower Right

356750.816,  702590.026

356750.816,  699097.526

361572.583,  702590.026

361572.583,  699097.526

Pixel size

0.5m

Do others see the same thing?

Cheers,

Matt Wilkie
Environment | Information Management & Technology | 867-667-8133 🌄 Yukon.ca<http://yukon.ca/>

T +44 (0)1756 799919
www.jbarisk.com<http://www.jbarisk.com>

[Visit our website]<http://www.jbarisk.com> [http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-LINKEDIN.png]  [Follow us on Twitter] <https://twitter.com/jbarisk>

Find out more about us here: www.jbarisk.com<http://www.jbarisk.com/> and follow us on Twitter @JBARisk<http://twitter.com/JBARisk> and LinkedIn<https://www.linkedin.com/company/2370847?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A2370847%2Cidx%3A2-1-2%2CtarId%3A1447414259786%2Ctas%3AJBA%20RISK%20MANAGEMENT>

The JBA Group supports the JBA Trust.

All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.

JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire, BD23 3FD, Telephone: +441756799919

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20210315/46c88cd2/attachment-0001.html>