[gdal-dev] ZSTD compression level 17 smaller than 18+?

Tue Mar 16 08:38:12 PDT 2021

Oh that’s so weird. Well that puts my take-away as: just use Level=17 and forget the rest exist!

-Matt

From: Daniel Evans <Daniel.Evans at jbarisk.com>
Sent: March 15, 2021 2:36 AM
To: Matt.Wilkie <Matt.Wilkie at yukon.ca>; gdal-dev at lists.osgeo.org
Subject: RE: ZSTD compression level 17 smaller than 18+?

*** External email: Do not click on links or attachments except from trusted senders. ***

******************************************************************************************

I don’t think it’s particularly unusual for any compression algorithm to have this sort of behaviour once it’s “maxed out”.

Running the in-built zstd benchmarks from the command line without any custom parameters – so entirely separate of how GDAL uses it – it shows your observed behaviour to an even greater extent, with most levels doing worse than level 1!

Level

Size

% Smaller than Level 1

1

3154783

0.00%

2

3130807

0.76%

3

3231415

-2.43%

4

3346308

-6.07%

5

3335646

-5.73%

6

3340284

-5.88%

7

3324298

-5.37%

8

3315170

-5.08%

9

3321085

-5.27%

10

3355473

-6.36%

11

3355932

-6.38%

12

3355857

-6.37%

13

3355401

-6.36%

14

3354678

-6.34%

15

3353801

-6.31%

16

3083731

2.25%

17

3142331

0.39%

18

3147466

0.23%

19

3146084

0.28%

20

3146548

0.26%

21

3145580

0.29%

22

3142972

0.37%

Table generated from output of `zstd -b1 -e22`, “zstd command line interface 64-bits v1.4.7”.

Dr Daniel Evans
Software Developer

From: gdal-dev <gdal-dev-bounces at lists.osgeo.org<mailto:gdal-dev-bounces at lists.osgeo.org>> On Behalf Of Matt.Wilkie at yukon.ca<mailto:Matt.Wilkie at yukon.ca>
Sent: 15 March 2021 04:04
To: gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Subject: [gdal-dev] ZSTD compression level 17 smaller than 18+?

So this is curious: output to Cloud optimized Geotiff with ZSTD compression to Level 17 is smaller than levels 18-22. Or at least that's my results when testing compression results with Gdal 3.1 via Qgis 3.18 on Windows 10. I expected compression levels to get smaller with higher levels, at expense of longer and longer computation time. Levels 1 to 16 do that. Even more curious than 17 being the smallest of the lot is that each step 18 to 22 is progressively larger, albeit by a miniscule amount.

Here's my results:

% smaller than biggest

Bytes

Filename

Compression

Predictor

Level

24.2%

200,968,821

cog-zstd-pyes-lev17.tif

zstd

yes

17

22.8%

204,567,155

cog-zstd-pyes-lev18.tif

zstd

yes

18

22.8%

204,594,678

cog-zstd-pyes-lev19.tif

zstd

yes

19

22.8%

204,594,678

cog-zstd-pyes-lev20.tif

zstd

yes

20

22.8%

204,626,722

cog-zstd-pyes-lev21.tif

zstd

yes

21

22.8%

204,635,985

cog-zstd-pyes-lev22.tif

zstd

yes

22

22.6%

205,026,301

cog-zstd-pyes-lev16.tif

zstd

yes

16

21.5%

208,039,012

cog-zstd-pyes-lev15.tif

zstd

yes

15

21.5%

208,094,559

cog-zstd-pyes-lev13.tif

zstd

yes

13

21.5%

208,094,560

cog-zstd-pyes-lev14.tif

zstd

yes

14

21.3%

208,714,731

cog-zstd-pyes-lev12.tif

zstd

yes

12

21.2%

208,989,301

cog-zstd-pyes-lev11.tif

zstd

yes

11

21.2%

208,989,994

cog-zstd-pyes-lev10.tif

zstd

yes

10

21.1%

209,071,549

cog-zstd-pyes-lev9.tif

zstd

yes

9

21.0%

209,453,858

cog-zstd-pyes-lev8.tif

zstd

yes

8

20.9%

209,634,360

cog-zstd-pyes-lev7.tif

zstd

yes

7

20.7%

210,182,178

cog-zstd-pyes-lev6.tif

zstd

yes

6

20.4%

210,992,074

cog-zstd-pyes-lev5.tif

zstd

yes

5

18.9%

214,894,175

cog-zstd-pyes-lev4.tif

zstd

yes

4

17.4%

218,911,762

cog-zstd-pyes-lev3.tif

lzw

2

3

16.6%

220,991,236

cog-dflt-p2.tif

deflate

2

13.0%

230,553,902

cog-zstd-pyes-lev2.tif

zstd

yes

2

11.6%

234,277,147

cog-zstd-pyes-lev1.tif

zstd

yes

1

5.7%

249,979,858

cog-dflt-p1.tif

deflate

1

0.0%

265,062,854

cog-lzw-p2.tif

lzw

2

Command line pattern:

gdal_translate ^

  -co compress=zstd ^

  -co predictor=yes ^

  -co level=3 ^

  -co num_threads=all_cpus ^

  -of cog ^

  mtn-view.tif ^

  mtn-view-cog-zstd-pyes-lev3.tif

Source image:

Name

AerialPhotos/AerialPhotos

URL

https://mapservices.gov.yk.ca/imagery/rest/services/AerialPhotos/AerialPhotos/ImageServer

Source

crs='EPSG:3578' format='JPGPNG' layer='' url='https://mapservices.gov.yk.ca/imagery/rest/services/AerialPhotos/AerialPhotos/ImageServer'

CRS

EPSG:3578 - NAD83 / Yukon Albers - Projected

Export extent

Upper Left, Lower Left, Upper Right, Lower Right

356750.816,  702590.026

356750.816,  699097.526

361572.583,  702590.026

361572.583,  699097.526

Pixel size

0.5m

Do others see the same thing?

Cheers,

Matt Wilkie
Environment | Information Management & Technology | 867-667-8133 🌄 Yukon.ca<http://yukon.ca/>

T +44 (0)1756 799919
www.jbarisk.com<http://www.jbarisk.com>

[Visit our website]<http://www.jbarisk.com> [http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-LINKEDIN.png]  [Follow us on Twitter] <https://twitter.com/jbarisk>

Find out more about us here: www.jbarisk.com<http://www.jbarisk.com/> and follow us on Twitter @JBARisk<http://twitter.com/JBARisk> and LinkedIn<https://www.linkedin.com/company/2370847?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A2370847%2Cidx%3A2-1-2%2CtarId%3A1447414259786%2Ctas%3AJBA%20RISK%20MANAGEMENT>

The JBA Group supports the JBA Trust.

All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.

JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire, BD23 3FD, Telephone: +441756799919

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20210316/f2bb83f3/attachment-0001.html>