<div dir="ltr"><div>In my understanding (derived from metrology), "accuracy" means "how close to an actual measured or measurable quantity is the number our system gives?"</div><div>and "precision" means "how many decimal places does the system give?". If the claimed precision is much more than the actual accuracy, what you get is a <br></div><div>lot of meaningless numbers (insignificant digits, if you will). 1 mm accuracy is hard, 0.1 mm attainable only under very special circumstances and over short</div><div>distances.</div><div><br></div><div>However, for the purpose of checking algorithms against each other, the "accuracy" becomes "does my algorithm give the same number as the one I'm comparing</div><div>against?" (which might be some "official" one). Since it is all arithmetic, the accuracy can in principle be zero, and so (in principle) you want very high precision.</div><div><br></div><div>So, when talking about the level of precision that is appropriate, I think it is important to separate mathematical and physical accuracy.<br></div><div><br></div><div>Duncan Agnew</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Dec 8, 2019 at 7:10 AM Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I'm experimenting with encoding grids with 16-bit integer values, with an <br>
offset and scale, instead of using IEEE 32-bit floats. There's some connection <br>
with my previous thread about accuracies of NTv2...<br>
<br>
Let's for example take the example of egm08_25.gtx. It is 149.3 MB large<br>
If converting it into a IEEE 32-bit float deflate-compressed (with floating-<br>
point predictor) tiled geotiff, it becomes 80.5 MB large (the compression <br>
method and floatin-point predictor are fully lossless. There's bit-to-bit <br>
equivalence for the elevation values regarding the original gtx file)<br>
Now observing that the range of values is in [-107,86], I can remap that to <br>
[0,65535] with an offset of -107 and scale of (86 - -107) / 65535 ~= 0.0029<br>
The resulting deflate-compressed (with integer predictor) tiled GeoTIFF is now <br>
23.1 MB large !<br>
<br>
Looking at the difference between the unscaled quantized values and the <br>
original ones, the error is in the [-1.5 mm, 1.5 mm] range (which is expected, <br>
being the half of the scale value), with a mean value of 4.5e-6 metre (so <br>
centered), and a standard deviation of 0.85 mm<br>
<br>
After that experimentation, I found this interesting page of the GeographicLib <br>
documentation<br>
<a href="https://geographiclib.sourceforge.io/html/geoid.html" rel="noreferrer" target="_blank">https://geographiclib.sourceforge.io/html/geoid.html</a><br>
which compares the errors introduced by the gridding itself and interpolation <br>
methods (the EGM model is originally a continuous model), with/without <br>
quantization. And one conclusion is "If, instead, Geoid were to use data files <br>
without such quantization artifacts, the overall error would be reduced but <br>
only modestly". Actually with the bilinear interpolation we use, the max and <br>
RMS errors with and without quantization are the same... So it seems perfectly <br>
valid to use such quantized products, at least for EGM2008, right ?<br>
<br>
Now looking at horizontal grids, let's consider Australia's <br>
GDA94_GDA2020_conformal.gsb. It is 83 MB large (half of this size due to the <br>
error channels, which are set to a dummy value of -1...)<br>
Converting it to a compressed tiled Float32 tif (without those useless error <br>
channels), make it down to 4.5 MB.<br>
And as a quantitized uint16 compressed tif, down to 1.4 MB (yes, almost 60 <br>
times smaller than original .gsb file). The maximum scale factor is 1.5e-7 <br>
arcsecond, hence a maximum error of 2.3 micrometre... I'm pretty sure we're <br>
several order of magnitudes beyond the accuracy of the original model, right ?<br>
In EPSG this transformation is reported to have an accuracy of 5cm.<br>
The fact that we get such a small scale factor is due to GDA94 -> GDA2020 <br>
conformal being mostly a uniform shift of ~1.8 m and that the grids is <br>
mentioned to "Gives identical results to Helmert transformation GDA94 to <br>
GDA2020 (1)"<br>
<br>
If we look at the France' ntf_r93.gsb, which has shifts of an amplitude up to <br>
130m, the maximum error introduced by the quantization is 0.6 mm. I would tend <br>
to think this is also acceptable (given the size of that particular file is <br>
small, compression gains are quite neglectable, but this is mostly to look if <br>
we can generalize such mechanism). What puzzles me is that in<br>
<a href="https://geodesie.ign.fr/contenu/fichiers/documentation/algorithmes/notice/NT111_V1_HARMEL_TransfoNTF-RGF93_FormatGrilleNTV2.pdf" rel="noreferrer" target="_blank">https://geodesie.ign.fr/contenu/fichiers/documentation/algorithmes/notice/<br>
NT111_V1_HARMEL_TransfoNTF-RGF93_FormatGrilleNTV2.pdf</a> where they compare the <br>
NTv2 approach regarding their native 3D geocentric correction approach, they <br>
underline in red a sample point where the difference between the 2 models is <br>
1.2 mm, as if it had really some importance. For that test point, using the <br>
quantized approach would increase this difference to 1.3 mm. But when looking <br>
at the accuracy reported in the grid at that point it is 1.6e-3 arc-second <br>
(which is the minimum value for the latitude error of the product, by the <br>
way), ie 5cm, so it seems to me that discussing about millimetric error <br>
doesn't make sense.<br>
In EPSG this transformation is reported to have an accuracy of 1 metre (which <br>
is consistent with the mean value of the latitude shift error)<br>
<br>
Now, let's look at the freshly introduced BWTA2017.gsb file. 392 MB large<br>
As a Float32 compressed geotiff: 73 MB (5.4x compression rate)<br>
As a Int16 compressed geotiff: 26 MB (15x compression rate)<br>
Maximum error added by quantization for latitude shift: 0.25 mm<br>
Minimum error value advertized for latitude shift: 1.61e-5 arc-second (not <br>
completely sure about the units...), ie 0.5mm<br>
Mean error value advertized for latitude shift: 6.33e-5 arc-second, ie 1.9mm<br>
Interestingly when looking at the ASCII version of the grid, the values of the <br>
shifts are given with a precision of 1e-6 arcseconds, that is 0.03 mm !<br>
<br>
For Canadian NTv2_0.gsb, on the first subgrid, the quantization error is<br>
0.9 mm for the latitude shift. The advertized error for latitude is in [0, <br>
13.35 m] (the 0.000 value is really surprising. reached on a couple points <br>
only), with a mean at 0.27 m and stddev at 0.48 m. In EPSG this transformation <br>
is reported to have an accuracy of 1.5 metre<br>
<br>
~~~~~~~~~~~~~<br>
<br>
So, TLDR, is it safe (and worth) to generate quantized products to reducte by <br>
about a factor of 2 to 3 the size of our grids compared to unquantized <br>
products, when the maximum error added by the quantization is ~ 1mm or less ? <br>
or will data producers consider we damage the quality of products they <br>
carefully crafted ? do some users need millimetric / sub-millimetric accuracy <br>
?<br>
<br>
Or do we need to condition quantization to a criterion or a combination of <br>
criteria like:<br>
- a maximum absolute error that quantization introduces (1 mm ? 0.1 mm ?)<br>
- a maximum value for the ratio between the maximum absolute error that <br>
quantization introduces over the minimum error value advertized (when known) <br>
below some value ? For the BWTA2017 product, this ratio is 0.5. For <br>
ntf_r93.gsb, 0.012. For NTv2_0.gsb, cannot be computed given sothe min error <br>
value advertized is 0...<br>
- or, variant of the above, a maximum value for the ratio between the maximum <br>
absolute that quantization introduces over the mean of the error value <br>
advertized (when known). For the BWTA2017 product, this ratio is 0.13. For <br>
ntf_r93.gsb, 5.5e-4. For NTv2_0.gsb, 3.3e-3<br>
- and perhaps consider that only for products above a given size (still larger <br>
than 10 MB after lossless compression ?)<br>
<br>
~~~~~~~~~~~~~~~~<br>
<br>
Jochem mentionned in the previous thread that the Netherlands grids have an <br>
accuracy of 1mm. I'm really intrigued by what that means. Does that mean that <br>
the position of control points used to build the grid is known at that <br>
accuracy, both in the source and target systems, and that when using bilinear <br>
interpolation with the grid, one remains within that accuracy ? Actually both <br>
the rdtrans2008.gsb and rdtrans2018.gsb grids report an accuracy of 1mm, but <br>
when comparing the positions corrected by those 2 grids, I get differences <br>
above 1mm.<br>
<br>
echo "6 53 0" | cct -d 9 +proj=hgridshift +grids=./rdtrans2008.gsb<br>
5.999476046 52.998912757<br>
<br>
echo "6 53 0" | cct -d 9 +proj=hgridshift +grids=./rdtrans2018.gsb<br>
5.999476020 52.998912753<br>
<br>
echo "52.998912757 5.999476046 52.998912753 5.999476020" | geod -I <br>
+ellps=GRS80<br>
-104d18'21.49" 75d41'38.51" 0.002<br>
<br>
That's a difference of 2 mm. I get that difference on a few other "random" <br>
points.<br>
<br>
If applying the quantization on rdtrans2018.gsb, we'd add an additional <br>
maximal error of 0.6 mm. The grid being 1.5 MB uncompressed, and 284 KB as a <br>
losslessly compressed TIFF, quantization isn't really worth considering <br>
(reduces the file size to 78 KB)<br>
<br>
Even<br>
<br>
-- <br>
Spatialys - Geospatial professional services<br>
<a href="http://www.spatialys.com" rel="noreferrer" target="_blank">http://www.spatialys.com</a><br>
_______________________________________________<br>
PROJ mailing list<br>
<a href="mailto:PROJ@lists.osgeo.org" target="_blank">PROJ@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/proj" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/proj</a><br>
</blockquote></div>