[postgis-devel] Proposal for sensible number output semantics

[Martin Davis]

On Sat, Apr 18, 2020 at 12:22 PM <rmrodriguez at carto.com> wrote:

> On Sat, Apr 18, 2020 at 6:18 PM Martin Davis <mtnclimb at gmail.com> wrote:
>
> > When you say "limit" do you mean truncate or round?   I don't think
> truncation is a good approach (based on what every other precision-limiting
> system I have seen does).
> > If you mean round then I *think* your suggestion is essentially the same
> as mine.
>
> Ideally, rounding.
>

Ah, that's good.

>
> > I would say precision 0..8 is most useful; beyond that not as much.
> >
> > But surely there has to be some transition point at which rounding stops
> and "full" precision is used?
>
> Yes, but rounding might stop at different points
>

My idea is that 15 decimal digits is about the limit of DP precision.
Below that rounding would be noticeable, beyond it it would not be.  It
could be 14 or 13 instead, I think.  The key point is that the current
default will produce full precision.

>> - There is no artificial distinction between < 15 and > 15, which
> >> eliminates inconsistencies. Note that in your proposal you would get
> >> odd things like this:
> >>   - 0.300000011920928955078125 with precision 10 -> 0.3000000119
> >>   - 0.300000011920928955078125 with precision 20 -> 0.3 (since 0.3 has
> >> the same double representation)
> >
> >
> > I don't see how  0.300000011920928955078125 has the same DP
> representation as 0.3. DP has about 16 digits of precision, so surely the
> first number would be represented as (close to) 0.300000011920929 ?
>
> Sorry, I took a 32bit float example (from tryu) so it doesn't work
> with 64bits. Let's see a 64 bit double example:
> ```
>
> postgis=# Select ST_AsText(ST_MakePoint(0,
> 0.299999999999999988897769753748), 30);
> st_astext
>
> POINT(0 0.299999999999999988897769753748)
> (1 row)
>
> postgis=# Select ST_AsText(ST_MakePoint(0, 0.3), 30);
> st_astext
>
> POINT(0 0.299999999999999988897769753748)
> (1 row)
> ```
>

Okay, that makes more sense.


>
> In both those examples, for anything except precision 0, IMO the
> output should be POINT(0 0.3). Why? Because both
> 0.29999999999999998889776975374 and 0.3 have the same double
> representation (in hex 0x3fd3333333333333) so when you import 0.3 you
> are getting the same value, thus it makes sense to output the shortest
> representation in all cases.
>

Yes, and I think that will happen with the proposed scheme?

>
> From what I understood of your proposal, there would be a precision
> (17+) where you would get more digits beyond 0.3. In my proposal you
> would never get any more digits.
>

No, at 15 and above requested digits you would get "full precision" and no
more.

The intent is NOT to allow the user to force a meaningless precision to be
output.  There are two goals:

a) allow output of numbers with a textual precision that matches the known
DP precision.  For instance, if ST_SnapToGrid(geom, .01) is used then
output should only show 2 decimal digits (even if roundoff actually causes
a bit of drift in the internal representation.
b) allow output of full (round-trippable) precision, with that as the
default

Part of the motivation for (a) is that I hoping that "soon" PostGIS will
gain the ability to execute some operations to a specified precision (in
particular overlay, and also a better geometry rounding capability) .  This
will make moving data in out of PostGIS more robust, since it will prevent
creating geometry in PostGIS that carries excess precision and then becomes
invalid when exported at a lower precision.

Ideally we should capture all this as an RFC, and then (if implemented)
document it in the manual (with examples).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-devel/attachments/20200418/d3898930/attachment.html>