[postgis-devel] Proposal for sensible number output semantics

[Martin Davis]

On Sat, Apr 18, 2020 at 5:50 AM <rmrodriguez at carto.com> wrote:

>
>
> > A proposal that fixes this problem, has relatively simple semantics, and
> doesn't require changing the precision default value is:
>
> Yesterday I spent some time looking into ryu's smallest round-trip
> string, and I think it's simpler and hopefully doable without too much
> complication to have a single common semantic for all double printing:
> Print only as many digits as necessary to fulfill round-trip
> conversion, but limit the amount of decimal digits to the ones
> requested by the user.
>

When you say "limit" do you mean truncate or round?   I don't think
truncation is a good approach (based on what every other precision-limiting
system I have seen does).

If you mean round then I *think* your suggestion is essentially the same as
mine.

This gives us several things:
> - The output is guaranteed to be the shortest possible, without losing
> information when imported back.
> - Requesting precision >= 17 is probably meaningless, since at most we
> are printing 17 digits for 64bit doubles. I think that once the round
> trip is guaranteed, nobody will need a 20 or 30 decimal precision
> since you can't (or shouldn't) operate on strings.
>

Agreed


> - Requesting precision < 15 will still have some use, especially
> things like precision 0..4.
>

I would say precision 0..8 is most useful; beyond that not as much.

But surely there has to be some transition point at which rounding stops
and "full" precision is used?


> - There is no artificial distinction between < 15 and > 15, which
> eliminates inconsistencies. Note that in your proposal you would get
> odd things like this:
>   - 0.300000011920928955078125 with precision 10 -> 0.3000000119
>   - 0.300000011920928955078125 with precision 20 -> 0.3 (since 0.3 has
> the same double representation)
>

I don't see how  0.300000011920928955078125 has the same DP representation
as 0.3. DP has about 16 digits of precision, so surely the first number
would be represented as (close to) 0.300000011920929 ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-devel/attachments/20200418/f80cc7b5/attachment.html>