[postgis-devel] Proposal for sensible number output semantics

[rmrodriguez at carto.com]

On Sat, Apr 18, 2020 at 12:18 AM Martin Davis <mtnclimb at gmail.com> wrote:
> Paul & I experimented a bit with numeric output, to see what the effect of the precision parameter really is currently.  It definitely seems a bit... odd.

It's nice to see it's not only me :D

> A proposal that fixes this problem, has relatively simple semantics, and doesn't require changing the precision default value is:

Yesterday I spent some time looking into ryu's smallest round-trip
string, and I think it's simpler and hopefully doable without too much
complication to have a single common semantic for all double printing:
Print only as many digits as necessary to fulfill round-trip
conversion, but limit the amount of decimal digits to the ones
requested by the user.

This gives us several things:
- The output is guaranteed to be the shortest possible, without losing
information when imported back.
- Requesting precision >= 17 is probably meaningless, since at most we
are printing 17 digits for 64bit doubles. I think that once the round
trip is guaranteed, nobody will need a 20 or 30 decimal precision
since you can't (or shouldn't) operate on strings.
- Requesting precision < 15 will still have some use, especially
things like precision 0..4.
- There is no artificial distinction between < 15 and > 15, which
eliminates inconsistencies. Note that in your proposal you would get
odd things like this:
  - 0.300000011920928955078125 with precision 10 -> 0.3000000119
  - 0.300000011920928955078125 with precision 20 -> 0.3 (since 0.3 has
the same double representation)

What do you think about this?

Note that I need to spend some more time tweaking ryu as currently it
only supports scientific notation for its shortest representation, but
I hope that getting the shortest representation in every case will
offset the impact in unit/regression tests.



--
Raúl Marín Rodríguez
carto.com