[postgis-devel] Proposal for sensible number output semantics

[rmrodriguez at carto.com]

On Sat, Apr 18, 2020 at 6:18 PM Martin Davis <mtnclimb at gmail.com> wrote:

> When you say "limit" do you mean truncate or round?   I don't think truncation is a good approach (based on what every other precision-limiting system I have seen does).
> If you mean round then I *think* your suggestion is essentially the same as mine.

Ideally, rounding.



> I would say precision 0..8 is most useful; beyond that not as much.
>
> But surely there has to be some transition point at which rounding stops and "full" precision is used?

Yes, but rounding might stop at different points


>> - There is no artificial distinction between < 15 and > 15, which
>> eliminates inconsistencies. Note that in your proposal you would get
>> odd things like this:
>>   - 0.300000011920928955078125 with precision 10 -> 0.3000000119
>>   - 0.300000011920928955078125 with precision 20 -> 0.3 (since 0.3 has
>> the same double representation)
>
>
> I don't see how  0.300000011920928955078125 has the same DP representation as 0.3. DP has about 16 digits of precision, so surely the first number would be represented as (close to) 0.300000011920929 ?

Sorry, I took a 32bit float example (from tryu) so it doesn't work
with 64bits. Let's see a 64 bit double example:
```

postgis=# Select ST_AsText(ST_MakePoint(0,
0.299999999999999988897769753748), 30);
st_astext

POINT(0 0.299999999999999988897769753748)
(1 row)

postgis=# Select ST_AsText(ST_MakePoint(0, 0.3), 30);
st_astext

POINT(0 0.299999999999999988897769753748)
(1 row)
```

In both those examples, for anything except precision 0, IMO the
output should be POINT(0 0.3). Why? Because both
0.29999999999999998889776975374 and 0.3 have the same double
representation (in hex 0x3fd3333333333333) so when you import 0.3 you
are getting the same value, thus it makes sense to output the shortest
representation in all cases.

>From what I understood of your proposal, there would be a precision
(17+) where you would get more digits beyond 0.3. In my proposal you
would never get any more digits.

-- 
Raúl Marín Rodríguez
carto.com