[postgis-devel] Inconsistencies in text output

[rmrodriguez at carto.com]

Also note that it would be possible to limit the number of digits
(integer or decimal) instead, which could be preferable but that's a
breaking change to all the extension functions (since they are
requesting 15 decimal digits, not 15 total digits).

On Fri, Apr 10, 2020 at 4:07 PM <rmrodriguez at carto.com> wrote:
>
> Hi everyone,
>
> Lately I've been trying to improve the performance of output functions
> and one of the areas where I got a massive (5x) improvement was
> anything that output text (ST_AsText, ST_AsGeoJSON and so on) but the
> implementation that I introduced for 3.1 has multiple hacks to keep
> the output almost exactly the same as it was for 3.0.
>
> Although most output functions have a `maxdecimaldigits` parameter
> that represents `maximum number of decimal digits after floating
> point` there are multiple cases where this isn't respected. Some
> examples:
>
> ```
> Select ST_AsText(ST_MakePoint(123456789012.1234567890123, 0), 4);
> POINT(123456789012.123459 0)
> ```
> The number should have 4 digits, but has 6.
>
> ```
> Select ST_AsText(ST_MakePoint(0, 92114.013999999996), 15);
> POINT(0 92114.014)
>
> Select ST_AsText(ST_MakePoint(0, 92114.013999999996), 20);
> POINT(0 92114.013999999995576)
> ```
>
> This number has a significant digit on the 12th decimal digits, but
> it's not shown if you request 15 decimal digits. But it is shown if
> you request 20 decimal digits.
>
> On the other hand, it doesn't work with ST_AsGeoJSON because it's
> limited internally:
> ```
> Select ST_AsGeoJSON(ST_MakePoint(0, 92114.013999999996), 20);
> {"type":"Point","coordinates":[0,92114.014]}
> ```
>
> So I want to propose some changes to unify the behaviour of all
> functions that output coordinates. They aren't big changes but the
> output of multiple regression tests will change, thus 3.0 and 3.1
> output will change in some cases too; but I think it's worth it as we
> get rid of bugs and we uniformize the output of all functions.
>
> The rules would be like this:
> - For (absolute) values under FP_TOLERANCE, we keep returning '0'.
> - For big numbers (absolute value bigger than 1E15), we keep the
> current behaviour and maintain the decimal digits to 8 - length of the
> decimal part (it's odd but I don't see the need to change it).
> - For smaller numbers:
>   - The integer part is always printed in full (no changes).
>   - The decimal part is always printed up to precision digits
> (removing trailing zeros).
>   - The decimal digit precision is capped between 0 and 20. This is
> more than enough to guarantee round trip (double -> text -> same
> double) and it allows us to know the max size of a double (so we can
> keep the static allocations) as sign + 15 integer digits + "." + 20
> decimal digits + "NULL".
>
> I'm going to draft a PR with the changes, but any comment is more than
> welcome before I push a change like this.
>
> --
> Raúl Marín Rodríguez
> carto.com



-- 
Raúl Marín Rodríguez
carto.com