[Proj] Numerical Precision in Proj, WKT, and everywhere

[Gerald I. Evenden]

On Friday 02 April 2010 3:52:24 pm Craig Bruce wrote:
> Gerald I. Evenden <geraldi.evenden at gmail.com> wrote:
> > BCD? Binary Coded Decimal? Just that I have not heard the term in maybe
> > 25 years. Seriously. I am not being sarcastic.
>
> All the common database systems use BCD to represent the very common
> NUMBER data type.  This allows them to represent terminating decimal
> fractions exactly and SQL allows numbers to have up to 38 decimal digits.
> See the section "Packed decimal numbers" near the bottom of:
>
> http://publib.boulder.ibm.com/infocenter/db2luw/v9r7/index.jsp?topic=/com.i
>bm.db2.luw.sql.ref.doc/doc/r0001030.html
>
> The down side is that calculations are slower (not preformed directly
> by hardware), but this slowness is hidden by the large overhead costs of
> database access.

I have long ago given up worrying about speed when dealing with DBMS issues. 
Gives headaches. Just a good excuse to buy a faster machine.

> > As I recall, the only place you demonstrated precision problems was
> > an example that went to full mantissa precision where I admit there
> > may be problems.  *But* I know of no place in cartography where such
> > precision is required.
>
> I'm speaking much more generally than cartography.  I'm saying that all
> doubles should be displayed in the way I suggest in general by default
> unless you know the values have errors in them or the application
> domain suggests using fewer digits.  (Most of the code I write is very
> general purpose.)  Otherwise, you're introducing arbitrary errors into
> the numbers, causing a greater divergence between a binary->binary and
> a binary->text->binary round trip if such a round trip can't be avoided.
> You need to use 17 digits to get a completely clean round trip, but this
> exposes the eyesore patterns in the IEEE encoding.

I philosophically disagree with you here. Know the characteristics of the data 
you are storing and the expected precision and use appropriate field widths.

If you are worrying about speed, the general case is likely to be very 
wasteful.

It is hypothetically possible that you may encounter extended precision data 
and need to specify 20, 30 or more digit precision. You're stuck if you 
assume a 64 bit float limit of 17 decimal.

My old addage: know your data.

-- 
The whole religious complexion of the modern world is due
to the absence from Jerusalem of a lunatic asylum.
-- Havelock Ellis (1859-1939) British psychologist