Should cast of xs:double to xs:decimal be implemented as BigDecimal.valueOf(double) or new BigDecimal(double)?

Question

XQuery, shared with XSLT and XPath 2.0 and later, supports various number data types, two of them are xs:double and xs:decimal. It is possible to cast an xs:double to an xs:decimal, as defined in http://www.w3.org/TR/xquery-operators/#casting-to-numerics.

Implementations done in Java seem to implement xs:double using the Java double data type and xs:decimal using the java.math.BigDecimal class. That class supports two ways of converting a double into a BigDecimal, namely doing BigDecimal.valueOf(doubleValue) and new BigDecimal(doubleValue). According to https://mcmap.net/q/159288/-bigdecimal-to-use-new-or-valueof, the former gives the more intuitive result while the latter gives the more correct result, as for instance BigDecimal.valueOf(1.1) results in 1.1 while new BigDecimal(1.1) results in 1.100000000000000088817841970012523233890533447265625.

When I try the cast of an xs:double to an xs:decimal with Saxon and Exist then

xquery version "1.0";

let $d1 as xs:double := 1.1E0
return xs:decimal($d1)

outputs 1.100000000000000088817841970012523233890533447265625 while with BaseX it outputs 1.1. I suppose the difference results from the different implementations, BaseX doing BigDecimal.valueOf(1.1), Saxon and Exist doing new BigDecimal(1.1).

My question is: which approach is the right one to implement the cast operation according to the http://www.w3.org/TR/xquery-operators/#casting-to-numerics?

Middleton

I would say either way is correct and also all implementations are implementing the spec correctly. It says there If ST is xs:float or xs:double, then TV is the xs:decimal value, within the set of xs:decimal values that the implementation is capable of representing, that is numerically closest to SV and I would say both return the numerically closest value the processor is able to handle. – Idio

Answer 1

An implementation of xs:decimal is required to support at least 18 significant decimal digits, and if it does so, the closest xs:decimal to the xs:double value 1.1 will be something like 1.100 000 000 000 000 088 rather than 1.1. (That's actually 19 digits rather than 18, but even with 18, the difference from 1.1 should show up.) So I believe that returning the decimal 1.1 is non-conformant. The number of digits is implementation-defined beyond 18 digits, but the first 18 must be essentially as shown.

The actual code in Saxon is

public DecimalValue(double in) throws ValidationException {
        try {
            BigDecimal d = new BigDecimal(in);
            value = d.stripTrailingZeros();
        } catch (NumberFormatException err) {
            //...
        }

Amherst

I don't quite follow. If the spec says "at least 18 significant decimal digits", why would returning 1.100 000 000 000 000 000 (which can be shorted to 1.1 without losing a significant number) be incorrect? – Idio

I dind't say it would. 1.1 and 1.100 000 000 000 000 000 are just different representations of the same number, so if you're not actually returning anything different. What I said was that the numeric value of the double 1.1e0 is not the same as the decimal 1.1, and if you hold decimals to 18 digits or more then the difference will show up. – Amherst

@MichaelKay, would format-number(xs:decimal(1.1E0), '0.00000000000000000') eq '1.10000000000000009' be a test case that an XQuery 3.0 implementation should pass if it implements xs:decimal with the at least required 18 significant digits? – Middleton

@MartinHonnen, yes I think that would be a legitimate test, though my appetite for putting edge cases into the test suite is diminished by the amount of discussion that they tend to engender... – Amherst

Answer 2

The XQuery Recommendation leaves it up to the implementation what the closest decimal representation of a double value is supposed to be. The reason is that the specification is supposed to support implementations in arbitrary programming languages.

However, thanks to your hint that BigDecimal.valueOf(d) and new BigDecimal(d) are not equivalent, the next release of BaseX (Version 8.0) will return the more accurate result.

Halfback