Will the scope of floating point variables affect their values?

Question

If we execute the following C# code on a console application, we will get a message as The sums are Not equal.

If we execute it after uncommenting the line System.Console.WriteLine(), we will get a message as The sums are equal.

    static void Main(string[] args)
    {
        float f = Sum(0.1f, 0.2f);
        float g = Sum(0.1f, 0.2f);

        //System.Console.WriteLine("f = " + f + " and g = " + g);

        if (f == g)
        {
            System.Console.WriteLine("The sums are equal");
        }
        else
        {
            System.Console.WriteLine("The sums are Not equal");
        }
    }

    static float Sum(float a, float b)
    {
        System.Console.WriteLine(a + b);
        return a + b;
    }

What is the actual reason for this behavior?

Frail

I have an inkling about this in general, but I don't understand why uncommenting that line would have that effect, as it's doing the same thing to both variables. It may well vary by processor type too, btw. – Hazel

When I run it (vs2010) I get that they are equal, what compiler are you using? – Shortcake

In both cases on my PC (commenting or uncommenting the line), I get the expected behavior (Sums are equal). There must be something different in your code than what you copied as they will always be the same in that scenario. – Elmaelmajian

@NahuelI.: No, there isn't necessarily anything different in the code. This sort of thing can be very subtle, and vary by processor, optimization level etc. – Hazel

I've reproduced the problem on my machine - building and running from the command line with csc /o+ /debug- /platform:x86 Test.cs. It may well still vary by subtle things like CLR version etc though. – Hazel

That's why doubles should always be compared with Math.Abs(a - b) < epsilon, where epsilon is required precision. True then a equal b, with reqired precision. – Docila

Please go to microsoft connect and report this as a bug. – Eulogize

@AK_: Floating point comparisons being black magic that vary based on basically anything is expected behaviour, even if this is astonishing I doubt you could get it classified as a bug. – Ramonitaramos

@Atomosk: The results should be exactly the same (since they are the results of the same function with exactly the same arguments), and the WriteLine should not change their values, so this is unexpected. It has nothing to do with the general problem of comparing FP values. – Disabled

@RudyVelthuis: Your statement that the results should be the same when given the same arguments is a moral statement. Yes, that it how the world should be. That is not the world that Intel gave us. The CLR jitter can choose entirely at its whim whether to keep intermediate results in high-precision registers vs low-precision stack or heap locations, and doing so can and does change the results of equalities like this one. If you think that's morally wrong, complain to Intel for giving us a chipset where precision differs based on storage location. – Izaguirre

@AK_: This is not a bug. This is documented, specified behaviour that is the unfortunate consequence of decisions made by the chip designers. The C# and CLR specifications both call out that this behaviour is possible, so it is not a bug. Don't complain to Microsoft; there's nothing Microsoft can do about it without wrecking performance of all floating point operations. The company to complain to is Intel. – Izaguirre

@EricLippert: The implementation may match the spec, but I would regard as defective any language specification which does not allow one to declare a variable with the assurance that when used in a single thread, every read will yield the last value written, and casting that value to the variable's type will be value preserving. It may be useful for a language to also include a "temp fp" type which weakens the semantics so a value could have higher precision but still guarantees that that every read following a write must yield the same value, and... – Winnifredwinning

...I would have no problem with the ECMA spec not forcing rounding when a variable is reused if languages added an "conv" instruction to force rounding when values are assigned to variables of types other than "temp fp". – Winnifredwinning

@supercat: I note that the CLR -- not C#, but the CLR -- does promise that if the variable is an array element or field then you'll get a truncated-to-64-bits double. But locals and temporary values can be jitted as 64 bit stack locations or 80 bit float registers as the jitter sees fit. Also, C# does not document, but does implement the fact that casting a double to double always truncates it back to 64 bits. – Izaguirre

Performance is worthless if the program doesn't do what it's supposed to. This is MS's fault. – Struthious

@EricLippert OK, I get what you are saying, and im not familiar with FP in .Net. But why doesn't the CLR preferrs SSE, which supposed to be a standtard by now (and IIRC deprecate the x87 FPU instructions)? Or why not provide a compiler option for better FP behaviour the same way VC++ does: msdn.microsoft.com/en-us/library/e7s85ffb.aspx – Eulogize

@EricLippert BTW why not have Extended Double percission i.e. 80 bits? – Eulogize

@AK_: Having programmers use extended double would be incompatible with Intel's desire to eliminate it. What should have happened ages ago, IMHO, would have been for languages to provide a means for programmers to indicate what kind of floating-point semantics should be used in different contexts. There are times when code will need to e.g. compute the sum of three or more float values numbers with accuracy beyond that which would be obtained if the intermediate results were rounded to float, and other times when one needs the exact result that would be achieved when results are rounded. – Winnifredwinning

@AK_: For floating-point calculations which are not performed in super-tight loops, the semantic advantages of the 8087 approach (with proper language support) would outweigh any speed penalty; people seeking to maximize FP performance will need to carefully examine the effects of rounding intermediate values but, if they determine that the effects are acceptable, could achieve better performance by doing math on shorter types. – Winnifredwinning

Answer 1

It's not related to scope. It's the combination of the stack dynamics and floating point handling. Some knowledge of compilers will help make this counterintuitive behavior clear.

When the Console.WriteLine is commented, the values f and g are on the evaluation stack and stay there until after you've passed the equality test in your Main method.

When Console.Writeline is not commented, the values f and g are moved from the evaluation stack to the call stack at the moment of the invocation, to be restored to the evaluation stack when Console.WriteLine returns. And your comparison if (f == g) is done afterwards. Some rounding can occur during this storing of values to the call stack and some information can be lost.

In the scenario where you do invoke Console.WriteLine, the f and the g in the comparison test are not the same values. They've been copied and restored to a format that has different rules on precision and rounding, by the virtual machine.

In your particular code, when the invocation of Console.WriteLine is commented, the evaluation stack is never stored to the call stack and no rounding occurs. Because it is permitted for implementations of the platform to provide improved precision on the evaluation stack, this discrepancy can arise.

EDIT What we're hitting in this case is allowed by the CLI specification. In section I.12.1.3 it reads:

Storage locations for floating-point numbers (statics, array elements, and fields of classes) are of fixed size. The supported storage sizes are float32 and float64. Everywhere else (on the evaluation stack, as arguments, as return types, and as local variables) floating-point numbers are represented using an internal floating-point type. In each such instance, the nominal type of the variable or expression is either float32or float64, but its value can be represented internally with additional range and/or precision. The size of the internal floating-point representation is implementation-dependent, can vary, and shall have precision at least as great as that of the variable or expression being represented.

The keywords from this quote are "implementation-dependent" and "can vary". In the OP's case, we see his implementation does indeed vary.

Non-strictfp floating point arithmetic in the Java platform also has a related issue, for more info check also my answer to Will floating point operations on the JVM give the same results on all platforms?

Swaine

But even so, even if they are stored and stored back, and even if rouding takes place when that happens, since they are exactly the same values, the rounded values should still be the same, right? Or is there a random element in rounding in C#? – Disabled

@RudyVelthuis unfortunately during the rounding needed to get the values into the proper IEEE format on the call stack, you lose information (precision, exponent) on certain implementations (like the OPs). Even if you restore the value to a format that supports improved precision, that data is gone, you don't get it back! – Swaine

I now see that, due to how the compiler and runtime work, only one value might lose information, and the other doesn't (see Jon Skeet's answer). That could explain the difference. If both are moved around and rounded, they should be rounded exactly the same way, so even after the loss of precision, their values should still be the same. But if only one is treated that way and not the other, a difference can result. I still don't think that a compiler/runtime, no matter which, should do this. – Disabled

@RudyVelthuis I agree the result is counterintuitive, but it seemed a good idea when they made the spec. Check the CLI spec at ecma-international.org/publications/files/ECMA-ST/ECMA-335.pdf in section I.12.1.3 (bottom of page 75). There they discuss storage locations and what does and doesn't have to have a fixed size. – Swaine

I note that there is similar text in the C# specification as well. – Izaguirre

@Rudy Basically x86 uses an 80-bit internal format (which is a good thing^TM) which consequently mean it'd be actually rather expensive and bad for accuracy to force you to keep all intermediate results in IEEE format. And that for something that's basically broken anyhow (don't compare floats for equivalence and you're good to go). Floating point arithmetic is counterintuitive in even the best situations one more or less really doesn't make a difference. – Jihad

@Voo: What is bad for accuracy is having loosy-goosy floating-point semantics. Computing intermediate results in a higher-precision format is generally a good thing but only when such semantics are consistent. Saying that after float foo = 1E8F+13F; float moo = foo-1E8f; moo might hold 13f rather than 16f is not helpful. – Winnifredwinning

@Winnifredwinning The only thing extended precision does is give you more accurate results for "free" but never less accurate ones. Except for things such as checking the correctness of a computation by equality (e.g. checking whether 2 algorithms generate the same results; but this is broken anyhow, use a delta), getting a more accurate result for free doesn't strike me as problematic. Don't rely on it, but if you get it where's the downside? – Jihad

@Voo: There are numerous downsides. Suppose you are writing a game engine which uses floats. You have the initial state of the game. Sixty times a second the engine consumes the latest user input and the current game state to produce the next game state. Since we have no guarantee that a decision like "did the bullet hit the target?" is ever computed the same way twice, this means that you cannot "play back" the sequence of inputs against the engine and get the same result in the game. – Izaguirre

@Voo: The downside is that many algorithms like Kahan summation work by computing the difference between the amount by which a variable changed when something was added to it, and the size of the thing added. If computing newTotal = delta+oldTotal+nextItem; delta += nextItem-(newTotal-oldTotal), it is absolutely imperative that (newTotal - oldTotal) reflect the amount by which newTotal actually changed. Having (newTotal-oldTotal) reflect the amount by which it should have changed is disastrous. – Winnifredwinning

@Winnifredwinning While I'd be fine with considering Eric's point a bit obscure (although still clearly a real life example), not being able to get an accurate sum of floats (the whole point of kahan after all) strikes me indeed as a problem for a large quantity of algorithms. Do you know how big the problem actually is and if people actually try to avoid this somehow? (I guess in C making the float volatile should work). Very interesting! – Jihad

@Voo: There are options to control FP "strictness", but not on a fine-grained level. I find the lack of such control in .NET especially irksome given the decision to require an explicit cast when storing to a float something whose "official" type is double. Actually, what's really needed IMHO is for languages to support more than one type for each FP format; at minimum, separate types for "exact value" and "approximate value". Any 64-bit type should be implicitly convertible to an "approximate" 32-bit type, but approximate 32-bit types should not be implicitly convertible to 64-bit... – Winnifredwinning

@Voo: ...unless there were a "fuzzy" 64-bit type [meaning computations would be performed as 64-bit, but assignment to an "exact" 64-bit type should require an explicit cast]. If newTotal was an "exact" type, then there would be no doubt about whether a compiler should force a rounding when storing the computation (and read back the rounded value). Unfortunately, I know of no plans to implement any such concept. – Winnifredwinning

@supercat: I agree with your fundamental point: that these sorts of details could have been captured in the type system where they belong; then language designers could choose to expose these details as much or as little as they like to the users of the languages. And while we're on the subject of how the world should be, I'd be a lot happier if there was a version of float that carried an error term around with its value. That would go a long way towards solving the sorts of practical problems people have with floats in scientific computations. – Izaguirre

@Voo: My example might be obscure to you, but when people ask me "why are my float computations not repeatable?" and I turn around and ask them "why do you care?", the most common answer I hear is "I'm writing a game." – Izaguirre

@Eric Sorry if that was a bit too strong, I didn't mean it in a negative way (looking it up in the oxford dictionary obscure really wasn't the right choice of words :-)). I just meant that it's a relatively specific case while "it worsens the accuracy of algorithms such as Kahan summation" is something that concerns many different problem areas. – Jihad

@EricLippert: Given that floating-point operations are performed by languages rather than the framework [so far as I can tell, double, unlike Decimal, doesn't define any operators other than comparisons, and integer primitives don't even define those], what would prevent a compiler from having multiple types which alias to each primitive, but having different types use different semantics with compiler-generated operators? – Winnifredwinning

@EricLippert: Types declared using existing keywords would have to behave the same way as existing types, and one would have to define attributes for exposed aspects of classes to allow semantics to work properly with things like method parameters, but otherwise I would think language designers have quite a bit of freedom in how they implement things. For example, while I'm not sure how such a thing could work well with existing code, I would think it useful to have the sum of two Int32 be a "constrained Int32", which--unlike a normal Int32, would not implicitly convert to Int64. – Winnifredwinning

@EricLippert: [actually, I'd suggest it shouldn't directly convert at all]. Although int32 to int64 is normally a lossless conversion, the implicit conversion in long1 = int1*int2; suggests the possibility of unintentional data loss in the expression feeding it. The expression should be written as either long1 = (int)(int1*int2); or long1 = (long)int1*int2;, so as to either avoid data loss or explicitly recognize the possibility. There's no "constrained Int32" in .NET, but that shouldn't prevent the compiler from recognizing such a concept. – Winnifredwinning

@supercat: In fact one of the static analyzers I work on is precisely to detect an unintentional post-overflow conversion from a 32 bit int multiplication to 64 bits. The fact that one has to write a static analyzer to catch this common error is good evidence that the language design could have been better here. – Izaguirre

@EricLippert: What obstacles would there be toward having a .NET language distinguish among multiple compile-time types for each numeric primitive? Method overloading would be a bit ickier than if multiple .NET types could derive from each primitive, but what other obstacles do you see? – Winnifredwinning

@supercat: The usual objections to such schemes aren't technical; you can typically sort those out. The problem is that it rapidly becomes (1) a user education problem, and (2) a potential backwards compatibility / inter-language-compatibility burden. We already tell people not to use uint and ulong, so why are they there in the first place? Because we want to be backwards compatible with the questionable choices of previous languages. – Izaguirre

@EricLippert: The technical debt from not addressing such issues keeps growing; the remedy I would like to see would be an open split with a language dialect which was designed so that most "sane" code in the old language will compile without change, and code that compiles in both and doesn't rely on weird language corner cases will work compatibly in the new one, and code in the old language which was highly "suspicuous" but actually correct could easily be changed to work compatibly in the new version. – Winnifredwinning

@EricLippert: Imagine how many thousands of lines of code could have been saved if .NET had made it possible for an interface IEnumerable to specify that if the type loader encounters a class foo which claims to implement IEnumerable but doesn't define IEnumerable.GetEnumerator() it would auto-generate IEnumerator IEnumerable.GetEnumerator() { IEnumerable.ClassHelper<foo>.GetEnumerator(this); } [which could on first invocation look for an IEnumerable<T>.GetEnumerator method, cache it, and chain to it]. And imagine how much better Count could work if it could do likewise. – Winnifredwinning

@Mishax: The fact that intermediate results are held on the evaluation stack and not simply in the actual variable is an optimization. But IMO, optimizations should never change the outcome of a program. This one does. – Disabled

@Voo: I am well aware of the internal 80 bit representation of floating point variables. I don't care if it is "expensive" to keep values in the chosen format but if someone chooses to store them as float, those two floats should be compared for equality, not some value that is kept in a register (or FPU stack or whatever) for optimization. Optimization should, IMO, never change the outcome and the logic of a program. – Disabled

@RudyVelthuis I'm not following your reasoning. If improved precision is an optimization, why else would you want this optimization if not to get different (better) results and hence an improved outcome of the program? I'd agree if we were only talking about performance optimization, but this optimization is for better performance and better precision. That the outcome of the program would be different is the whole idea. – Swaine

Improved precision? The programmer chose a float, so he is not interested in improved precision. The optimization is keeping a value in a register (or in the FPU) and not storing it. Both values should be stored in the float variables before WriteLine was called, to keep the result the same. After all, the programmer is comparing two values that ought to be equal. If optimization makes them different, the result of the program is falsified. – Disabled

In other words: an optimization should never change the result of a program. The outcome of the program is not improved at all, it is changed for the worse, since the test, which ought to be that both values are equal, returns a different result. That is not an improvement. – Disabled

@RudyVelthuis: I would regard as fundamentally broken any equality operator which, given operands whose run-time type differs from the specified compile-time type, does not coerce the operands to the compile-time type before the comparison [or behave as though it does so]. I find it ironic that compilers won't allow someFloat += someDouble without a cast (requiring someFloat += (float)someDouble; but given someFloat += (float)double1; someFloat += (float)double2; may translate that as someFloat = (float)((double)someFloat+(double)(float)double1+(double)(float)double2);. – Winnifredwinning

Answer 2

What is the actual reason for this behaviour?

I can't provide details for exactly what's going on in this specific case, but I understand the general problem, and why using Console.WriteLine can change things.

As we saw in your previous post, sometimes operations are performed on floating point types at a higher precision than the one specified in the variable type. For local variables, that can include how the value is stored in memory during the execution of a method.

I suspect that in your case:

• the Sum method is being inlined (but see later)
• the sum itself is being performed with greater precision than the 32-bit float you'd expect
• the value of one of the variables (f say) is being stored in a high-precision register
  • for this variable, the "more precise" result is being stored directly
• the value of the other variable (g) is being stored on the stack as a 32-bit value
  • for this variable, the "more precise" result is being reduced to 32 bits
• when the comparison is performed, the variable on the stack is being promoted to a higher-precision value and compared with the other higher-precision value, and the difference is due to one of them having previously lost information and the other not

When you uncomment the Console.WriteLine statement, I'm guessing that (for whatever reason) forces both variables to be stored in their "proper" 32-bit precision, so they're both being treated the same way.

This hypothesis is all somewhat messed up by the fact that adding

[MethodImpl(MethodImplOptions.NoInlining)]

... does not change the result as far as I can see. I may be doing something else wrong along those lines though.

Really, we should look at the assembly code which is executing - I don't have the time to do that now, unfortunately.

Hazel

This looks like JIT bug. Both results have the same bit pattern, but the issue can only be reproduced if at most one of f and g are referenced. – Mahaliamahan

Changing Sum to return a double also 'fixes' the issue. – Mahaliamahan

David Monniaux wrote an article about why this and more insidious behaviors happen. It is in the context of C, and reflects the state of standard-compliance of C compilers at the time (the situation has improved since), but I think it is still a good read in the context of this question: arxiv.org/abs/cs/0701192 – Rader

@PascalCuoq: Thanks. I'll read up on it when I get a chance :) – Hazel

Is this behavior specific to C#, or can this type of promoting also happen in a language like C, C++, Java? – Wicklund

@MartijnCourteaux: I believe it can affect Java as well... and I'd expect it to potentially affect C and C++ as well (as they typically have even looser definitions of what's allowed in the name of optimization). However, you may well need to poke the runtime in very different ways to see the same effects. – Hazel

And some more info: blogs.msdn.com/b/clrcodegeneration/archive/2013/10/28/… – Mahaliamahan

Answer 3

(Not a real answer but hopefully some supporting documentation)

Configuration: Core i7, Windows 8.1, Visual Studio 2013

Platform x86:

Version      Optimized Code?        Debugger Enabled?          Outcome
4.5.1        Yes                    No                         Not equal
4.5.1        Yes                    Yes                        Equal
4.5.1        No                     No                         Equal
4.5.1        No                     Yes                        Equal
2.0          Yes                    No                         Not Equal
2.0          Yes                    Yes                        Equal
2.0          No                     No                         Equal
2.0          No                     Yes                        Equal

Platform x64:

Version      Optimized Code?        Debugger Enabled?          Outcome
4.5.1        Yes                    No                         Equal
4.5.1        Yes                    Yes                        Equal
4.5.1        No                     No                         Equal
4.5.1        No                     Yes                        Equal
2.0          Yes                    No                         Equal
2.0          Yes                    Yes                        Equal
2.0          No                     No                         Equal
2.0          No                     Yes                        Equal

The situation only seems to occur with optimized code on x86 configurations.

Kotick

I confirm on a similar platform (i5, Win8.1, VS2013) that the only conditions I can get Not Equal is with Release builds without debugger. This happens in AnyCPU configuration when "Prefer 32bit" is checked or in x86, but not in x64 or when unchecked. – Binetta