I know the following behavior is an old problem, but still I don't understand. System.out.println(0.1 + 0.1 + 0.1); Or even though I use BigDecimal System.out.println(new BigDecimal(0.1).doubl...

I'm running on a 32-bit machine and I'm able to confirm that long values can tear using the following code snippet which hits very quickly. static void TestTearingLong() { System.Threading.Thre...

I have the following piece of code: #include <cstdio> int main() { if ((1.0 + 0.1) != (1.0 + 0.1)) printf("not equal\n"); else printf("equal\n"); return 0; } When compiled with O3 usi...

I have a distance formula using latitude and longitude: distance = EARTH_MILES_RADIUS * Math.acos(Math.sin(lat1 / RADIAN_CONV) * Math.sin(lat2 / RADIAN_CONV) + Math.cos(lat1 / RADIAN_CONV) * M...

I have been making a matrix class (as a learning exercise) and I have come across and issue whilst testing my inverse function. I input a arbitrary matrix as such: 2 1 1 1 2 1 1 1 2 And got it ...

I was testing some of my code, in javascript I added .1+.2 and it gives me .30000000000000004 instead of .3 . I don't understand this. But when I added .1+.3 it gives me .4. I googled it and find i...

I'd like to send some double precision floating point numbers over the network. (standard C, standard sockets) There is no htond or ntohd to convert the data to and from network byte order. What sh...

In trying to mix precision in a simple program - using both real and double - and use the ddot routine from BLAS, I'm coming up with incorrect output for the double precision piece. Here's the code...

I noticed that when I store a double value such as e.g. x = 0.56657011973046234 in an sqlite database, and then retrieve it later, I get y = 0.56657011973046201. According to the sqlite spec and th...

I'm working on porting the sqrt function (for 64-bit doubles) from fdlibm to a model-checker tool I'm using at the moment (cbmc). As part of my doings, I read a lot about the ieee-754 standard, but...

Consider the following code snippet float num = 281.583f; int amount = (int) Math.round(num*100f); float rounded = amount/100.0f; double dblPrecision = rounded; double dblPrecision2 = num; System....

I'm writing a function that calculates the value of PI, and returns it as a double. So far so good. But once the function gets to 14 digits after the decimal place, it can't hold any more. I'm assu...

While working on a simple programming exercise, I produced a while loop (DO loop in Fortran) that was meant to exit when a real variable had reached a precise value. I noticed that due to the prec...

Is it even that inaccurate? I re-implented the whole thing with Apfloat arbitrary precision and it made no difference which I should have known to start with!! public static double bearing(LatLng ...

What are the advantages and disadvantages of using one instead of the other in C++?