Someone just asked why sum(myfloats) differed from sum(reversed(myfloats)). Quickly got duped to Is floating point math broken? and deleted. But it made me curious: How many different sums can we g...

While going through this post at SO by the user @skrebbel who stated that the google testing framework does a good and fast job for comparing floats and doubles. So I wrote the following code to ch...

I am working on some programming problems where I have to transition probabilities between standard space and log-space. For this purpose, I am trying to find out the maximum absolute error for the...

I've heard of "error" when using floating point variables. Now I'm trying to solve this puzzle and I think I'm getting some rounding/floating point error. So I'm finally going to figure out the bas...

I am trying to figure this out, the market data returns money values as a string that is 8 places after the digit long. money = "124.19000540" I need this to be 124.19, any idea how to achieve t...

I'm trying to separate the decimal and integer parts of a double in swift. I've tried a number of approaches but they all run into the same issue... let x:Double = 1234.5678 let n1:Double = x % 1....

i had a problem where i was trying to reconstruct the the formula used in an existing system, a fairly simple formula of one input and one output: y = f(x) After a lot of puzzling, we managed to...

Using the IEEE Standard for float, how many different numbers can a 32 bit float represent?

In the C++ standard libraries I found only a floating point log method. Now I use log to find the level of an index in a binary tree ( floor(2log(index)) ). Code (C++): int targetlevel = int(log...

The difference of two products and the sum of two products are two primitives found in a variety of common computations. diff_of_products (a,b,c,d) := ab - cd and sum_of_products(a,b,c,d) := ab + c...

I'm writing a unit test for a class for 3D vector objects and its algebra (dot product, cross product etc.) and just observed a behavior I can somehow understand, but not to its fully extent. What ...

In the following query declare @a float(23) declare @b float(23) declare @c float(53) set @a = 123456789012.1234 set @b = 1234567.12345678 set @c = @a * @b select @c select LTRIM(STR((@c),32,12))...

I'm having an infinite loop when I run this code: intensity = 0.50 while intensity != 0.65: print(intensity) intensity = intensity + 0.05 intensity values should be like 0.50 -> 0.55 -> ...

What's the threshold value of binary_accuracy in keras Metrices is used to predicted one sample as positive and negative cases? is that threshold value 0.5? how to adjust it? I want to set the thre...

On Intel processors, x87 trigonometric instructions such as FSIN have limited accuracy due to the use of a 66-bit approximation of pi even though the computation itself is otherwise accurate to the...

FLT_DIG, DBL_DIG, LDBL_DIG are the number of decimal digits that can be accurately represented by float, double, and long double types respectively. #include <stdio.h> #include <flo...

In Java the floating point arithmetic is not represented precisely. For example this java code: float a = 1.2; float b= 3.0; float c = a * b; if(c == 3.6){ System.out.println("c is 3.6"); } el...

I have just noticed that the following code returns true: Mathf.Approximately(0.0f, float.Epsilon); // true I have read the Mathf.Approximately Documentation and it states that: Approximatel...

My question is not about floating precision. It is about why Equals() is different from ==. I understand why .1f + .2f == .3f is false (while .1m + .2m == .3m is true). I get that == is reference ...

For large quotients, integer division (//) doesn't seem to be necessarily equal to the floor of regular division (math.floor(a/b)). According to Python docs (https://docs.python.org/3/reference/ex...

say I have the following list: my_list = [3.5, 1.6, 2.4, 8.9, 5.6] I want to find the top 3 largest number in its original place, so the result should be: [3.5, 8.9, 5.6] How could I do that?...

TL;DR: The question is about multiplication ACCURACY I have to multiply matrices A (100x8000), B (8000x27) and C (27x1). Since matrices B and C are constant and A is variable, I prefer to calculate...

I have a series I get from an outside source (x). It's all positive, and is mostly zero. x.describe() count 23275.000000 mean 0.015597 std 0.411720 min 0.000000 25% 0.000000 50% 0.000000 75% 0.000...

I'm trying to write a function to handle movement within a game I'm programming. What I have nearly works, but there are a couple situations where it breaks down. I've coded up a minimal demonstra...

I've always assumed, that numpy uses a kind of pairwise-summation, which ensures high precision also for float32 - operations: import numpy as np N=17*10**6 # float32-precision no longer enough to...