floating-accuracy Questions
11
A single/double/extended-precision floating-point representation of Pi is accurate up to how many decimal places?
Sipe asked 3/2, 2009 at 16:24
7
Solved
The following code is obviously wrong. What's the problem?
i <- 0.1
i <- i + 0.05
i
## [1] 0.15
if(i==0.15) cat("i equals 0.15") else cat("i does not equal 0.15")
## i does not equal 0.15
Estabrook asked 29/2, 2012 at 23:46
4
Solved
Code:
class Main {
public static void main (String[] args) {
System.out.print("float: ");
System.out.println(1.35f-0.00026f);
System.out.print("double: ");
System.out.println(1.35-0.00026);
...
Sterner asked 18/10, 2013 at 7:34
1
Background
It is well known that the exact product of two floating point numbers is not always a floating point number, but the error exact(a*b) - float(a*b) is. Some codes for exact multiplicatio...
Presber asked 21/2, 2018 at 12:4
34
Solved
Consider the following code:
0.1 + 0.2 == 0.3 -> false
0.1 + 0.2 -> 0.30000000000000004
Why do these inaccuracies happen?
Submediant asked 25/2, 2009 at 21:39
13
Solved
Does anyone know the most efficient representation for lat/long coordinates? Accuracy level should be enough for consumer GPS devices.
Most implementations seem to use double for each unit, but I'...
Relativity asked 3/8, 2009 at 1:28
4
Solved
I have to analyse a large amount of data using Python3 (PyPy implementation), where I do some operations on quite large floats, and must check if the results are close enough to integers.
To exempl...
Fleda asked 12/10, 2023 at 21:58
15
When a float number needs to be truncated to a certain digit after the floating point, it turns out that it is not easy to be done. For example if the truncating has to be done to second digit afte...
Donothing asked 12/1, 2011 at 12:11
3
Solved
In programming, I used only Integers. But this time for some calculations. I need to calculate Euler-Mascheroni Constant γ . up to n-th decimal.{Though n ∈ [30, 150] is enough for me.
[x] = gif(x...
Oneill asked 22/9, 2021 at 12:13
5
Solved
I'd like to allow the user to vary the precision of a String generated from a double. Right now, I'm trying something like
String foo = String.format("%.*f\n", precision, my_double);
How...
Flexed asked 20/2, 2010 at 4:26
4
Solved
This question refers to the IEEE standard floating point numbers used on C/x86.
Is it possible to represent any numeric (i.e. excluding special values such as NaN) float or double as a decimal str...
Nineveh asked 24/1, 2012 at 16:32
7
Solved
I am trying out the fast Exp(x) function that previously was described in this answer to an SO question on improving calculation speed in C#:
public static double Exp(double x)
{
var tmp = (long)...
Barefoot asked 11/5, 2012 at 13:26
3
Solved
I have picked two random double numbers:
double a = 7918.52;
double b = 5000.00;
I would expect to get 2918.52 from a - b.
Well, it gives me a result of 2918.5200000000004, which seems odd.
pri...
Taciturnity asked 16/6, 2019 at 13:45
7
Solved
I encountered negative zero in output from python; it's created for example as follows:
k = 0.0
print(-k)
The output will be -0.0.
However, when I compare the -k to 0.0 for equality, it yields ...
Dacoity asked 3/11, 2010 at 0:57
32
Solved
I am trying to truncate decimal numbers to decimal places. Something like this:
5.467 -> 5.46
985.943 -> 985.94
toFixed(2) does just about the right thing but it rounds off the value. I d...
Fetching asked 6/2, 2011 at 10:42
1
The Lambert W function is the inverse function of f(w) = wew. It is a multi-valued function that has infinitely many branches over the complex numbers, but has only two branches over the real numbe...
Zubkoff asked 18/9, 2022 at 4:29
12
Solved
I know UIKit uses CGFloat because of the resolution independent coordinate system.
But every time I want to check if for example frame.origin.x is 0 it makes me feel sick:
if (theView.frame.origi...
Tails asked 26/4, 2012 at 13:41
1
Solved
I have noticed that using the statistical test fligner.test from the r stats package provides different results with a simple transformation, even though this shouldn't be the case.
Here an example...
Bigod asked 23/8, 2022 at 11:51
11
Solved
bc doesn't like numbers expressed in scientific notation (aka exponential notation).
$ echo "3.1e1*2" | bc -l
(standard_in) 1: parse error
but I need to use it to handle a few records that are e...
Intromit asked 14/10, 2012 at 13:19
1
Solved
Someone reverse-sorted by 1/i instead of the usual -i and it made me wonder: What is the smallest positive integer case where that fails*? I think it must be where two consecutive integers i and i+...
Rewire asked 6/7, 2022 at 14:43
2
As the title says, the functionality I'm after is provided by C++11's math libraries to find the next floating point value towards a particular value.
Aside from pulling the code out of the std li...
Hyperbaton asked 2/5, 2013 at 10:41
8
Solved
I would like to have the closest number below 1.0 as a floating point. By reading wikipedia's article on IEEE-754 I have managed to find out that the binary representation for 1.0 is 3FF00000000000...
Count asked 16/10, 2010 at 0:53
2
Solved
Say x_1, x_2, ..., x_n are n objects and one wants to pick one of them so that the probability of choosing x_i is proportional to some number u_i. Numpy provides a function for that:
x, u = np.arra...
Enravish asked 25/2, 2022 at 7:33
1
Solved
I'm currently working on creating a subtype of AbstractArray in Julia, which allows you to store a vector in addition to an Array itself. You can think of it as the column "names", with e...
Catnip asked 9/11, 2021 at 13:53
4
I have noticed that if A is a NxN matrix and it has the inverse matrix. But what the inv() and pinv() function output is different.
- My environment is Win7x64 SP1, Matlab R2012a, Cygwin Octave 3....
Subcartilaginous asked 17/10, 2013 at 9:41
1 Next >
© 2022 - 2024 — McMap. All rights reserved.