floating-accuracy Questions
1
Solved
In this example, the column-wise sum of an array pr is computed in two different ways:
(a) take the sum over the first axis using p.sum's axis parameter
(b) slice the array along the the second...
Senhorita asked 2/4, 2019 at 12:0
1
Solved
Here's the code:
#include <iostream>
#include <math.h>
const double ln2per12 = log(2.0) / 12.0;
int main() {
std::cout.precision(100);
double target = 9.800000000000000710542735760...
Peadar asked 14/2, 2019 at 15:50
7
Solved
My Java program is centered around high precision calculations, which need to be accurate to at least 120 decimal places.
Consequentially, all non-integer numbers will be represented by BigDecimals...
Hyperon asked 8/4, 2012 at 3:29
3
Solved
My apologies if this has been asked before, but I cannot find it.
I was wondering if there is a way to calculate the point at which a single precision floating point number that is used as a count...
Chamfron asked 7/11, 2018 at 16:6
2
I have a MEMS IMU on which I've been collecting data and I'm using pandas to get some statistical data from it. There are 6 32-bit floats collected each cycle. Data rates are fixed for a given coll...
Divert asked 29/10, 2018 at 9:19
5
Solved
This is an interesting question that I was trying to work through the other day. Is it possible to force the significand or exponent of one float to be the same as another float in Python?
The que...
Monophagous asked 28/1, 2016 at 8:37
5
Solved
Just for fun and because it was really easy, I've written a short program to generate Grafting numbers, but because of floating point precision issues it's not finding some of the larger examples.
...
Furring asked 17/7, 2012 at 12:54
4
Solved
I am being paranoid that one of these functions may give an incorrect result like this:
std::floor(2000.0 / 1000.0) --> std::floor(1.999999999999) --> 1
or
std::ceil(18 / 3) --> std::ceil...
Diageotropism asked 25/1, 2016 at 16:4
3
In constexpr: Introduction, the speaker mentioned "Compile-time floating point calculations might not have the same results as runtime calculations":
And the reason is related to "c...
Incursion asked 21/6, 2018 at 1:10
1
Solved
How to deal with floating point arithmetic in Rust?
For example:
fn main() {
let vector = vec![1.01_f64, 1.02, 1.03, 1.01, 1.05];
let difference: Vec<f64> = vector.windows(2).map(|...
Gosse asked 16/5, 2018 at 1:16
1
Solved
I have a numpy array arr. It's a numpy.ndarray, size is (5553110,), dtype=float32.
When I do:
(arr > np.pi )[3154950]
False
(arr[3154950] > np.pi )
True
Why is the first comparison gettin...
Doubleton asked 26/4, 2018 at 15:46
12
Solved
I have aproblem with long sums of floats or doubles in Java.
If I execute:
for ( float value = 0.0f; value < 1.0f; value += 0.1f )
System.out.println( value );
I get:
0.0
0.1
0.2
0.3
0.4...
Dripps asked 10/3, 2011 at 8:35
1
Solved
given two floating point number x and y, suppose all floating point arithmetic conforming the IEEE754 standard, and a certain implementation of square root function sqrt(),
if x < y, is it tru...
Horvitz asked 14/2, 2018 at 8:8
2
We know that floating point is broken, because decimal numbers can't always be perfectly represented in binary. They're rounded to a number that can be represented in binary; sometimes that number ...
Hyalite asked 22/1, 2018 at 4:5
1
Solved
I have a number that I have to deal with that I hate (and I am sure there are others).
It is
a17=0.0249999999999999
a18=0.02499999999999999
Case 1:
round(a17,2) gives 0.02
round(a18,2) gives ...
Corroboration asked 4/1, 2018 at 12:34
1
This is concerning Java. From what I've understood, 0.1 cannot be perfectly represented by Java because of binary representations. That makes
0.1 + 0.1 + 0.1 == 0.3
false. However, why does
0...
Minimal asked 25/11, 2017 at 10:46
2
If we use algorithms with double and float arithmetic, how can we guarantee that the results are the same running it in Python and C, in x86 and x64 Linux and Windows computers and ARM microcontrol...
Aeolipile asked 17/10, 2017 at 17:44
1
Solved
Is there a "best practice" for less-than-equal comparisons with floating point number after a series of floating-point arithmetic operations?
I have the following example in R (although the questi...
Trawl asked 14/10, 2017 at 0:20
3
Python (and almost anything else) has known limitations while working with floating point numbers (nice overview provided here).
While problem is described well in the documentation it avoids prov...
Rubbing asked 12/10, 2017 at 0:43
2
Solved
I have a set of floating point calculation based on numbers I receive via a json packet. At the end of my calculation I require one of the numbers to be >= -0.5. I'm finding that sometimes I have a...
Mound asked 8/8, 2017 at 12:56
2
Solved
I'm using float 24 bit to store a floating point value in a compiler MRK III from NXP. It stores the 24 bit float value as 3 byte Hex in Data memory.
Now when I'm using IEEE 754 float point conver...
Homogeny asked 14/11, 2014 at 12:32
1
Solved
Can anyone explain why these two variable of the same value can output different values when i use setprecision()?
#include <iostream>
#include <iomanip>
int main()
{
float a=98.765;
...
Moselle asked 4/8, 2017 at 4:8
1
Solved
By banker's rounding I mean
"round to nearest, ties to even"
as recommended by IEEE 754:
Rounds to the nearest value; if the number falls midway it is rounded to the nearest value with...
Taynatayra asked 20/7, 2017 at 19:35
21
According to this java.sun page == is the equality comparison operator for floating point numbers in Java.
However, when I type this code:
if(sectionID == currentSectionID)
into my editor and r...
Phocine asked 6/7, 2009 at 17:28
1
Solved
For IEEE-754 arithmetic, is there a guarantee of 0 or 1 units in the last place accuracy for reciprocals? From that, is there a guaranteed error-bound on the reciprocal of a reciprocal?
Nieves asked 19/6, 2017 at 6:11
© 2022 - 2024 — McMap. All rights reserved.