floating-accuracy Questions
1
Solved
float fv = orginal_value; // original_value may be any float value
...
double dv = (double)fv;
...
fv = (float)dv;
SHOULD fv be equal to original_value exactly? Any precision may be lost?
Pe asked 25/4, 2016 at 12:9
3
Solved
I'm a new to python and I found a confusing result when using Python3.5.1 on my mac, I simply ran this command in my terminal
1 // 0.05
However, it printed 19.0 on my screen. From my point of ...
Bifilar asked 9/3, 2016 at 16:47
1
Solved
java.lang.Math docs say for many functions, such as Math.pow:
The computed result must be within 1 ulp of the exact result.
But I haven't been able to find what does this mean precisely. Is it...
Chiron asked 26/2, 2016 at 18:2
6
Solved
In various contexts, for example for the argument reduction for mathematical functions, one needs to compute (a - K) / (a + K), where a is a positive variable argument and K is a constant. In many ...
Heartthrob asked 16/2, 2016 at 4:51
7
Solved
Comparing two floating point number by something like a_float == b_float is looking for trouble since a_float / 3.0 * 3.0 might not be equal to a_float due to round off error.
What one normally do...
Ravi asked 1/7, 2013 at 12:28
3
Solved
I was doing some calculations for planning an improved implementation of my prime sieve when I noticed that the Libre Office spreadsheet was showing the wrong values for numbers far below 2^53, whi...
Lucarne asked 28/12, 2015 at 19:8
3
Solved
I am a bit confused about how rounding in Python works.
Could someone please explain why Python behaves like this?
Example:
>>> round(0.05,1) # this makes sense
0.1
>>> round(0....
Drawstring asked 5/1, 2016 at 20:27
2
Solved
The inverse hyperbolic function asinh() is closely related to the natural logarithm. I am trying to determine the most accurate way to compute asinh() from the C99 standard math function log1p(). F...
Kristine asked 30/12, 2015 at 18:42
1
Referring to the documentation of BigDecimal class,
n,m = a.precs
prec returns number of significant digits (n) and
maximum number of significant digits (m) of a.
I am puzzled by the followi...
Orelia asked 18/12, 2015 at 4:6
3
Solved
I often need to floor or ceil a CGFloat to an int, for calculation of an array index.
The problem I permanently see with floorf(theCGFloat) or ceilf(theCGFloat) is that there can be troubles with ...
Durga asked 16/8, 2012 at 19:16
4
Solved
for example, 0 , 0.5, 0.15625 , 1 , 2 , 3... are values converted from IEEE 754. Are their hardcode version precise?
for example:
is
float a=0;
if(a==0){
return true;
}
always return true? o...
Acetate asked 11/11, 2015 at 7:44
3
I have to check an inequality containing square roots. To avoid incorrect results due to floating point inaccuracy and rounding, I use std::nextafter() to get an upper/lower bound:
#include <cf...
Cornucopia asked 25/8, 2015 at 15:25
4
Highcharts 3.0 seems to have floating point number accuracy issue when displaying tooltips on a pie chart. I was able to recreate the exact error by using one of the highcharts demo pie chart - Pie...
Spencer asked 9/4, 2013 at 16:32
1
Solved
The select returns right at 23,000 rows
The except will return between 60 to 200 rows (and not the same rows)
The except should return 0 as it is select a except select a
PK: [docSVenum1].[enumID...
Lunnete asked 24/10, 2015 at 20:8
3
Solved
I'm subsetting a dataset before plotting, but the key being numeric I cannot use the strict equality testing of match() or %in% (it misses a few values).
I wrote the following alternative, but I i...
Tantara asked 20/10, 2015 at 0:34
1
Solved
Does the standard guarantee that functions return the exact same result across all implementations?
Take for example pow(float,float) for 32bit IEEE floats. Is the result across all implementation...
Mom asked 18/10, 2015 at 18:17
2
Solved
So I'm learning that Lisp does fractions which is great. But why then does this equality check return NIL:
* (= 0.2 1/5)
NIL
...while it returns True if it's converted to float first:
* (= 0...
Titania asked 1/10, 2015 at 14:3
1
Solved
I am seeking some more understanding about the "resolution" parameter of a numpy float (I guess any computer defined float for that matter).
Consider the following script:
import numpy as np
a =...
Kilby asked 8/9, 2015 at 19:11
1
Solved
The documentation of std::hypot says that:
Computes the square root of the sum of the squares of x and y, without undue overflow or underflow at intermediate stages of the computation.
I strug...
Foretoken asked 7/9, 2015 at 9:55
4
Solved
Assuming n and m are positive integers, and nm is within the range of an integer, will (int)pow(n,m) ever give a wrong answer?
I have tried many n for m=2 and have not gotten any wrong answers so ...
Gaulin asked 2/9, 2015 at 12:33
4
Solved
Yesterday during debugging something strange happened to me and I can't really explain it:
So maybe I am not seeing the obvious here or I misunderstood something about decimals in .NET but sho...
Uralite asked 25/8, 2015 at 7:34
5
I'm trying to divide screenwidth by a variable in order to draw equally spaced UI parts in webview.
However, when the variable is 3, 100 / 3 results 33.3333333333333336 in JavaScript, and the thi...
Labored asked 27/7, 2013 at 21:53
6
Solved
the round(numeric,integer) function in PostgreSQL only rounds up:
round(cast (41.0255 as numeric),3) ==> 41.026
Since we need a round function which returns 41.025 and (quite surprisingly) th...
Chorea asked 21/9, 2012 at 8:48
3
Let a and b be two numbers between 0 and 1.
How to calculate pow(a,10000)/(pow(a,10000)+pow(b,10000))?
Ex:-
This following code gives -nan as output instead of 0.5
double a = 0.5,b = 0.5;
cout&l...
Murrhine asked 8/7, 2015 at 20:47
4
Solved
From http://msdn.microsoft.com/en-us/library/system.math.pow.aspx
int value = 2;
for (int power = 0; power <= 32; power++)
Console.WriteLine("{0}^{1} = {2:N0}",
value, power, (long) Math.Pow(...
Encrinite asked 25/6, 2012 at 20:17
© 2022 - 2024 — McMap. All rights reserved.