floating-point-precision Questions
3
Solved
My code:
def calc_pi(acc):
pos = False
sum = 4.0
for i in range(2, acc):
if not pos:
sum -= 4.0/(2*i-1)
pos = True
else:
sum += 4.0/(2*i-1)
pos = False
return float(sum)
print(calc_pi(5...
Individual asked 4/7, 2015 at 20:43
3
Solved
I've come across two different precision formulas for floating-point numbers.
⌊(N-1) log10(2)⌋ = 6 decimal digits (Single-precision)
and
N log10(2) ≈ 7.225 decimal digits (Single-prec...
Coltun asked 6/6, 2015 at 23:1
10
Solved
I keep getting mixed answers of whether floating point numbers (i.e. float, double, or long double) have one and only one value of precision, or have a precision value which can vary.
One topic ca...
Disentwine asked 29/5, 2015 at 19:11
4
Solved
From my previous question "Is floating point precision mutable or invariant?" I received a response which said,
C provides DBL_DIG, DBL_DECIMAL_DIG, and their float and long double
counterparts...
Sabbatical asked 2/6, 2015 at 5:10
2
Solved
XCode 6.3.1 Swift 1.2
let value: Int = 220904525
let intmax = Int.max
let float = Float(value) // Here is an error probably
let intFromFloat = Int(float)
let double = Double(value)
println("intmax...
Undersized asked 15/5, 2015 at 13:28
4
I'm facing an issue with Math.floor function of javascript for the below scenario:
1) from the value betwwen 8192 and 10484,
if I type 8192.8 -> The Math.floor converts it into 8192.79
if I...
Freitag asked 14/5, 2015 at 1:35
3
Solved
Is there a existing Haskell function which provide an engineering notation formatting (as String)?
If not, I read that printf can be extended by adding an instance to PrintfArg. Do you believe thi...
Gallnut asked 28/4, 2015 at 16:17
3
Solved
For example,
float a = 1.0;
float b = 1.2;
puts(a == b? "equal": "not equal");
Does compiler deal with it bitwisely or by some other methods?
(I know it's not a good choice to decide the equal...
Formation asked 6/5, 2015 at 3:46
3
Solved
I keep on seeing this nonsense about 53 bits of precision in 64-bit IEEE floating point representation. Would someone please explain to me how in the world a bit that is stuck with a 1 in it contri...
Athletic asked 23/8, 2013 at 18:27
3
Solved
For what values of x does the test (x == 0) return true? Is there some kind of margin or does the test return true if and only if the value of x = 0?
Westernize asked 16/10, 2013 at 7:42
1
Solved
If I run:
>>> import math
>>> print(math.pi)
3.141592653589793
Then pi is printed with 16 digits,
However, according to:
>>> import sys
>>> sys.float_info.d...
Evitaevitable asked 13/2, 2015 at 5:32
1
In Ruby 2.2.0, why does:
BigDecimal.new(34.13985572755337, 9)
equal 34.0 but
BigDecimal.new(34.13985572755338, 9)
equal 34.1398557?
Note that I am running this on a 64 bit machine.
Freudian asked 3/2, 2015 at 9:35
1
I'm using the boost multiprecision library, and more precisely the boost::multiprecision::float128 type. Using ICPC for compiling, I get some errors when trying to to do something like:
double a =...
Menorca asked 30/1, 2015 at 17:43
1
How can I define my own float-point format (type) with specific precision and certain bitness of exponent and significand? For example, 128-bit float-point number with 20-bit exponent and 107-bit s...
Kennethkennett asked 10/1, 2015 at 23:32
3
Solved
I know that float arithmetic is tricky, but I am not sure if it is possible to get a mistake if you have a division followed by the inverse multiplication. Written in code, is it possible, that thi...
Stenotype asked 28/12, 2014 at 20:29
2
Solved
Why does this assertion fail?
import std.conv;
void main()
{
auto y = 0.6, delta=0.1;
auto r = to!int(y/delta);
assert(r == 6);
}
r's value should be 6 and yet it's 5, Why?
Deepsix asked 26/12, 2014 at 7:24
1
Solved
I just learned the Decimal class in Python, and I have some issues in modifying the precision of the decimal numbers. Code:
from decimal import *
def main() :
getcontext().prec = 50
print Decim...
New asked 24/12, 2014 at 13:54
2
Solved
For example...
$aa = 10694994.89;
$bb = 10696193.86;
$ab = $aa - $bb;
// result is:-1198.9699999988 not the -1198,97
But in this exampe:
$cc = 0.89;
$dd = 0.86;
$cd = $cc - $dd;
//Result is: 0....
Chaudfroid asked 18/11, 2014 at 21:52
1
Solved
I'm trying to generate a string that involves an occasional float with trailing zeros. This is a MWE of the text string and my attempt at removing them with {0:g}:
xn, cod = 'r', 'abc'
ccl = [546....
Disremember asked 10/9, 2014 at 21:13
3
Solved
Let's say that we have declared the following variables
float a = 1.2291;
float b = 3.99;
float variables have precision 6, which (if I understand correctly) means that the difference between th...
Canyon asked 6/9, 2014 at 21:4
4
Consider two very simple multiplications below:
double result1;
long double result2;
float var1=3.1;
float var2=6.789;
double var3=87.45;
double var4=234.987;
result1=var1*var2;
result2=var3*var4...
Guthrun asked 14/8, 2014 at 7:35
2
Solved
I was delving into multi-precision arithmetics, and there is a nice fast class of algorithms, described in Jonathan Richard Shewchuk, "Adaptive Precision Floating-Point Arithmetic and Fast Robust G...
Brahman asked 16/7, 2014 at 11:20
4
Solved
How can I find the index of the minimum item in a Python list of floats? If they were integers, I would simply do:
minIndex = myList.index(min(myList))
However, with a list of floats I get...
Hildebrand asked 9/11, 2012 at 1:45
3
Solved
So I was trying to understand JavaScript's behavior when dealing with large numbers. Consider the following (tested in Firefox and Chrome):
console.log(9007199254740993) // 9007199254740992
consol...
Richey asked 24/6, 2014 at 23:35
2
I've found this problem in many interview exams, but don't see how to work out the proper solution myself. The problem is:
How many digits of accuracy can be represented by a floating point numb...
Jubal asked 24/6, 2014 at 1:33
© 2022 - 2024 — McMap. All rights reserved.