floating-point-precision Questions
4
Solved
In my question about Analysis of float/double precision in 32 decimal digits, one answer said to take a look at __float128.
I used it and the compiler could find it, but I can not print it, since ...
Complexity asked 14/5, 2014 at 12:25
4
Solved
Code:
class Main {
public static void main (String[] args) {
System.out.print("float: ");
System.out.println(1.35f-0.00026f);
System.out.print("double: ");
System.out.println(1.35-0.00026);
...
Sterner asked 18/10, 2013 at 7:34
4
Solved
I am a new bash learner. I want to print the result of an expression given as input having 3 digits after decimal point with rounding if needed.
I can use the following code, but it does not round....
Infract asked 29/6, 2015 at 20:22
4
I have the following repetitive simple code repeated several times that I would like to make a function for:
for i in range(10):
id = "some id string looked up in dict"
val = 63.4568900932840928...
Sennight asked 6/4, 2011 at 22:28
9
Solved
Is there a printf width specifier which can be applied to a floating point specifier that would automatically format the output to the necessary number of significant digits such that when scanning...
Eternize asked 30/5, 2013 at 15:4
4
Solved
My output is like this -
ruby-1.9.2-p290 :011 > 2.32 * 3
=> 6.959999999999999
And I remember sometime back on another machine I had got it like.. 2.32 * 3 = 6
What is my mistake?
Thanks a...
Expropriate asked 20/10, 2011 at 20:53
6
Solved
I have an XSD that requires me to use a BigDecimal for a lat/lon. I currently have the lat/lon as doubles, and convert them to BigDecimal, but I am only required to use about 12 places of precision...
Surfing asked 28/2, 2012 at 13:35
3
I am confused about what max_digits10 represents. According to its documentation, it is 0 for all integral types. The formula for floating-point types for max_digits10 looks similar to int's digits...
Arne asked 17/3, 2014 at 15:24
4
Solved
I'm creating a numpy array of random values and adding them to an existing array containing 32-bit floats. I'd like to generate the random values using the same dtype as the target array, so that I...
Constitution asked 29/4, 2014 at 4:23
3
Solved
As part of a unit test, I need to test some boundary conditions. One method accepts a System.Double argument.
Is there a way to get the next-smallest double value? (i.e. decrement the mantissa by ...
Telegraphese asked 11/3, 2013 at 3:18
8
Solved
I've got a python Decimal (a currency amount) which I want to round to two decimal places. I tried doing this using the regular round() function. Unfortunately, this returns a float, which makes it...
Felon asked 10/5, 2014 at 16:3
4
Solved
Is it the case that:
Representable floating point values are densest in the real number line near zero?
Representable floating point values grow sparser (exponentially?) as the number line moves ...
Chaste asked 21/8, 2012 at 1:16
4
Solved
In C++ (and C), a floating point literal without suffix defaults to double, while the suffix f implies a float. But what is the suffix to get a long double?
Without knowing, I would define, say,
...
Clamshell asked 4/2, 2014 at 16:23
3
Solved
I have read several topics about the display of floating point numbers display in C++ and I couldn't find a satisfying answer.
My question is: how to display all the significant digits of a...
Intend asked 26/10, 2013 at 18:12
2
Solved
I was trying to write a function to approximate square roots (I know there's the math module...I want to do it myself), and I was getting screwed over by the floating point arithmetic. How ca...
Legitimize asked 20/10, 2013 at 3:56
4
My problem is that I have to use a thrid-party function/algorithm which takes an array of double-precision values as input, but apparently can be sensitive to very small changes in the input data. ...
Bryson asked 4/1, 2013 at 2:7
6
Sample code:
int main() {
float f = 123.542;
int i = (int)f;
printf("%d\n",i);
}
Carpospore asked 10/3, 2012 at 6:3
2
Solved
I'm using float 24 bit to store a floating point value in a compiler MRK III from NXP. It stores the 24 bit float value as 3 byte Hex in Data memory.
Now when I'm using IEEE 754 float point conver...
Homogeny asked 14/11, 2014 at 12:32
3
Solved
What is the difference between machine epsilon and least positive number in floating point representation?
If I try to show the floating point number on a number line .Is the gap between exact 0 a...
Melanesian asked 10/10, 2014 at 15:31
1
Solved
I'm trying to display a number as a percent by using _.round and then by multiplying the number by 100. For some reason, when I multiply the rounded number, the precision gets messed up. Here's wha...
Plunge asked 1/9, 2015 at 15:55
3
In making a somewhat large refactoring change that did not modify any kind of arithmetic, I managed to somehow change the output of my program (an agent based simulation system). Various numbers in...
Holloman asked 30/7, 2014 at 21:27
3
Solved
I often need to floor or ceil a CGFloat to an int, for calculation of an array index.
The problem I permanently see with floorf(theCGFloat) or ceilf(theCGFloat) is that there can be troubles with ...
Durga asked 16/8, 2012 at 19:16
1
In R software
a <- 123456789123456789123456789
sprintf("%27f",a)
#[1] "123456789123456791337762816.000000"
I got the wrong answer. I want exact a value.
Why is the system showing the wrong v...
Suchlike asked 3/9, 2015 at 6:37
1
I am working on some c++ problem which should output 22258199.5000000 this number, I have stored the result in double datatype and when I tried to print that variable using std::cout it rounds off ...
Seagraves asked 26/8, 2015 at 16:19
3
I have a piece of code which computes the Helmholtz-Hodge Decomposition.
I've been running on my Mac OS Yosemite and it was working just fine. A month ago, however, my Mac got pretty slow (it was r...
Dafodil asked 13/5, 2015 at 0:44
1 Next >
© 2022 - 2024 — McMap. All rights reserved.