precision Questions
2
I have this code in VBA that needs to be in Python. I'm new to Python, sorry for that.
Sub cdec_test()
Dim val1, val2, result1, result2, x1, x2 As Variant
val = CDec(5930 / 3000) 'val1 : 1.97666666...
1
Solved
Intro:
With Java floats, I noticed that when you add 1.0 to a certain range of tiny negative numbers, it equals 1.0. I decided to investigate this and learned a lot about how floats work in my ques...
8
I am using BigDecimal to get some price values. Requirement is something like this, what ever the value we fetch from database, the displayed valued should have 2 decimal points.
Eg:
fetched va...
Apologue asked 25/1, 2013 at 5:1
8
Solved
So, I know I can get current time in milliseconds using JavaScript. But, is it possible to get the current time in nanoseconds instead?
Extraterritoriality asked 14/5, 2011 at 15:29
6
Solved
I'm trying to implement some machine learning algorithms, but I'm having some difficulties putting the data together.
In the example below, I load a example data-set from UCI, remove lines with mi...
3
I have incoming data and I want to compute the average, 95th and 99th percentile of that data - I am most interested in the last 1000 values. At any time, I'd like to query this object to get any o...
Hungary asked 8/5, 2013 at 22:23
9
Solved
In my C++ program, I need to pull a 64 bit float from an external byte sequence. Is there some way to ensure, at compile-time, that doubles are 64 bits? Is there some other type I should use to sto...
14
Solved
This isn't a question as it is more of a be aware. I updated an application that uses json_encode() to PHP7.1.1 and I was seeing an issue with floats being changed to sometimes extend out 17 digits...
4
Solved
In my question about Analysis of float/double precision in 32 decimal digits, one answer said to take a look at __float128.
I used it and the compiler could find it, but I can not print it, since ...
Complexity asked 14/5, 2014 at 12:25
13
Solved
Is there a reliable way in JavaScript to obtain the number of decimal places of an arbitrary number?
It's important to note that I'm not looking for a rounding function. I am looking for a function that returns the number of decimal places in an arbitrary number's simplified decimal representation...
Vindictive asked 2/3, 2012 at 19:53
11
A single/double/extended-precision floating-point representation of Pi is accurate up to how many decimal places?
Sipe asked 3/2, 2009 at 16:24
9
Solved
I have a floating point number that I want to truncate to 3 places but I don't want to round up.
For example, convert 1.0155555555555555 to 1.015 (not 1.016).
How would I go about doing this in ...
Lyndy asked 12/11, 2011 at 18:43
4
Solved
Currently I have a column that is declared as a NUMBER. I want to change the precision of the column to NUMBER(14,2).
SO, I ran the command
alter table EVAPP_FEES modify AMOUNT NUMBER(14,2)'
...
14
Solved
I have a list of float values and I want to print them with cout with 2 decimal places.
For example:
10.900 should be printed as 10.90
1.000 should be printed as 1.00
122.345 should be printed as ...
Thou asked 6/5, 2011 at 5:12
34
Solved
Consider the following code:
0.1 + 0.2 == 0.3 -> false
0.1 + 0.2 -> 0.30000000000000004
Why do these inaccuracies happen?
Submediant asked 25/2, 2009 at 21:39
24
Solved
I am facing a strange behavior of the round() function:
for i in range(1, 15, 2):
n = i / 2
print(n, "=>", round(n))
This code prints:
0.5 => 0
1.5 => 2
2.5 => 2
3.5 => 4
4.5 =...
Interlanguage asked 8/10, 2015 at 15:11
2
In C# (.Net 7):
float a = 13082987520.00f;
Console.WriteLine(a);
Console.WriteLine($"{a:N2}");
Console.WriteLine($"{a:0.00}");
I get:
1.3082988E+10
13,082,987,520.00
1308299000...
Horseradish asked 17/10, 2023 at 13:53
1
Solved
What is the reason of the difference of the same values when doing average between 2 different JVM's (Java 8 and Java 17)?
Is that because the floating point? Or has something else changed between ...
Lakin asked 26/10, 2023 at 8:47
4
Solved
I have to analyse a large amount of data using Python3 (PyPy implementation), where I do some operations on quite large floats, and must check if the results are close enough to integers.
To exempl...
Fleda asked 12/10, 2023 at 21:58
29
Given a double, I want to round it to a given number of points of precision after the decimal point, similar to PHP's round() function.
The closest thing I can find in the Dart docs is double.toSt...
Disenthrall asked 9/2, 2015 at 21:0
8
Solved
These lines in C#
decimal a = 2m;
decimal b = 2.0m;
decimal c = 2.00000000m;
decimal d = 2.000000000000000000000000000m;
Console.WriteLine(a);
Console.WriteLine(b);
Console.WriteLine(c);
Console....
2
Solved
Note: this question originated from this answer.
How to compute precision for conversion specification %.*f to maintain precision of floating-point value?
Note: Here the "maintain precision&qu...
Donell asked 22/9, 2023 at 10:22
10
I've been trying to find out the reason, but I couldn't.
Can anybody help me?
Look at the following example.
float f = 125.32f;
System.out.println("value of f = " + f);
double d = (double) 125.32...
Dhaulagiri asked 6/7, 2013 at 16:29
4
Solved
I need an acos() function with double precision within a compute shader. Since there is no built-in function of acos() in GLSL with double precision, I tried to implement my own.
At first, I imple...
Mahau asked 10/3, 2015 at 16:33
7
Solved
I've found this issue with t-tests and chi-squared in R but I assume this issue applies generally to other tests. If I do:
a <- 1:10
b <- 100:110
t.test(a,b)
I get: t = -64.6472, df = 18...
Pharmacist asked 7/8, 2011 at 4:16
1 Next >
© 2022 - 2024 — McMap. All rights reserved.