Home Web Front-end JS Tutorial Detailed explanation of JavaScript methods to avoid precision errors in numerical calculations_javascript skills

Detailed explanation of JavaScript methods to avoid precision errors in numerical calculations_javascript skills

May 16, 2016 pm 04:57 PM
javascript Accuracy

If I ask you what is 0.1 0.2? You might give me a blank look, 0.1 0.2 = 0.3 Ah, do you still need to ask? Even the children in kindergarten can answer such a pediatric question. But you know, the same problem in a programming language may not be as simple as imagined.
Don’t believe it? Let’s look at a piece of JS first.

var numA = 0.1;
var numB = 0.2;
alert( (numA numB) === 0.3 );

The execution result is false. Yes, when I saw this code for the first time, I took it for granted that it was true, but the execution result surprised me. Is my opening method wrong? No, no. Let's try executing the following code again and we will know why the result is false.

var numA = 0.1;
var numB = 0.2;
alert( numA numB );

It turns out, 0.1 0.2 = 0.30000000000000004. Isn’t it weird? In fact, for the four arithmetic operations of floating point numbers, almost all programming languages ​​will have similar precision error problems, but in languages ​​such as C/C#/Java, methods have been encapsulated to avoid precision problems, and JavaScript is a weak type. The language does not have a strict data type for floating point numbers from the design concept, so the problem of precision error is particularly prominent. Let’s analyze why there is this accuracy error and how to fix it.

First of all, we have to think about the seemingly pediatric problem of 0.1 0.2 from the perspective of a computer. We know that what can be read by computers is binary, not decimal, so let’s first convert 0.1 and 0.2 into binary and take a look:

0.1 => 0.0001 1001 1001 1001… (infinite loop)
0.2 => 0.0011 0011 0011 0011… (infinite loop)

The decimal part of a double-precision floating point number supports up to 52 digits, so after adding the two, we get a string of 0.0100110011001100110011001100110011001100110011001100. A binary number that is truncated due to the limitation of decimal places in floating point numbers. At this time, we convert it to decimal, That becomes 0.30000000000000004.

So that’s the case, then how to solve this problem? The result I want is 0.1 0.2 === 0.3 Ah! ! !

The simplest solution is to give clear precision requirements. During the process of returning the value, the computer will automatically round, such as:

var numA = 0.1;
var numB = 0.2;
alert( parseFloat((numA numB).toFixed(2)) === 0.3 );

But obviously this is not a once and for all method. It would be great if there was a method that could help us solve the precision problem of these floating point numbers. Let’s try this method:

Math.formatFloat = function(f, digit) {
var m = Math.pow(10, digit);
return parseInt(f * m, 10) / m;
}

var numA = 0.1;
var numB = 0.2;

alert(Math.formatFloat(numA numB, 1) === 0.3);

What does this method mean? In order to avoid precision differences, we need to multiply the number to be calculated by 10 to the nth power, convert it into an integer that the computer can accurately recognize, and then divide it by 10 to the nth power. This is how most programming languages ​​handle precision differences. , we will use it to deal with the precision error of floating point numbers in JS.

If someone asks you next time what is 0.1 0.2, you should be careful in your answer! !

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot Article Tags

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to implement an online speech recognition system using WebSocket and JavaScript How to implement an online speech recognition system using WebSocket and JavaScript Dec 17, 2023 pm 02:54 PM

How to implement an online speech recognition system using WebSocket and JavaScript

WebSocket and JavaScript: key technologies for implementing real-time monitoring systems WebSocket and JavaScript: key technologies for implementing real-time monitoring systems Dec 17, 2023 pm 05:30 PM

WebSocket and JavaScript: key technologies for implementing real-time monitoring systems

How to implement an online reservation system using WebSocket and JavaScript How to implement an online reservation system using WebSocket and JavaScript Dec 17, 2023 am 09:39 AM

How to implement an online reservation system using WebSocket and JavaScript

Causes and avoidance strategies of PHP floating point calculation errors Causes and avoidance strategies of PHP floating point calculation errors Feb 27, 2024 pm 06:33 PM

Causes and avoidance strategies of PHP floating point calculation errors

How to use JavaScript and WebSocket to implement a real-time online ordering system How to use JavaScript and WebSocket to implement a real-time online ordering system Dec 17, 2023 pm 12:09 PM

How to use JavaScript and WebSocket to implement a real-time online ordering system

JavaScript and WebSocket: Building an efficient real-time weather forecasting system JavaScript and WebSocket: Building an efficient real-time weather forecasting system Dec 17, 2023 pm 05:13 PM

JavaScript and WebSocket: Building an efficient real-time weather forecasting system

Simple JavaScript Tutorial: How to Get HTTP Status Code Simple JavaScript Tutorial: How to Get HTTP Status Code Jan 05, 2024 pm 06:08 PM

Simple JavaScript Tutorial: How to Get HTTP Status Code

How to use insertBefore in javascript How to use insertBefore in javascript Nov 24, 2023 am 11:56 AM

How to use insertBefore in javascript

See all articles