Re-understand unicode and utf8 encoding

Re-recognize unicode and utf8 encoding

---

Until today, just now to be precise, I didn’t know that UTF-8 encoding and Unicode encoding are different, there is a difference囧
There is a certain connection between them Yes, look at their differences:

<code>UTF-8的长度是不一定的，有可能是1、2、3字节
Unicode长度一定，2个字节（USC-2）
UTF-8可以和Unicode互相转换</code>

The relationship between unicode and utf8

---

Unicode (16)	UTF-8 (binary)
0000 - 007F	0xxxxxxx
0080 - 07FF	110xxxxx 10xxxxxx
0800 - FFFF	1110xxxx 10xxxxxx 10xxxxxx

The above table has two meanings. The first one is obviously Unicode Correspondence to UTF-8 character range , there is another way to see how Unicode and UTF-8 can be converted to each other:

Let’s talk about the conversion from UTF-8 to Unicode first

UTF-8 encoded binary is matched with the above three formats, and the fixed bits are removed after matching (Non-x position in the table), and then every 8-bit group from right to left. If there are not enough 8-bits, the left side will not be collected. Make up 2 bytes and 16 bits. These 16 bits represent the Unicode encoding corresponding to UTF-8. , take a look at the following examples:

The text encoding format in the above picture is UTF-8, and you can use WinHex to see its hexadecimal representation

<code>字符	=> UTF-8	  => UTF-8二进制=> 去掉固定位置凑够16位的二进制 => 16进制

汉 	=> E6B189 => 11100110 10110001 10001001	=> 01101100 01001001 => 6C49
汉 	=> E5AD97 => 11100101 10101101 10010111	=> 01011011	01010111 => 5B57

#下面是在chrome命令行下面运行的结果
'\u6C49'
"汉"
'\u5B57'
"字"

#到这里的话，从UTF-8转换到Unicode已经是一件非常容易的事了，看看转换的伪代码
读取一个字节，11100110
判断该UTF-8字符的格式，属于第三种，3个字节
继续读取2个字节得到 11100101 10101101 10010111
按照格式去掉固定位     1011011 01010111
不够16位，左边补零    01011011 01010111  => 5B57</code>

Let’s look at the conversion from Unicode to UTF-8

<code>5B57
获取5B57所在的Unicode范围，0800 <= 5B57 <= FFFF，得知5B57的UTF-8有三个字节，形式为1110xxxx 10xxxxxx 10xxxxxx
获取5B57的二进制编码 101101101010111
用上一步骤的二进制编码从右至左拼接UTF-8编码 11100101 10101101 10010111 </code>

Let’s talk about the problem

---

Let’s talk about the cause of today’s problem. Many words are input from the front end. Each word in UTF-8 format has a maximum of 30 bytes, so verification will be done on the front end and backend respectively. JavaScript uses Unicode encoding. The back-end program uses UTF-8 encoding, and the current solution is as follows

Front-end

<code>function utf8_bytes(str)
{
	var len = 0, unicode;
	for(var i = 0; i < str.length; i++)
	{
		unicode = str.charCodeAt(i);
		if(unicode < 0x0080) {
			++len;
		} else if(unicode < 0x0800) {
			len += 2;
		} else if(unicode <= 0xFFFF) {
			len += 3;
		}else {
			throw "characters must be USC-2!!"
		}
	}
	return len;
}

#例子
utf8_bytes('asdasdas')
8
utf8_bytes('yrt燕睿涛')
12</code>

Backend

<code>#对于GBK字符串
$len = ceil(strlen(bin2hex(iconv('GBK', 'UTF-8', $word)))/2);
#对于UTF8字符串
$len = ceil(strlen(bin2hex($word))/2);</code>

5/21/2015 8:21:53 PM

The copyright of this article belongs to the author iforever(luluyrt＠163 .com), any form of reprinting is prohibited without the consent of the author. After reprinting the article, a link to the author and the original text must be provided in an obvious position on the article page, otherwise we reserve the right to pursue legal liability.

The above has introduced the re-understanding of unicode and utf8 encoding, including aspects of it. I hope it will be helpful to friends who are interested in PHP tutorials.