Detailed introduction to some things related to codepoint and UTF-16 in Java

The relationship between Unicode and UTF-8/UTF-16/UTF-32

The relationship between Unicode and UTF-8/UTF-16/UTF-32 is Character set and encoding Relationship. The concept of character set actually includes two aspects, one is the set of characters and the other is the encoding scheme. A character set defines all the symbols it contains. A character set in a narrow sense does not include an encoding scheme. It just defines all the symbols that belong to this character set. But generally speaking, a character set doesn't just define a collection of characters, it also defines a binary encoding for each symbol. When we mention GB2312 or ASCII, it implicitly indicates that the encoding scheme is GB2312 or ASCII. In these cases, the character set and encoding scheme can be considered equal.

But Unicode has multiple encoding schemes. The standard encoding scheme specified by the Unicode character set is UCS-2 (UTF-16), which uses two bytes to represent a Unicode character (two bytes in UTF-16 are basic multilingual flat characters, 4 bytes are auxiliary plane characters). UCS-4 (UTF-32) uses 4 bytes to represent a Unicode character. Another commonly used Unicode encoding scheme - UTF-8 uses 1 to 4 variable-length bytes to represent a Unicode character, and can be obtained directly from UTF-16 from a simple conversion algorithm. Therefore, there are multiple encoding schemes when using the Unicode character set, which are used in appropriate scenarios.

To put it more simply, the Unicode character set is equivalent to a dictionary, which records all characters (i.e. images) and their corresponding Unicode codes (regardless of the specific encoding scheme), UTF-8/ UTF-16/UTF-32 code is the data calculated by Unicode code through corresponding formulas and actually stored and transmitted.

UTF-16

The JVM specification clearly states that the encoding scheme used by java's char type is UTF-16, so let's first understand UTF-16.

Unicode's encoding space ranges from U+0000 to U+10FFFF. There are 1112064 code points (code points) that can be used to map characters. The code point is the digital form of the character. This part of the coding space can be divided into 17 planes, each plane containing 2^16 (65536) code bits. The first plane is called the Basic Multilingual Plane (BMP), or Plane 0. Other planes are called Supplementary Planes. In the basic multilingual plane, the code point blocks from U+D800 to U+DFFF are permanently reserved and are not mapped to Unicode characters. UTF-16 uses the reserved code bits of the 0xD800-0xDFFF section to encode the code bits of the auxiliary plane characters.

The most commonly used characters are included in BMP and are represented by 2 bytes. The code bits in the auxiliary plane are encoded into a pair of 16-bit long code elements in UTF-16, called a surrogate pair. The specific method is:

• Subtract 0×10000 from the code bit, and the resulting value ranges from 0 to 0xFFFFF, which is 20 bits long.

• The high 10-bit value (the value range is 0~0x3FF) is added to 0xD800 to obtain the first code element or called the high surrogate, the value range It is 0xD800~0xDBFF. Since the value of the high-order proxy is smaller than that of the low-order proxy, in order to avoid confusion, the Unicode standard now calls the high-order proxy lead surrogates.

• The low 10-bit value (the value range is also 0~0x3FF) is added with 0xDC00 to get the second symbol or called the low surrogate. The current value is The range is 0xDC00~0xDFFF. Since the low-order surrogate has a larger value than the high-order surrogate, in order to avoid confusion, the Unicode standard now calls the low-order surrogate trail surrogates.

For example, U+10437 encoding:

• 0×10437 minus 0×10000, the result is 0×00437, which in binary is 0000 0000 0100 0011 0111.

• Partition its upper 10-bit value and lower 10-bit value (using binary): 0000000001 and 0000110111.

• Add 0xD800 to the upper value to form the high bit: 0xD800 + 0×0001 = 0xD801.

• Add 0xDC00 to the lower value to form the low bit: 0xDC00 + 0×0037 = 0xDC37.

Since the code points of the leading agent, trailing agent, and valid characters in BMP do not overlap with each other, it is impossible for a part of one character encoding to be the same as another character encoding during search. Different parts overlap. So you can determine the starting code element of the next character for a given character by examining only one code element (the basic unit that makes up a code point, 2 bytes).

Codepoint related in java

For a string object, its content is stored through a char array. The char type is stored in 2 bytes, and these 2 bytes actually store the code elements under UTF-16 encoding. When we use the charAt and length methods, what is actually returned is a code element and the number of code elements. Although there is generally no problem, if the character is an auxiliary plane character, the above two methods will not get the correct result. The correct processing method is as follows:

int character = aString.codePointAt(i);
int length = aString.codePointCount(0, aString.length())；

It should be noted that the return value of codePointAt is int instead of char. This value is the Unicode code.

The codePointAt method calls codePointAtImpl:

static int codePointAtImpl(char[] a, int index, int limit) {
        char c1 = a[index];
        if (isHighSurrogate(c1) && ++index < limit) {
            char c2 = a[index];
            if (isLowSurrogate(c2)) {
                return toCodePoint(c1, c2);
            }
        }
        return c1;
    }

isHighSurrogate method determines whether the 2 bytes of the subscript character are leading surrogates in UTF-16 (0xD800~0xDBFF):

public static boolean isHighSurrogate(char ch) {
        // Help VM constant-fold; MAX_HIGH_SURROGATE + 1 == MIN_LOW_SURROGATE
        return ch >= MIN_HIGH_SURROGATE && ch < (MAX_HIGH_SURROGATE + 1);
    }

public static final char MIN_HIGH_SURROGATE = '\uD800';
public static final char MAX_HIGH_SURROGATE = '\uDBFF';

然后++index,isLowSurrogate方法判断下一个字符的2个字节是否为后尾代理（0xDC00~0xDFFF）：

public static boolean isLowSurrogate(char ch) {
        return ch >= MIN_LOW_SURROGATE && ch < (MAX_LOW_SURROGATE + 1);
    }

public static final char MIN_LOW_SURROGATE  = '\uDC00';
public static final char MAX_LOW_SURROGATE  = '\uDFFF';

toCodePoint方法将这2个码元组装成一个Unicode码：

public static int toCodePoint(char high, char low) {
        // Optimized form of:
        // return ((high - MIN_HIGH_SURROGATE) << 10)
        //         + (low - MIN_LOW_SURROGATE)
        //         + MIN_SUPPLEMENTARY_CODE_POINT;
        return ((high << 10) + low) + (MIN_SUPPLEMENTARY_CODE_POINT
                                       - (MIN_HIGH_SURROGATE << 10)
                                       - MIN_LOW_SURROGATE);
    }

这个过程就是以上将一个辅助平面的Unicode码位转换成2个码元的逆过程。

所以，枚举字符串的正确方法：

for (int i = 0; i < aString.length();) {
	int character = aString.codePointAt(i);
	//如果是辅助平面字符，则i+2
	if (Character.isSupplementaryCodePoint(character)) i += 2;
	else ++i;
}

将codePoint转换为char[]可调用Character.toChars方法，然后可进一步转换为字符串：

new String(Character.toChars(codePoint));

toChars方法所做的就是以上将Unicode码位转换为2个码元的过程。

The above is the detailed content of Detailed introduction to some things related to codepoint and UTF-16 in Java. For more information, please follow other related articles on the PHP Chinese website!