How Does Go Handle and Detect Unconvertible Bytes in Strings?

Detection of Unconvertible Bytes in Go Strings

In Go, certain byte sequences cannot be interpreted as valid Unicode characters. Detecting these invalid sequences is crucial for seamless string handling. Here's a detailed explanation:

UTF-8 Validity Check:

As mentioned by Tim Cooper, utf8.Valid can be used to ascertain UTF-8 validity. However, it's important to note that Go strings can contain non-UTF-8 characters. This is because a string is essentially a slice of bytes, and these bytes may not always conform to UTF-8 encoding.

Decoding Behavior:

Go only performs UTF-8 decoding in two specific instances:

• Iterating over a string using for i, r := range s, where r represents a Unicode code point (of type rune)
• Converting a string to a slice of runes using []rune(s)

In both these cases, invalid UTF-8 characters are replaced with the Unicode code point U FFFD, which serves as a placeholder for unsupported glyphs.

Exceptions to Crashing:

Note that these conversions never crash. Therefore, explicitly checking for UTF-8 validity is necessary only if your application requires it, such as when handling input that cannot accept U FFFD as a substitute.

Example:

The following code demonstrates how Go handles invalid UTF-8 bytes:

package main

import "fmt"

func main() {
    a := []byte{0xff}
    s := string(a)
    fmt.Println(s)
    for _, r := range s {
        fmt.Println(r)
    }
    rs := []rune(s)
    fmt.Println(rs)
}

Output:

�
65533
[65533]

As you can see, the invalid byte sequence is displayed as � when the string is printed as a whole. When iterated over, it returns the Unicode code point for U FFFD. And when converted to a slice of runes, it returns a single rune representing U FFFD.

Therefore, while Go does not crash when converting invalid UTF-8 bytes to strings, it is important to be aware of the specific behavior and handle it accordingly in your applications.

The above is the detailed content of How Does Go Handle and Detect Unconvertible Bytes in Strings?. For more information, please follow other related articles on the PHP Chinese website!