Can you tell me the capacity of string to byte slice in Go?

The previous article talked about slicing. The magical problem I encountered today is still related to slicing. What is the specific magic method? Let’s take a look at the following phenomena

Phenomenon 1

a := "abc"
bs := []byte(a)
fmt.Println(bs, len(bs), cap(bs))
// 输出： [97 98 99] 3 8

Phenomenon Two

a := "abc"
bs := []byte(a)
fmt.Println(len(bs), cap(bs))
// 输出: 3 32

Phenomenon Three

bs := []byte("abc")
fmt.Println(len(bs), cap(bs))
// 输出: 3 3

Phenomenon Four

a := ""
bs := []byte(a)
fmt.Println(bs, len(bs), cap(bs))
// 输出: [] 0 0

Phenomenon Five

a := ""
bs := []byte(a)
fmt.Println(len(bs), cap(bs))
// 输出: 0 32

Analysis

I am full here There are question marks in my mind

字符串变量转切片

一个小小的字符串转切片， 内部究竟发生了什么， 竟然如此的神奇。这种时候只好祭出前一篇文章的套路了， 看看汇编代码(希望之后有机会能够对go的汇编语法进行简单的介绍)有没有什么关键词能够帮助我们

以下为现象一转换的汇编代码关键部分

"".main STEXT size=495 args=0x0 locals=0xd8
0x0000 00000 (test.go:5)    TEXT    "".main(SB), ABIInternal, $216-0
0x0000 00000 (test.go:5)    MOVQ    (TLS), CX
0x0009 00009 (test.go:5)    LEAQ    -88(SP), AX
0x000e 00014 (test.go:5)    CMPQ    AX, 16(CX)
0x0012 00018 (test.go:5)    JLS 485
0x0018 00024 (test.go:5)    SUBQ    $216, SP
0x001f 00031 (test.go:5)    MOVQ    BP, 208(SP)
0x0027 00039 (test.go:5)    LEAQ    208(SP), BP
0x002f 00047 (test.go:5)    FUNCDATA    $0, gclocals·7be4bbacbfdb05fb3044e36c22b41e8b(SB)
0x002f 00047 (test.go:5)    FUNCDATA    $1, gclocals·648d0b72bb9d7f59fbfdbee57a078eee(SB)
0x002f 00047 (test.go:5)    FUNCDATA    $2, gclocals·2dfddcc7190380b1ae77e69d81f0a101(SB)
0x002f 00047 (test.go:5)    FUNCDATA    $3, "".main.stkobj(SB)
0x002f 00047 (test.go:6)    PCDATA  $0, $1
0x002f 00047 (test.go:6)    PCDATA  $1, $0
0x002f 00047 (test.go:6)    LEAQ    go.string."abc"(SB), AX
0x0036 00054 (test.go:6)    MOVQ    AX, "".a+96(SP)
0x003b 00059 (test.go:6)    MOVQ    $3, "".a+104(SP)
0x0044 00068 (test.go:7)    MOVQ    $0, (SP)
0x004c 00076 (test.go:7)    PCDATA  $0, $0
0x004c 00076 (test.go:7)    MOVQ    AX, 8(SP)
0x0051 00081 (test.go:7)    MOVQ    $3, 16(SP)
0x005a 00090 (test.go:7)    CALL    runtime.stringtoslicebyte(SB)
0x005f 00095 (test.go:7)    MOVQ    40(SP), AX
0x0064 00100 (test.go:7)    MOVQ    32(SP), CX
0x0069 00105 (test.go:7)    PCDATA  $0, $2
0x0069 00105 (test.go:7)    MOVQ    24(SP), DX
0x006e 00110 (test.go:7)    PCDATA  $0, $0
0x006e 00110 (test.go:7)    PCDATA  $1, $1
0x006e 00110 (test.go:7)    MOVQ    DX, "".bs+112(SP)
0x0073 00115 (test.go:7)    MOVQ    CX, "".bs+120(SP)
0x0078 00120 (test.go:7)    MOVQ    AX, "".bs+128(SP)

以下为现象二转换的汇编代码关键部分

"".main STEXT size=393 args=0x0 locals=0xe0
0x0000 00000 (test.go:5)    TEXT    "".main(SB), ABIInternal, $224-0
0x0000 00000 (test.go:5)    MOVQ    (TLS), CX
0x0009 00009 (test.go:5)    LEAQ    -96(SP), AX
0x000e 00014 (test.go:5)    CMPQ    AX, 16(CX)
0x0012 00018 (test.go:5)    JLS 383
0x0018 00024 (test.go:5)    SUBQ    $224, SP
0x001f 00031 (test.go:5)    MOVQ    BP, 216(SP)
0x0027 00039 (test.go:5)    LEAQ    216(SP), BP
0x002f 00047 (test.go:5)    FUNCDATA    $0, gclocals·0ce64bbc7cfa5ef04d41c861de81a3d7(SB)
0x002f 00047 (test.go:5)    FUNCDATA    $1, gclocals·00590b99cfcd6d71bbbc6e05cb4f8bf8(SB)
0x002f 00047 (test.go:5)    FUNCDATA    $2, gclocals·8dcadbff7c52509cfe2d26e4d7d24689(SB)
0x002f 00047 (test.go:5)    FUNCDATA    $3, "".main.stkobj(SB)
0x002f 00047 (test.go:6)    PCDATA  $0, $1
0x002f 00047 (test.go:6)    PCDATA  $1, $0
0x002f 00047 (test.go:6)    LEAQ    go.string."abc"(SB), AX
0x0036 00054 (test.go:6)    MOVQ    AX, "".a+120(SP)
0x003b 00059 (test.go:6)    MOVQ    $3, "".a+128(SP)
0x0047 00071 (test.go:7)    PCDATA  $0, $2
0x0047 00071 (test.go:7)    LEAQ    ""..autotmp_5+64(SP), CX
0x004c 00076 (test.go:7)    PCDATA  $0, $1
0x004c 00076 (test.go:7)    MOVQ    CX, (SP)
0x0050 00080 (test.go:7)    PCDATA  $0, $0
0x0050 00080 (test.go:7)    MOVQ    AX, 8(SP)
0x0055 00085 (test.go:7)    MOVQ    $3, 16(SP)
0x005e 00094 (test.go:7)    CALL    runtime.stringtoslicebyte(SB)
0x0063 00099 (test.go:7)    MOVQ    40(SP), AX
0x0068 00104 (test.go:7)    MOVQ    32(SP), CX
0x006d 00109 (test.go:7)    PCDATA  $0, $3
0x006d 00109 (test.go:7)    MOVQ    24(SP), DX
0x0072 00114 (test.go:7)    PCDATA  $0, $0
0x0072 00114 (test.go:7)    PCDATA  $1, $1
0x0072 00114 (test.go:7)    MOVQ    DX, "".bs+136(SP)
0x007a 00122 (test.go:7)    MOVQ    CX, "".bs+144(SP)
0x0082 00130 (test.go:7)    MOVQ    AX, "".bs+152(SP)

在看汇编代码之前， 我们首先来看一看runtime.stringtoslicebyte的函数签名

func stringtoslicebyte(buf *tmpBuf, s string) []byte

到这里只靠关键词已经无法看出更多的信息了,还是需要稍微了解一下汇编的语法,笔者在这里列出一点简单的分析， 之后我们还是可以通过取巧的方法发现更多的东西

// 现象一给runtime.stringtoslicebyte的传参
0x002f 00047 (test.go:6)    LEAQ    go.string."abc"(SB), AX // 将字符串"abc"放入寄存器AX
0x0036 00054 (test.go:6)    MOVQ    AX, "".a+96(SP) // 将AX中的内容存入变量a中
0x003b 00059 (test.go:6)    MOVQ    $3, "".a+104(SP) // 将字符串长度3存入变量a中
0x0044 00068 (test.go:7)    MOVQ    $0, (SP) // 将0 传递个runtime.stringtoslicebyte(SB)的第一个参数(笔者猜测对应go中的nil)
0x004c 00076 (test.go:7)    PCDATA  $0, $0 // 据说和gc有关， 具体还不清楚， 一般情况可以忽略
0x004c 00076 (test.go:7)    MOVQ    AX, 8(SP) // 将AX中的内容传递给runtime.stringtoslicebyte(SB)的第二个参数
0x0051 00081 (test.go:7)    MOVQ    $3, 16(SP) // 将字符串长度传递给runtime.stringtoslicebyte(SB)的第二个参数
0x005a 00090 (test.go:7)    CALL    runtime.stringtoslicebyte(SB) // 调用函数， 此行后面的几行代码是将返回值赋值给变量bs

// 现象二给runtime.stringtoslicebyte的传参
0x002f 00047 (test.go:6)    LEAQ    go.string."abc"(SB), AX // 将字符串"abc"放入寄存器AX
0x0036 00054 (test.go:6)    MOVQ    AX, "".a+120(SP) // 将AX中的内容存入变量a中
0x003b 00059 (test.go:6)    MOVQ    $3, "".a+128(SP) // 将字符串长度3存入变量a中
0x0047 00071 (test.go:7)    PCDATA  $0, $2
0x0047 00071 (test.go:7)    LEAQ    ""..autotmp_5+64(SP), CX // 将内部变量autotmp_5放入寄存器CX
0x004c 00076 (test.go:7)    PCDATA  $0, $1
0x004c 00076 (test.go:7)    MOVQ    CX, (SP) // 将CX中的内容传递给runtime.stringtoslicebyte(SB)的第一个参数
0x0050 00080 (test.go:7)    PCDATA  $0, $0
0x0050 00080 (test.go:7)    MOVQ    AX, 8(SP) // 将AX中的内容传递给runtime.stringtoslicebyte(SB)的第二个参数
0x0055 00085 (test.go:7)    MOVQ    $3, 16(SP) // 将字符串长度传递给runtime.stringtoslicebyte(SB)的第二个参数
0x005e 00094 (test.go:7)    CALL    runtime.stringtoslicebyte(SB)

通过上面汇编代码的分析可以知道，现象一和现象二的区别就是传递给runtime.stringtoslicebyte的第一个参数不同。通过对runtime包中stringtoslicebyte函数分析，第一个参数是否有值和字符串长度会影响代码执行的分支，从而生成不同的切片， 因此容量不一样也是常理之中， 下面我们看源码

func stringtoslicebyte(buf *tmpBuf, s string) []byte {
var b []byte
if buf != nil && len(s) <= len(buf) {
        *buf = tmpBuf{}
        b = buf[:len(s)]
    } else {
        b = rawbyteslice(len(s))
    }
copy(b, s)
return b
}

然而， stringtoslicebyte的第一个参数什么情况下才会有值，什么情况下为nil, 我们仍然不清楚。那怎么办呢， 只好祭出全局搜索大法：

# 在go源码根目录执行下面的命令
grep stringtoslicebyte -r . | grep -v "//"

最终在go的编译器源码cmd/compile/internal/gc/walk.go发现了如下代码块

我们查看mkcall 函数签名可以知道, 从第四个参数开始的所有变量都会作为参数传递给第一个参数对应的函数， 最后生成一个*Node的变量。其中Node结构体解释如下:

// A Node is a single node in the syntax tree.
// Actually the syntax tree is a syntax DAG, because there is only one
// node with Op=ONAME for a given instance of a variable x.
// The same is true for Op=OTYPE and Op=OLITERAL. See Node.mayBeShared.

综合上述信息我们得出的结论是，编译器会对stringtoslicebyte的函数调用生成一个AST(抽象语法树)对应的节点。因此我们也知道传递给stringtoslicebyte函数的第一个变量也就对应于上图中的变量a.

其中a的初始值为nodnil()的返回值，即默认为nil. 但是n.Esc == EscNone时，a会变成一个数组。我们看一下EscNone的解释.

// 此代码位于cmd/compile/internal/gc/esc.go中
const (
// ...
    EscNone           // Does not escape to heap, result, or parameters.
    ...
)

由上可知, EscNone用来判断变量是否逃逸,到这儿了我们就很好办了，接下来我们对现象一和现象二的代码进行逃逸分析.

# 执行变量逃逸分析命令: go run -gcflags '-m -l' test.go
# 现象一逃逸分析如下：
./test.go:7:14: ([]byte)(a) escapes to heap
./test.go:8:13: main ... argument does not escape
./test.go:8:13: bs escapes to heap
./test.go:8:21: len(bs) escapes to heap
./test.go:8:30: cap(bs) escapes to heap
[97 98 99] 3 8
# 现象二逃逸分析如下：
./test.go:7:14: main ([]byte)(a) does not escape
./test.go:8:13: main ... argument does not escape
./test.go:8:17: len(bs) escapes to heap
./test.go:8:26: cap(bs) escapes to heap
3 32

根据上面的信息我们知道在现象一中，bs变量发生了逃逸，现象二中变量未发生逃逸，也就是说stringtoslicebyte函数的第一个参数在变量未发生逃逸时其值不为nil,变量发生逃逸时其值为nil。到这里我们已经搞明白stringtoslicebyte的第一个参数了， 那我们继续分析stringtoslicebyte的内部逻辑

我们在runtime/string.go中看到stringtoslicebyte第一个参数的类型定义如下：

const tmpStringBufSize = 32

type tmpBuf [tmpStringBufSize]byte

综上: 现象二中bs变量未发生变量逃逸, stringtoslicebyte第一个参数不为空且是一个长度为32的byte数组, 因此在现象二中生成了一个容量为32的切片

根据对stringtoslicebyte的源码分析， 我们知道现象一调用了rawbyteslice函数

func rawbyteslice(size int) (b []byte) {
cap := roundupsize(uintptr(size))
    p := mallocgc(cap, nil, false)
if cap != uintptr(size) {
        memclrNoHeapPointers(add(p, uintptr(size)), cap-uintptr(size))
    }

    *(*slice)(unsafe.Pointer(&b)) = slice{p, size, int(cap)}
return
}

由上面的代码知道， 切片的容量通过runtime/msize.go中的roundupsize函数计算得出, 其中_MaxSmallSize和class_to_size均定义在runtime/sizeclasses.go

func roundupsize(size uintptr) uintptr {
if size < _MaxSmallSize {
if size <= smallSizeMax-8 {
return uintptr(class_to_size[size_to_class8[(size+smallSizeDiv-1)/smallSizeDiv]])
        } else {
return uintptr(class_to_size[size_to_class128[(size-smallSizeMax+largeSizeDiv-1)/largeSizeDiv]])
        }
    }
if size+_PageSize < size {
return size
    }
return round(size, _PageSize)
}

由于字符串abc的长度小于_MaxSmallSize(32768)，故切片的长度只能取数组class_to_size中的值， 即0, 8, 16, 32, 48, 64, 80, 96, 112, 128....s

至此, 现象一中切片容量为什么为8也真相大白了。相信到这里很多人已经明白现象四和现象五是怎么回事儿了, 其逻辑分别与现象一和现象二是一致的， 有兴趣的， 可以在自己的电脑上面试一试。

字符串直接转切片

那你说了这么多， 现象三还是不能解释啊。请各位看官莫急， 接下来我们继续分析。

相信各位细心的小伙伴应该早就发现了我们在上面的cmd/compile/internal/gc/walk.go源码图中折叠了部分代码， 现在我们就将这块神秘的代码赤裸裸的展示出来

We analyzed this code and found that the go compiler is divided into three steps when converting strings to byte slices to generate AST.

1. First determine whether the variable is a constant string. If it is a constant string, directly create an array with the same length as the string through types.NewArray

2. The slice variable generated by the constant string must also be subjected to escape analysis and determine whether its size is greater than the maximum length allowed by the function stack to be allocated to the variable, thereby determining whether the node is allocated on the stack or in On the heap

3. Finally, if the string length is greater than 0, copy the string content into the byte slice and then return. Therefore, it is completely clear that the slicing capacity in Phenomenon 3 is 3

Conclusion

The steps for string to byte slicing are as follows

1. Determine whether it is a constant. If it is a constant, convert it into a byte slice of equal capacity and length.

2. If it is a variable, first determine whether the generated slice is A variable escape occurs

• If the escape or the string length is >32, different capacities can be calculated based on the string length

• If there is no escape and the string length is

Extension

Common escape situations

1. ##Function returns local pointer

2. Insufficient stack space escape

3. Dynamic type escape, many function parameters are interface types, such as fmt.Println(a...interface{}), it is difficult to determine during compilation The specific type of the parameter will also escape.

4. The closure reference object escapes

The above is the detailed content of Can you tell me the capacity of string to byte slice in Go?. For more information, please follow other related articles on the PHP Chinese website!