Using goroutine to build Zip, slice is out of range, capacity is 4096

php editor Xiaoxin has brought a technical article about using goroutine to build Zip. In this article, we will learn how to use goroutine to perform Zip operations concurrently to improve the efficiency of the program. In addition, we will discuss the issue of slice out-of-range and show how to use a slice with a capacity of 4096 to solve this problem. Whether you are a beginner or an experienced developer, this article will help you better understand and apply the knowledge related to goroutines and slicing. Let’s explore together!

Question content

I'm trying to improve the function that builds the zip by adding a goroutine to handle each file that must be archived. But it ended in panic panic: runtime error: slice out of range [4126:4096]

The target directory contains 190 files (500 mo). I really don't understand what's wrong Thanks in advance for your help

Function:

func buildarchive() error {                                                        
                                                                                   
    var files []string                                                             
    err := filepath.walk("/tmp/dir-to-zip", func(filepath string, info os.fileinfo, err error) error {
        if info.isdir() {                                                          
            return nil                                                             
        }                                                                          
        if err != nil {                                                            
            fmt.println(err)                                                       
            return err                                                             
        }                                                                          
        files = append(files, filepath)                                            
                                                                                   
        return nil                                                                 
    })                                                                             
    if err != nil {                                                                
        return err                                                                 
    }                                                                              
                                                                                   
    bundle, err := os.create("/tmp/archive.zip")                                   
    if err != nil {                                                                
        return err                                                                 
    }                                                                              
    bundlewriter := zip.newwriter(bundle)                                          
                                                                                   
    var wg sync.waitgroup                                                          
    wg.add(len(files))                                                             
                                                                                   
    for _, filepath := range files {                                               
        go func(filepath string) {                                                 
            defer wg.done()                                                        
            relpath := strings.trimprefix(filepath, fmt.sprintf("%v/", filepath.dir("/tmp/dir-to-zip")))
                                                                                   
            bundlefile, err := bundlewriter.create(relpath)                        
            if err != nil {                                                        
                fmt.println(err)                                                   
            }                                                                      
            fsfile, err := os.open(filepath)                                       
            if err != nil {                                                        
                fmt.println(err)                                                   
            }                                                                      
            _, err = io.copy(bundlefile, fsfile)                                   
            if err != nil {                                                        
                fmt.println(err)                                                   
            }                                                                      
                                                                                   
        }(filepath)                                                                
    }                                                                              
    wg.wait()                                                                      
                                                                                   
    err = bundlewriter.close()                                                     
    if err != nil {                                                                
        return err                                                                 
    }                                                                              
    return nil                                                                     
}

The error occurred here:

_, err = io.copy(bundlefile, fsfile)                                   
        if err != nil {                                                        
            fmt.println(err)                                                   
        }

Stack trace:

goroutine 48 [running]:                                                            
bufio.(*Writer).Write(0xc00002a100, {0xc00041a400?, 0x3d?, 0xc00041a400?})         
    /usr/local/go/src/bufio/bufio.go:670 +0x1c8                                    
archive/zip.(*countWriter).Write(0xc00000c138, {0xc00041a400?, 0x3d?, 0x4afa20?})
    /usr/local/go/src/archive/zip/writer.go:601 +0x2e                              
io.WriteString({0x4e7538, 0xc00000c138}, {0xc0000212c9, 0x3d})                     
    /usr/local/go/src/io/io.go:314 +0x91                                           
archive/zip.writeHeader({0x4e7538, 0xc00000c138}, 0xc000220090)                    
    /usr/local/go/src/archive/zip/writer.go:422 +0x5ec                             
archive/zip.(*Writer).CreateHeader(0xc0000760a0, 0xc00021e1b0)                     
    /usr/local/go/src/archive/zip/writer.go:378 +0x797                             
archive/zip.(*Writer).Create(0x4e7698?, {0xc0000212c9, 0x3d})                      
    /usr/local/go/src/archive/zip/writer.go:223 +0x6c                              
main.BuildArchive.func2({0xc0000212c0, 0x46})                                      
    /home/simba/go/src/foobar/main.go:79 +0x1c5                                    
created by main.BuildArchive                                                       
    /home/simba/go/src/foobar/main.go:73 +0x5aa                                    
panic: runtime error: slice bounds out of range [:4126] with capacity 4096         
                                                                                   
goroutine 6 [running]:                                                             
bufio.(*Writer).Flush(0xc00002a100)                                                
    /usr/local/go/src/bufio/bufio.go:634 +0x171                                    
bufio.(*Writer).Write(0xc00002a100, {0xc0001b4200?, 0xc000199b20?, 0xc000199b20?})
    /usr/local/go/src/bufio/bufio.go:672 +0xd8                                     
archive/zip.(*countWriter).Write(0xc00000c138, {0xc0001b4200?, 0x0?, 0xc000199b40?})
    /usr/local/go/src/archive/zip/writer.go:601 +0x2e                              
archive/zip.(*countWriter).Write(0xc000220018, {0xc0001b4200?, 0xc0001b02c0?, 0xc0001b02f0?})
    /usr/local/go/src/archive/zip/writer.go:601 +0x2e                              
compress/flate.(*huffmanBitWriter).write(...)                                      
    /usr/local/go/src/compress/flate/huffman_bit_writer.go:136                     
compress/flate.(*huffmanBitWriter).writeCode(0xc0001b41e0?, {0x6000?, 0x22?})   
    /usr/local/go/src/compress/flate/huffman_bit_writer.go:347 +0xe5               
compress/flate.(*huffmanBitWriter).writeTokens(0xc0001b41e0, {0xc002558000, 0x4001, 0x403f800000403f?}, {0xc0001aa900, 0x11e, 0x108129000000000f?}, {0xc0001ac100, 0x1e, 0x1e})
    /usr/local/go/src/compress/flate/huffman_bit_writer.go:583 +0xb9               
compress/flate.(*huffmanBitWriter).writeBlock(0xc0001b41e0, {0xc002558000?, 0x20?, 0xd79?}, 0x0, {0x0, 0x0, 0x0})
    /usr/local/go/src/compress/flate/huffman_bit_writer.go:495 +0x490              
compress/flate.(*compressor).writeBlock(0xc0005a2000, {0xc002558000?, 0xc000032f00?, 0xc000199d28?}, 0x47739b?)
    /usr/local/go/src/compress/flate/deflate.go:170 +0x9c                          
compress/flate.(*compressor).deflate(0xc0005a2000)                                 
    /usr/local/go/src/compress/flate/deflate.go:509 +0x59b                         
compress/flate.(*compressor).write(0xc0005a2000, {0xc00256a000?, 0x8000, 0xf311b6fd?})
    /usr/local/go/src/compress/flate/deflate.go:554 +0x82                          
compress/flate.(*Writer).Write(...)                                                
    /usr/local/go/src/compress/flate/deflate.go:712                                
archive/zip.(*pooledFlateWriter).Write(0xc00020c040?, {0xc00256a000?, 0x8000?, 0x4af140?})
    /usr/local/go/src/archive/zip/register.go:51 +0xc5                             
archive/zip.(*countWriter).Write(...)                                              
    /usr/local/go/src/archive/zip/writer.go:601                                    
archive/zip.(*fileWriter).Write(0xc000222000, {0xc00256a000, 0x8000, 0x8000})   
    /usr/local/go/src/archive/zip/writer.go:533 +0x97                              
io.copyBuffer({0x4e7558, 0xc000222000}, {0x4e7678, 0xc0001f8008}, {0x0, 0x0, 0x0})
    /usr/local/go/src/io/io.go:428 +0x204                                          
io.Copy(...)                                                                       
    /usr/local/go/src/io/io.go:385                                                 
main.BuildArchive.func2({0xc00001c0c0, 0x35})                                      
    /home/simba/go/src/foobar/main.go:89 +0x385                                    
created by main.BuildArchive                                                       
    /home/simba/go/src/foobar/main.go:73 +0x5aa                                    
exit status 2

Solution

zip.Writer Not safe for concurrent use. You start multiple goroutines, each goroutine creates and writes zip entries (files).

Writer.Create() Record:

Create adds the file to the zip file using the provided name. It returns a Writer into which the file contents should be written.

[...] The file contents must be written to io.Writer before the next call to Create, CreateHeader, or Close.

You cannot create zips at the same time. Each zip entry's io.Writers write to the same underlying file (or in general the same io.Writer), even if your application doesn't panic or crash. The resulting zip archive may also be invalid.

The above is the detailed content of Using goroutine to build Zip, slice is out of range, capacity is 4096. For more information, please follow other related articles on the PHP Chinese website!