Table of Contents
Correct Answer
Home Backend Development Golang Golang uses multipart to upload large files to external API. How to avoid `io.Copy(io.Writer, io.Reader)` problems

Golang uses multipart to upload large files to external API. How to avoid `io.Copy(io.Writer, io.Reader)` problems

Feb 06, 2024 am 09:45 AM
api call

Golang 使用多部分将大文件上传到外部 API。如何避免`io.Copy(io.Writer, io.Reader)`问题

Question content

My goal is to use golang's built-in net/http package to upload a large file to POST https://somehost /media .

HTTP format for Api call

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

POST /media HTTP/1.1

Host: somehost

Content-Length: 434

Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW

 

------WebKitFormBoundary7MA4YWxkTrZu0gW

Content-Disposition: form-data; name="detail"

 

More and more detail

------WebKitFormBoundary7MA4YWxkTrZu0gW

Content-Disposition: form-data; name="file"; filename="some_big_video.mp4"

Content-Type: <Content-Type header here>

 

(data)

------WebKitFormBoundary7MA4YWxkTrZu0gW--

Copy after login

In golang, this is the code.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

package main

 

import (

  "fmt"

  "bytes"

  "mime/multipart"

  "os"

  "path/filepath"

  "io"

  "net/http"

  "io/ioutil"

)

 

func main() {

 

  url := "https://somehost/media"

  method := "POST"

 

  payload := &bytes.Buffer{}

  writer := multipart.NewWriter(payload)

  _ = writer.WriteField("details", "more and more details")

  file, errFile3 := os.Open("/Users/vajahat/Downloads/some_big_video.mp4")

  defer file.Close()

  part3,errFile3 := writer.CreateFormFile("file","some_big_video.mp4")

  _, errFile3 = io.Copy(part3, file)

  if errFile3 != nil {

    fmt.Println(errFile3)

    return

  }

  err := writer.Close()

  if err != nil {

    fmt.Println(err)

    return

  }

  client := &http.Client {}

  req, err := http.NewRequest(method, url, payload)

 

  if err != nil {

    fmt.Println(err)

    return

  }

 

  req.Header.Set("Content-Type", writer.FormDataContentType())

  res, err := client.Do(req)

  if err != nil {

    fmt.Println(err)

    return

  }

  defer res.Body.Close()

 

  body, err := ioutil.ReadAll(res.Body)

  if err != nil {

    fmt.Println(err)

    return

  }

  fmt.Println(string(body))

}

Copy after login

How to avoidio.Copy(io.Writer, io.Reader)Problems

The above code works fine, but on the line _, errFile3 = io.Copy(part3, file). This essentially copies everything in the file into main memory.

How to avoid this situation?

Is there any way I can stream large files to the api via multipart-formdata?

The program will be run on the remote server. May crash if opening a very large file.


Correct Answer


Use io.Pipe and a goroutine to copy the file to the request without loading the entire file in memory.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

pr, pw := io.Pipe()

writer := multipart.NewWriter(pw)

ct := writer.FormDataContentType()

go func() {

    _ = writer.WriteField("details", "more and more details")

    file, err := os.Open("/Users/vajahat/Downloads/some_big_video.mp4")

    if err != nil {

        pw.CloseWithError(err)

        return

    }

    defer file.Close()

    part3, err := writer.CreateFormFile("file", "some_big_video.mp4")

    if err != nil {

        pw.CloseWithError(err)

        return

    }

    _, err = io.Copy(part3, file)

    if err != nil {

        pw.CloseWithError(err)

        return

    }

    pw.CloseWithError(writer.Close())

}()

 

client := &http.Client{}

req, err := http.NewRequest(method, url, pr)

 

if err != nil {

    fmt.Println(err)

    return

}

 

req.Header.Set("Content-Type", ct)

// remaining code as before

Copy after login

The above is the detailed content of Golang uses multipart to upload large files to external API. How to avoid `io.Copy(io.Writer, io.Reader)` problems. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot Article

Hot tools Tags

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to use PHP to call web services and APIs? How to use PHP to call web services and APIs? Jun 30, 2023 pm 03:03 PM

How to use PHP to call web services and APIs?

Summary of FAQs for DeepSeek usage Summary of FAQs for DeepSeek usage Feb 19, 2025 pm 03:45 PM

Summary of FAQs for DeepSeek usage

Exploring the boundaries of agents: AgentQuest, a modular benchmark framework for comprehensively measuring and improving the performance of large language model agents Exploring the boundaries of agents: AgentQuest, a modular benchmark framework for comprehensively measuring and improving the performance of large language model agents Apr 11, 2024 pm 08:52 PM

Exploring the boundaries of agents: AgentQuest, a modular benchmark framework for comprehensively measuring and improving the performance of large language model agents

Can software compiled by Mingw be used in a Linux environment? Can software compiled by Mingw be used in a Linux environment? Mar 20, 2024 pm 05:06 PM

Can software compiled by Mingw be used in a Linux environment?

View your Litecoin wallet address View your Litecoin wallet address Apr 07, 2024 pm 05:12 PM

View your Litecoin wallet address

Let Siri no longer be mentally retarded! Apple defines a new client-side model, which is 'much better than GPT-4. It gets rid of text and visually simulates screen information. The minimum parameter model is still 5% better than the baseline system. Let Siri no longer be mentally retarded! Apple defines a new client-side model, which is 'much better than GPT-4. It gets rid of text and visually simulates screen information. The minimum parameter model is still 5% better than the baseline system. Apr 02, 2024 pm 09:20 PM

Let Siri no longer be mentally retarded! Apple defines a new client-side model, which is 'much better than GPT-4. It gets rid of text and visually simulates screen information. The minimum parameter model is still 5% better than the baseline system.

Metis: The ReGenesis project we launched is a rebirth of the blockchain ecosystem Metis: The ReGenesis project we launched is a rebirth of the blockchain ecosystem Mar 04, 2025 pm 10:06 PM

Metis: The ReGenesis project we launched is a rebirth of the blockchain ecosystem

How to solve external resource calls and access in PHP development How to solve external resource calls and access in PHP development Oct 08, 2023 pm 06:37 PM

How to solve external resource calls and access in PHP development

See all articles