How to Stream File Uploads to AWS S3 Using Go?

Linda Hamilton
Release: 2024-11-07 20:13:03
Original
1003 people have browsed it

How to Stream File Uploads to AWS S3 Using Go?

Stream File Upload to AWS S3 Using Go

Overview

Uploading large files directly to AWS S3 while minimizing memory and disk footprint is an essential task in cloud computing. This guide will demonstrate how to achieve this using the AWS SDK for Go.

Solution

To stream a file upload directly to S3, you can utilize the s3manager package. Here's a step-by-step solution:

  1. Configure AWS Credentials and Session:

    • Set your AWS access key and secret, or use the default credentials provider.
    • Initialize an AWS session with the specified configuration.
  2. Create an S3 Uploader:

    • Initialize an S3 uploader with the session and optional configuration settings.
    • You can configure parameters such as part size, concurrency, and max upload parts.
  3. Open the File:

    • Open the file you want to upload using the os.Open function.
  4. Upload the File:

    • Use the uploader.Upload method with the appropriate file information (bucket, key, and file pointer).

Example Code

The following code sample demonstrates how to stream large file upload to AWS S3 using s3manager:

package main

import (
    "fmt"
    "os"

    "github.com/aws/aws-sdk-go/aws/credentials"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3/s3manager"
)

const (
    filename      = "file_name.zip"
    myBucket      = "myBucket"
    myKey         = "file_name.zip"
    accessKey     = ""
    accessSecret  = ""
)

func main() {
    var awsConfig *aws.Config
    if accessKey == "" || accessSecret == "" {
        //load default credentials
        awsConfig = &aws.Config{
            Region: aws.String("us-west-2"),
        }
    } else {
        awsConfig = &aws.Config{
            Region:      aws.String("us-west-2"),
            Credentials: credentials.NewStaticCredentials(accessKey, accessSecret, ""),
        }
    }

    // The session the S3 Uploader will use
    sess := session.Must(session.NewSession(awsConfig))

    // Create an uploader with the session and default options
    //uploader := s3manager.NewUploader(sess)

    // Create an uploader with the session and custom options
    uploader := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {
        u.PartSize = 5 * 1024 * 1024 // The minimum/default allowed part size is 5MB
        u.Concurrency = 2            // default is 5
    })

    //open the file
    f, err := os.Open(filename)
    if err != nil {
        fmt.Printf("failed to open file %q, %v", filename, err)
        return
    }
    //defer f.Close()

    // Upload the file to S3.
    result, err := uploader.Upload(&s3manager.UploadInput{
        Bucket: aws.String(myBucket),
        Key:    aws.String(myKey),
        Body:   f,
    })

    //in case it fails to upload
    if err != nil {
        fmt.Printf("failed to upload file, %v", err)
        return
    }
    fmt.Printf("file uploaded to, %s\n", result.Location)
}
Copy after login

By following these steps, you can efficiently upload large multipart/form-data files directly to AWS S3 with minimal memory utilization.

The above is the detailed content of How to Stream File Uploads to AWS S3 Using Go?. For more information, please follow other related articles on the PHP Chinese website!

source:php.cn
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Latest Articles by Author
Popular Tutorials
More>
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template