Uploading large files directly to AWS S3 while minimizing memory and disk footprint is an essential task in cloud computing. This guide will demonstrate how to achieve this using the AWS SDK for Go.
To stream a file upload directly to S3, you can utilize the s3manager package. Here's a step-by-step solution:
Configure AWS Credentials and Session:
Create an S3 Uploader:
Open the File:
Upload the File:
The following code sample demonstrates how to stream large file upload to AWS S3 using s3manager:
package main import ( "fmt" "os" "github.com/aws/aws-sdk-go/aws/credentials" "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/s3/s3manager" ) const ( filename = "file_name.zip" myBucket = "myBucket" myKey = "file_name.zip" accessKey = "" accessSecret = "" ) func main() { var awsConfig *aws.Config if accessKey == "" || accessSecret == "" { //load default credentials awsConfig = &aws.Config{ Region: aws.String("us-west-2"), } } else { awsConfig = &aws.Config{ Region: aws.String("us-west-2"), Credentials: credentials.NewStaticCredentials(accessKey, accessSecret, ""), } } // The session the S3 Uploader will use sess := session.Must(session.NewSession(awsConfig)) // Create an uploader with the session and default options //uploader := s3manager.NewUploader(sess) // Create an uploader with the session and custom options uploader := s3manager.NewUploader(sess, func(u *s3manager.Uploader) { u.PartSize = 5 * 1024 * 1024 // The minimum/default allowed part size is 5MB u.Concurrency = 2 // default is 5 }) //open the file f, err := os.Open(filename) if err != nil { fmt.Printf("failed to open file %q, %v", filename, err) return } //defer f.Close() // Upload the file to S3. result, err := uploader.Upload(&s3manager.UploadInput{ Bucket: aws.String(myBucket), Key: aws.String(myKey), Body: f, }) //in case it fails to upload if err != nil { fmt.Printf("failed to upload file, %v", err) return } fmt.Printf("file uploaded to, %s\n", result.Location) }
By following these steps, you can efficiently upload large multipart/form-data files directly to AWS S3 with minimal memory utilization.
The above is the detailed content of How to Stream File Uploads to AWS S3 Using Go?. For more information, please follow other related articles on the PHP Chinese website!