


Why is downloading in memory slower than downloading in the file system from aws s3?
Why is downloading in memory slower than downloading in the file system from AWS S3? When downloading files, we usually choose to download from an AWS S3 bucket to the local file system. However, sometimes we find that using the in-memory download method is slower than downloading directly from S3 to the file system. This is because downloading in memory involves some extra steps and resource consumption. First, in-memory downloads require reading the file contents into memory and then writing them to the file system. This process involves additional memory operations and IO operations, which will cause the download speed to be slower than downloading directly from S3 to the file system. In addition, in-memory downloads may also be affected by memory limitations. When the downloaded files are large, it may cause insufficient memory problems, thereby affecting the download speed. Therefore, when choosing a download method, you need to weigh the pros and cons according to the specific situation and choose the most suitable method for downloading.
Question content
I am using aws gosdk to download from a certain bucket. The following are two implementations for download
- Download to file
func (a *awsclient) downloadtofile(ctx context.context, objectkey string) (string, error) { params := &awss3.getobjectinput{ bucket: aws.string(a.bucket), key: aws.string(objectkey), } downloadpath := "some/valid/path" f, err := os.create(downloadpath) defer f.close() _, err = a.downloader.download(ctx, f, params) return downloadpath, err }
- Download to memory
func (a *AwsClient) DownloadToMemory(ctx context.Context, objectKey string) (string, error) { params := &awsS3.GetObjectInput{ Bucket: aws.String(a.bucket), Key: aws.String(objectKey), } buffer := manager.NewWriteAtBuffer([]byte{}) _, err = a.downloader.Download(ctx, buffer, params) return buffer.Bytes(), err }
For a 100 mb file, it takes 30 seconds to download to memory and only 8 seconds to download to the file system. My expectation is that memory downloads should be much faster. My system (apple m1, ventura, 8gb ram) has enough ram available so this is not an issue. Can anyone help me understand this behavior?
Solution
Downloading large S3 objects into dynamic buffers is very inefficient. The buffer was reallocated multiple times to handle 100M of data and multiple download threads. Memory reallocation requires a lot of CPU time.
Try to allocate 100M at the beginning instead of using null byte slices.
If the object size is unknown, you can use S3.HeadObject to get the object length in real time.
The above is the detailed content of Why is downloading in memory slower than downloading in the file system from aws s3?. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics



Real-time Bitcoin USD Price Factors that affect Bitcoin price Indicators for predicting future Bitcoin prices Here are some key information about the price of Bitcoin in 2018-2024:

Yes, H5 page production is an important implementation method for front-end development, involving core technologies such as HTML, CSS and JavaScript. Developers build dynamic and powerful H5 pages by cleverly combining these technologies, such as using the <canvas> tag to draw graphics or using JavaScript to control interaction behavior.

The method of customizing resize symbols in CSS is unified with background colors. In daily development, we often encounter situations where we need to customize user interface details, such as adjusting...

Regarding the reasons and solutions for misaligned display of inline-block elements. When writing web page layout, we often encounter some seemingly strange display problems. Compare...

How to use JavaScript or CSS to control the top and end of the page in the browser's printing settings. In the browser's printing settings, there is an option to control whether the display is...

The problem of container opening due to excessive omission of text under Flex layout and solutions are used...

How to achieve the 45-degree curve effect of segmenter? In the process of implementing the segmenter, how to make the right border turn into a 45-degree curve when clicking the left button, and the point...

Tips for Implementing Segmenter Effects In user interface design, segmenter is a common navigation element, especially in mobile applications and responsive web pages. ...
