Table of Contents
Question content
Solution
Home Backend Development Golang Why is downloading in memory slower than downloading in the file system from aws s3?

Why is downloading in memory slower than downloading in the file system from aws s3?

Feb 09, 2024 am 11:57 AM
overflow

为什么在内存中下载比从 aws s3 在文件系统中下载慢?

Why is downloading in memory slower than downloading in the file system from AWS S3? When downloading files, we usually choose to download from an AWS S3 bucket to the local file system. However, sometimes we find that using the in-memory download method is slower than downloading directly from S3 to the file system. This is because downloading in memory involves some extra steps and resource consumption. First, in-memory downloads require reading the file contents into memory and then writing them to the file system. This process involves additional memory operations and IO operations, which will cause the download speed to be slower than downloading directly from S3 to the file system. In addition, in-memory downloads may also be affected by memory limitations. When the downloaded files are large, it may cause insufficient memory problems, thereby affecting the download speed. Therefore, when choosing a download method, you need to weigh the pros and cons according to the specific situation and choose the most suitable method for downloading.

Question content

I am using aws gosdk to download from a certain bucket. The following are two implementations for download

  1. Download to file
func (a *awsclient) downloadtofile(ctx context.context, objectkey string) (string, error) {
    params := &awss3.getobjectinput{
        bucket: aws.string(a.bucket),
        key:    aws.string(objectkey),
    }

    downloadpath := "some/valid/path"
    f, err := os.create(downloadpath)
    defer f.close()
    _, err = a.downloader.download(ctx, f, params)
    return downloadpath, err
}
Copy after login
  • Download to memory
  • func (a *AwsClient) DownloadToMemory(ctx context.Context, objectKey string) (string, error) {
        params := &awsS3.GetObjectInput{
            Bucket: aws.String(a.bucket),
            Key:    aws.String(objectKey),
        }
    
        buffer := manager.NewWriteAtBuffer([]byte{})  
        _, err = a.downloader.Download(ctx, buffer, params)
        return buffer.Bytes(), err
    }
    Copy after login

    For a 100 mb file, it takes 30 seconds to download to memory and only 8 seconds to download to the file system. My expectation is that memory downloads should be much faster. My system (apple m1, ventura, 8gb ram) has enough ram available so this is not an issue. Can anyone help me understand this behavior?

    Solution

    Downloading large S3 objects into dynamic buffers is very inefficient. The buffer was reallocated multiple times to handle 100M of data and multiple download threads. Memory reallocation requires a lot of CPU time.

    Try to allocate 100M at the beginning instead of using null byte slices.

    If the object size is unknown, you can use S3.HeadObject to get the object length in real time.

    The above is the detailed content of Why is downloading in memory slower than downloading in the file system from aws s3?. For more information, please follow other related articles on the PHP Chinese website!

    Statement of this Website
    The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

    Hot AI Tools

    Undresser.AI Undress

    Undresser.AI Undress

    AI-powered app for creating realistic nude photos

    AI Clothes Remover

    AI Clothes Remover

    Online AI tool for removing clothes from photos.

    Undress AI Tool

    Undress AI Tool

    Undress images for free

    Clothoff.io

    Clothoff.io

    AI clothes remover

    AI Hentai Generator

    AI Hentai Generator

    Generate AI Hentai for free.

    Hot Tools

    Notepad++7.3.1

    Notepad++7.3.1

    Easy-to-use and free code editor

    SublimeText3 Chinese version

    SublimeText3 Chinese version

    Chinese version, very easy to use

    Zend Studio 13.0.1

    Zend Studio 13.0.1

    Powerful PHP integrated development environment

    Dreamweaver CS6

    Dreamweaver CS6

    Visual web development tools

    SublimeText3 Mac version

    SublimeText3 Mac version

    God-level code editing software (SublimeText3)

    The latest price of Bitcoin in 2018-2024 USD The latest price of Bitcoin in 2018-2024 USD Feb 15, 2025 pm 07:12 PM

    Real-time Bitcoin USD Price Factors that affect Bitcoin price Indicators for predicting future Bitcoin prices Here are some key information about the price of Bitcoin in 2018-2024:

    Is H5 page production a front-end development? Is H5 page production a front-end development? Apr 05, 2025 pm 11:42 PM

    Yes, H5 page production is an important implementation method for front-end development, involving core technologies such as HTML, CSS and JavaScript. Developers build dynamic and powerful H5 pages by cleverly combining these technologies, such as using the <canvas> tag to draw graphics or using JavaScript to control interaction behavior.

    How to customize the resize symbol through CSS and make it uniform with the background color? How to customize the resize symbol through CSS and make it uniform with the background color? Apr 05, 2025 pm 02:30 PM

    The method of customizing resize symbols in CSS is unified with background colors. In daily development, we often encounter situations where we need to customize user interface details, such as adjusting...

    Why are the inline-block elements misaligned? How to solve this problem? Why are the inline-block elements misaligned? How to solve this problem? Apr 04, 2025 pm 10:39 PM

    Regarding the reasons and solutions for misaligned display of inline-block elements. When writing web page layout, we often encounter some seemingly strange display problems. Compare...

    How to control the top and end of pages in browser printing settings through JavaScript or CSS? How to control the top and end of pages in browser printing settings through JavaScript or CSS? Apr 05, 2025 pm 10:39 PM

    How to use JavaScript or CSS to control the top and end of the page in the browser's printing settings. In the browser's printing settings, there is an option to control whether the display is...

    The text under Flex layout is omitted but the container is opened? How to solve it? The text under Flex layout is omitted but the container is opened? How to solve it? Apr 05, 2025 pm 11:00 PM

    The problem of container opening due to excessive omission of text under Flex layout and solutions are used...

    How to use the clip-path attribute of CSS to achieve the 45-degree curve effect of segmenter? How to use the clip-path attribute of CSS to achieve the 45-degree curve effect of segmenter? Apr 04, 2025 pm 11:45 PM

    How to achieve the 45-degree curve effect of segmenter? In the process of implementing the segmenter, how to make the right border turn into a 45-degree curve when clicking the left button, and the point...

    How to achieve segmentation effect with 45 degree curve border? How to achieve segmentation effect with 45 degree curve border? Apr 04, 2025 pm 11:48 PM

    Tips for Implementing Segmenter Effects In user interface design, segmenter is a common navigation element, especially in mobile applications and responsive web pages. ...

    See all articles