Table of Contents
Question content
Solution
Home Backend Development Golang Why is downloading in memory slower than downloading in the file system from aws s3?

Why is downloading in memory slower than downloading in the file system from aws s3?

Feb 09, 2024 am 11:57 AM
overflow

为什么在内存中下载比从 aws s3 在文件系统中下载慢?

Why is downloading in memory slower than downloading in the file system from AWS S3? When downloading files, we usually choose to download from an AWS S3 bucket to the local file system. However, sometimes we find that using the in-memory download method is slower than downloading directly from S3 to the file system. This is because downloading in memory involves some extra steps and resource consumption. First, in-memory downloads require reading the file contents into memory and then writing them to the file system. This process involves additional memory operations and IO operations, which will cause the download speed to be slower than downloading directly from S3 to the file system. In addition, in-memory downloads may also be affected by memory limitations. When the downloaded files are large, it may cause insufficient memory problems, thereby affecting the download speed. Therefore, when choosing a download method, you need to weigh the pros and cons according to the specific situation and choose the most suitable method for downloading.

Question content

I am using aws gosdk to download from a certain bucket. The following are two implementations for download

  1. Download to file
func (a *awsclient) downloadtofile(ctx context.context, objectkey string) (string, error) {
    params := &awss3.getobjectinput{
        bucket: aws.string(a.bucket),
        key:    aws.string(objectkey),
    }

    downloadpath := "some/valid/path"
    f, err := os.create(downloadpath)
    defer f.close()
    _, err = a.downloader.download(ctx, f, params)
    return downloadpath, err
}
Copy after login
  • Download to memory
  • func (a *AwsClient) DownloadToMemory(ctx context.Context, objectKey string) (string, error) {
        params := &awsS3.GetObjectInput{
            Bucket: aws.String(a.bucket),
            Key:    aws.String(objectKey),
        }
    
        buffer := manager.NewWriteAtBuffer([]byte{})  
        _, err = a.downloader.Download(ctx, buffer, params)
        return buffer.Bytes(), err
    }
    Copy after login

    For a 100 mb file, it takes 30 seconds to download to memory and only 8 seconds to download to the file system. My expectation is that memory downloads should be much faster. My system (apple m1, ventura, 8gb ram) has enough ram available so this is not an issue. Can anyone help me understand this behavior?

    Solution

    Downloading large S3 objects into dynamic buffers is very inefficient. The buffer was reallocated multiple times to handle 100M of data and multiple download threads. Memory reallocation requires a lot of CPU time.

    Try to allocate 100M at the beginning instead of using null byte slices.

    If the object size is unknown, you can use S3.HeadObject to get the object length in real time.

    The above is the detailed content of Why is downloading in memory slower than downloading in the file system from aws s3?. For more information, please follow other related articles on the PHP Chinese website!

    Statement of this Website
    The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

    Hot AI Tools

    Undresser.AI Undress

    Undresser.AI Undress

    AI-powered app for creating realistic nude photos

    AI Clothes Remover

    AI Clothes Remover

    Online AI tool for removing clothes from photos.

    Undress AI Tool

    Undress AI Tool

    Undress images for free

    Clothoff.io

    Clothoff.io

    AI clothes remover

    AI Hentai Generator

    AI Hentai Generator

    Generate AI Hentai for free.

    Hot Article

    R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
    2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
    Repo: How To Revive Teammates
    1 months ago By 尊渡假赌尊渡假赌尊渡假赌
    Hello Kitty Island Adventure: How To Get Giant Seeds
    4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

    Hot Tools

    Notepad++7.3.1

    Notepad++7.3.1

    Easy-to-use and free code editor

    SublimeText3 Chinese version

    SublimeText3 Chinese version

    Chinese version, very easy to use

    Zend Studio 13.0.1

    Zend Studio 13.0.1

    Powerful PHP integrated development environment

    Dreamweaver CS6

    Dreamweaver CS6

    Visual web development tools

    SublimeText3 Mac version

    SublimeText3 Mac version

    God-level code editing software (SublimeText3)

    The price of Bitcoin since its birth 2009-2025 The most complete summary of BTC historical prices The price of Bitcoin since its birth 2009-2025 The most complete summary of BTC historical prices Jan 15, 2025 pm 08:11 PM

    Since its inception in 2009, Bitcoin has become a leader in the cryptocurrency world and its price has experienced huge fluctuations. To provide a comprehensive historical overview, this article compiles Bitcoin price data from 2009 to 2025, covering major market events, changes in market sentiment, and important factors influencing price movements.

    What to do if the time is gone in the lower right corner of Windows 11_What to do if the time is gone in the lower right corner of Windows 11 What to do if the time is gone in the lower right corner of Windows 11_What to do if the time is gone in the lower right corner of Windows 11 May 06, 2024 pm 01:20 PM

    1. First, right-click on the blank space of the taskbar at the bottom of Windows 11 and select [Taskbar Settings]. 2. Find [taskbarcorneroverflow] on the right in the taskbar settings. 3. Then find [clock] or [Clock] above it and select to turn it on. Method 2: 1. Press the keyboard shortcut [win+r] to call up run, enter [regedit] and press Enter to confirm. 2. Open the registry editor, find [HKEY_CURRENT_USERControlPanel] in it, and delete it. 3. After deletion, restart the computer and you will be prompted for configuration. When you return to the system, the time will be displayed.

    Are there any community forums or discussion groups for Java functions where I can ask questions and discuss them? Are there any community forums or discussion groups for Java functions where I can ask questions and discuss them? Apr 28, 2024 pm 02:12 PM

    Answer: The following community forums and discussion groups are available for Java functional programming questions: StackOverflow: The world's largest programming Q&A website with a community of Java functional programming experts. JavaFunctionalProgramming: A community forum focused on Java functional programming, providing discussions on concepts, language features, and best practices. Redditr/functionaljava: A subreddit focused on functional programming in Java, focusing on tools, libraries, and technologies. Discord: JavaFunctional Programming: Discord service that provides real-time discussion, code sharing and collaboration

    How to use other people's code in python How to use other people's code in python May 05, 2024 pm 07:54 PM

    How do I use other people's Python code? Find code repositories: Find the code you need on platforms like PyPI and GitHub. Installation code: Use pip or clone the GitHub repository to install. Import modules: Use the import statement in your script to import installed modules. Working with code: Access functions and classes in modules. (Optional) Adapt the code: Modify the code as needed to fit your project.

    What should I do if the time on my win11 computer is always wrong? How to adjust the wrong time on Windows 11 computer What should I do if the time on my win11 computer is always wrong? How to adjust the wrong time on Windows 11 computer May 03, 2024 pm 09:20 PM

    What should I do if the time on my win11 computer is always wrong? We all set the time or calendar when using win11 system, but many users are asking that the computer time is always wrong, so what is going on? Users can directly click on the taskbar below, and then find taskbarcorneroverflow to set it up. Let this site introduce to users in detail how to adjust the time error on Win11 computers. How to adjust the computer time error in Windows 11. Method 1: 1. We first right-click on the blank space of the taskbar below and select Taskbar Settings. Method 2: 1. Press the keyboard shortcut win+r to call up run, enter regedit and press Enter to confirm.

    Common exception types and their repair measures in Java function development Common exception types and their repair measures in Java function development May 03, 2024 pm 02:09 PM

    Common exception types and their repair measures in Java function development During the development of Java functions, various exceptions may be encountered, which affect the correct execution of the function. The following are common exception types and their repair measures: 1. NullPointerException Description: Thrown when accessing an object that has not been initialized. Fix: Make sure you check the object for non-null before using it. Sample code: try{Stringname=null;System.out.println(name.length());}catch(NullPointerExceptione){

    What does overflow mean in css What does overflow mean in css Apr 28, 2024 pm 03:15 PM

    overflow is a property of CSS that is used to control the display mode of element content when it exceeds the container. Available values ​​include: visible: the content is visible, the overflow container is hidden: the overflow content is cut scroll: the scroll bar is displayed to view the overflow content auto: the browser automatically determines Whether to display the scroll bar inherit: inherit the overflow attribute of the parent element

    Doesn't anyone take care of Douyin's random accounts? Can I appeal a second time? Doesn't anyone take care of Douyin's random accounts? Can I appeal a second time? May 03, 2024 am 09:37 AM

    As a world-renowned short video platform, Douyin has a huge user base and content creators. However, as the platform rules are constantly updated and improved, some users may encounter account bans. This has raised public questions about the transparency and fairness of platform management. This article will discuss the issue of Douyin account bans and whether users have ways to appeal after their accounts are banned. There may be many reasons for being banned on the Douyin platform, including but not limited to illegal content, violation of platform regulations, infringement of other people's rights, etc. In order to maintain the order of the platform and the interests of users, Douyin has set up a series of rules and review mechanisms. When some users violate the rules, their accounts may be banned. However, some users may question or be dissatisfied with the reasons for the ban

    See all articles