Home PHP Framework Laravel Detailed explanation on using Lazy Collections to improve the performance of Laravel Excel reading (easily supports millions of data)

Detailed explanation on using Lazy Collections to improve the performance of Laravel Excel reading (easily supports millions of data)

Mar 21, 2020 am 08:56 AM
laravel

A new type of collection has been added in Laravel 6: Lazy Collections. They are great if you need to process very large data sets (thousands or millions of rows) without running into memory constraints.

Recommended: laravel tutorial

My most recent task was to refactor Excel export in a project at work. The problem is that the export can no longer be created because the dataset is too large for Laravel to handle. The database query returned approximately 300,000 results! The application times out or consistently runs out of memory.

A naive approach is to increase the timeout or memory limit and hope that the next time something goes wrong, someone else will handle it. But that's not how I work. I don't like Band-Aids. I like concrete, long-term solutions.

Laravel Excel extension pack is already quite flexible. It does a great job of reducing database load by using "chunks" when using FromQuery-concerns. However, our export still struggles with large data sets.

My colleagues and I discussed whether we should completely rewrite this feature: push the export to a queue and send a notification to the user when the export is complete. However, this feature is just a minor thing in this app. It doesn't make sense to us to add so much overhead just for a simple export.

Later that day, I had a little "Eureka" moment because I remembered that Laravel had LazyCollections.

I rewrote the export: it now uses FromCollection-concern instead of FromQuery. The only change I had to make to the collection() method was to replace the get() method at the end of the query builder chain with a cursor().

Below is a simplified version of our export functionality. The Request object is passed through the constructor so we can adapt the query based on what the user selects in the UI.

<?php 
namespace App\Exports;
use App\User;
use Maatwebsite\Excel\Concerns\Exportable;
use Maatwebsite\Excel\Concerns\FromCollection;
use Illuminate\Http\Request;
class UsersExport implements FromCollection
{
    use Exportable;
    protected Request $request;
    public function __construct(Request $request)
    {
        $this->request = $request;
    }
    public function collection()
    {
        return User::query()
            ->when($this->request->get(&#39;include_subscribed&#39;), function ($q) {
                return $q->where(&#39;is_subscribed&#39;, true);
            })
            ->cursor(); // ← 重要的一点
    }
}
Copy after login

I believe you are experiencing memory issues in your project. You increased the memory limit and hopefully that solved the problem (I've done this countless times myself).

If it were in a Laravel project, I hope, I could have you revisit that code and rewrite it using LazyCollections.

Fixing this issue was very interesting, so I did a little benchmark: our export can now easily export millions of rows without hitting memory limits. so cool!

The above is the detailed content of Detailed explanation on using Lazy Collections to improve the performance of Laravel Excel reading (easily supports millions of data). For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to get the return code when email sending fails in Laravel? How to get the return code when email sending fails in Laravel? Apr 01, 2025 pm 02:45 PM

Method for obtaining the return code when Laravel email sending fails. When using Laravel to develop applications, you often encounter situations where you need to send verification codes. And in reality...

Laravel schedule task is not executed: What should I do if the task is not running after schedule: run command? Laravel schedule task is not executed: What should I do if the task is not running after schedule: run command? Mar 31, 2025 pm 11:24 PM

Laravel schedule task run unresponsive troubleshooting When using Laravel's schedule task scheduling, many developers will encounter this problem: schedule:run...

In Laravel, how to deal with the situation where verification codes are failed to be sent by email? In Laravel, how to deal with the situation where verification codes are failed to be sent by email? Mar 31, 2025 pm 11:48 PM

The method of handling Laravel's email failure to send verification code is to use Laravel...

How to implement the custom table function of clicking to add data in dcat admin? How to implement the custom table function of clicking to add data in dcat admin? Apr 01, 2025 am 07:09 AM

How to implement the table function of custom click to add data in dcatadmin (laravel-admin) When using dcat...

Laravel - Dump Server Laravel - Dump Server Aug 27, 2024 am 10:51 AM

Laravel - Dump Server - Laravel dump server comes with the version of Laravel 5.7. The previous versions do not include any dump server. Dump server will be a development dependency in laravel/laravel composer file.

Laravel Redis connection sharing: Why does the select method affect other connections? Laravel Redis connection sharing: Why does the select method affect other connections? Apr 01, 2025 am 07:45 AM

The impact of sharing of Redis connections in Laravel framework and select methods When using Laravel framework and Redis, developers may encounter a problem: through configuration...

Laravel multi-tenant extension stancl/tenancy: How to customize the host address of a tenant database connection? Laravel multi-tenant extension stancl/tenancy: How to customize the host address of a tenant database connection? Apr 01, 2025 am 09:09 AM

Custom tenant database connection in Laravel multi-tenant extension package stancl/tenancy When building multi-tenant applications using Laravel multi-tenant extension package stancl/tenancy,...

Laravel - Action URL Laravel - Action URL Aug 27, 2024 am 10:51 AM

Laravel - Action URL - Laravel 5.7 introduces a new feature called “callable action URL”. This feature is similar to the one in Laravel 5.6 which accepts string in action method. The main purpose of the new syntax introduced Laravel 5.7 is to directl

See all articles