How to export large amounts of data in php
In the process of PHP development, sometimes there is data that needs to be exported. You can use some packages in various frameworks to directly obtain the data and then throw it into the corresponding processing method. However, when encountering a large amount of data that needs to be exported, At that time, we need to use other methods to achieve it.
Generally, there are several solutions to deal with big data: Generate data by adding or modifying event trigger scripts; generate data by executing scheduled task scripts; directly use phpexcel to export large amounts of data; use fputcsv real-time data flow to write . Use a picture to represent it as follows:
#Here we mainly record the use of fputcsv to export data.
public function exportData()
{
set_time_limit(0);
ini_set('memory_limit', '1024M');
$columns = [ '列名1', '列名2', '列名3' //需要几列，定义好列名
]; //设置好告诉浏览器要下载excel文件的headers
header('Content-Description: File Transfer');
header('Content-Type: application/vnd.ms-excel');
header('Content-Disposition: attachment; filename="导出数据-'.date('Y-m-d', time()).'.csv"');
header('Expires: 0');
header('Cache-Control: must-revalidate');
header('Pragma: public'); $fp = fopen('php://output', 'a');//打开output流
mb_convert_variables('GBK', 'UTF-8', $columns);
fputcsv($fp, $columns);//将数据格式化为CSV格式并写入到output流中
//添加查询条件，获取需要的数据
$query = Model::class()->where();
//获取总数，分页循环处理
$accessNum = $query->count();
$perSize = 1000;
$pages = ceil($accessNum / $perSize);
for($i = 1; $i <= $pages; $i++) {
$db_data = $query->limit($perSize)->offset(($i-1)*$perSize)->get();
foreach($db_data as $key => $value) {
$rowData = []; //获取每列数据，转换处理成需要导出的数据
//需要格式转换，否则会乱码
mb_convert_variables('GBK', 'UTF-8', $rowData);
fputcsv($fp, $rowData);
} //释放变量的内存
unset($db_data); //刷新输出缓冲到浏览器
ob_flush(); //必须同时使用 ob_flush() 和flush() 函数来刷新输出缓冲。
flush();
}
fclose($fp);
exit();
}This will generate and write the downloaded file at the same time, improving the processing speed.
The package processing used before generates a 25M data, which takes 20 minutes. This method of processing takes more than 1 minute (also related to the actual processing environment).
Related recommendations:
PHP csv large amount of data export split processing
php import large amount of data to mysql (example)
The above is the detailed content of How to export large amounts of data in php. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1664
14
1423
52
1317
25
1268
29
1246
24
PHP and Python: Comparing Two Popular Programming Languages
Apr 14, 2025 am 12:13 AM
PHP and Python each have their own advantages, and choose according to project requirements. 1.PHP is suitable for web development, especially for rapid development and maintenance of websites. 2. Python is suitable for data science, machine learning and artificial intelligence, with concise syntax and suitable for beginners.
PHP in Action: Real-World Examples and Applications
Apr 14, 2025 am 12:19 AM
PHP is widely used in e-commerce, content management systems and API development. 1) E-commerce: used for shopping cart function and payment processing. 2) Content management system: used for dynamic content generation and user management. 3) API development: used for RESTful API development and API security. Through performance optimization and best practices, the efficiency and maintainability of PHP applications are improved.
Explain secure password hashing in PHP (e.g., password_hash, password_verify). Why not use MD5 or SHA1?
Apr 17, 2025 am 12:06 AM
In PHP, password_hash and password_verify functions should be used to implement secure password hashing, and MD5 or SHA1 should not be used. 1) password_hash generates a hash containing salt values to enhance security. 2) Password_verify verify password and ensure security by comparing hash values. 3) MD5 and SHA1 are vulnerable and lack salt values, and are not suitable for modern password security.
How does PHP handle file uploads securely?
Apr 10, 2025 am 09:37 AM
PHP handles file uploads through the $\_FILES variable. The methods to ensure security include: 1. Check upload errors, 2. Verify file type and size, 3. Prevent file overwriting, 4. Move files to a permanent storage location.
PHP: A Key Language for Web Development
Apr 13, 2025 am 12:08 AM
PHP is a scripting language widely used on the server side, especially suitable for web development. 1.PHP can embed HTML, process HTTP requests and responses, and supports a variety of databases. 2.PHP is used to generate dynamic web content, process form data, access databases, etc., with strong community support and open source resources. 3. PHP is an interpreted language, and the execution process includes lexical analysis, grammatical analysis, compilation and execution. 4.PHP can be combined with MySQL for advanced applications such as user registration systems. 5. When debugging PHP, you can use functions such as error_reporting() and var_dump(). 6. Optimize PHP code to use caching mechanisms, optimize database queries and use built-in functions. 7
How does PHP type hinting work, including scalar types, return types, union types, and nullable types?
Apr 17, 2025 am 12:25 AM
PHP type prompts to improve code quality and readability. 1) Scalar type tips: Since PHP7.0, basic data types are allowed to be specified in function parameters, such as int, float, etc. 2) Return type prompt: Ensure the consistency of the function return value type. 3) Union type prompt: Since PHP8.0, multiple types are allowed to be specified in function parameters or return values. 4) Nullable type prompt: Allows to include null values and handle functions that may return null values.
The Enduring Relevance of PHP: Is It Still Alive?
Apr 14, 2025 am 12:12 AM
PHP is still dynamic and still occupies an important position in the field of modern programming. 1) PHP's simplicity and powerful community support make it widely used in web development; 2) Its flexibility and stability make it outstanding in handling web forms, database operations and file processing; 3) PHP is constantly evolving and optimizing, suitable for beginners and experienced developers.
PHP vs. Python: Understanding the Differences
Apr 11, 2025 am 12:15 AM
PHP and Python each have their own advantages, and the choice should be based on project requirements. 1.PHP is suitable for web development, with simple syntax and high execution efficiency. 2. Python is suitable for data science and machine learning, with concise syntax and rich libraries.
