请教大家关于php for循环中的 大数据量处理时自动exit的问题
<code> /** * @function getSplitCache 对excelId基础数据进行分片缓存 * @param type $excelId excel文件id tax_excel的id * @param type $beforeAndAfterInsertid details_excel 插入前后的id * @return 无 统一excel的缓存文件标准为: EXCEL_ID_NUM */ public function getSplitCache($excelId, $beforeAndAfterInsertid){ processTip( '100%', 'Excel数据分片开始.....................'); $where['id'] = array('gt', $beforeAndAfterInsertid['beforeId']); $where['excel_id'] = array('eq', $excelId); //是否可能存在两个表同时插入情况,如果无可省略此项 $allExcelData = $this->modelDetailsExcel->where($where)->group('nsrsbh,id')->select(); $totalNum = count($allExcelData); $this->_setAllRow($totalNum); // $interval = ceil( ($beforeAndAfterInsertid[1]-$beforeAndAfterInsertid[0])/$this->excelDataCacheNum);//多个人同时出现操作插入文件是可能错误 $interval = ceil($totalNum/$this->excelDataCacheNum); for($i = 0; $iexcelDataCacheNum; $i++ ){ $excelData[$i] = array_slice($allExcelData, $i*$interval, $interval); } unset($allExcelData); //1一个在开头一个在结尾利用时间差可以不再细分 //2必须分在一起以下为分在一起的 $i = 0; for($i; $iexcelDataCacheNum-1; $i++ ){ if($excelData[$i][$interval-1]['nsrsbh'] === $excelData[$i+1][0]['nsrsbh']){ $j = 0; while( $excelData[$i][$interval-1]['nsrsbh'] === $excelData[$i+1][$j]['nsrsbh']){ $excelData[$i][] = $excelData[$i+1][$j++]; P($j); } array_splice($excelData[$i+1], 0, $j); } //输出前台展示进度 processTip( ($i/$this->excelDataCacheNum*100), 'Excel数据分片'.$i.'.....................'); //Thinkphp的F缓存方法 指定位置和名称 F('Excel_'.$excelId.'/'.$i , $excelData[$i]); } F('Excel_'.$excelId.'/'. $i , $excelData[$i]); processTip( '100%', 'Excel数据分片结束.....................'); return $this->excelDataCacheNum; } </code>
基本原理是 将数据库数据按照nsrsbh 和 id排序 读取出来,然后将数据按照分片数据 分割指定数目的文件(用于前台ajax请求,模拟多线程操作,js发送请求后端接收后处理传过来的数据分片的文件name,然后处理改文件内的数据)
系统环境 apche php 分配的memoryliimit win(win10 64bit 4G内存)为2g linux(ubuntu14.04 64位 8g内存)为4g
经过测试当数据量在3500条时分片1000正常执行。
当数据量在11770条时,linux下和win下开始出现这个问题
到978片缓存生成时就进行不下去,php程序自动跳出 效果如下 然后进程就退出了。
当分片为500的时候出现到488 就会卡 然后到了过个2分钟左右才会输出489 然后没有然后了 进程跳出了
回复内容:
<code> /** * @function getSplitCache 对excelId基础数据进行分片缓存 * @param type $excelId excel文件id tax_excel的id * @param type $beforeAndAfterInsertid details_excel 插入前后的id * @return 无 统一excel的缓存文件标准为: EXCEL_ID_NUM */ public function getSplitCache($excelId, $beforeAndAfterInsertid){ processTip( '100%', 'Excel数据分片开始.....................'); $where['id'] = array('gt', $beforeAndAfterInsertid['beforeId']); $where['excel_id'] = array('eq', $excelId); //是否可能存在两个表同时插入情况,如果无可省略此项 $allExcelData = $this->modelDetailsExcel->where($where)->group('nsrsbh,id')->select(); $totalNum = count($allExcelData); $this->_setAllRow($totalNum); // $interval = ceil( ($beforeAndAfterInsertid[1]-$beforeAndAfterInsertid[0])/$this->excelDataCacheNum);//多个人同时出现操作插入文件是可能错误 $interval = ceil($totalNum/$this->excelDataCacheNum); for($i = 0; $iexcelDataCacheNum; $i++ ){ $excelData[$i] = array_slice($allExcelData, $i*$interval, $interval); } unset($allExcelData); //1一个在开头一个在结尾利用时间差可以不再细分 //2必须分在一起以下为分在一起的 $i = 0; for($i; $iexcelDataCacheNum-1; $i++ ){ if($excelData[$i][$interval-1]['nsrsbh'] === $excelData[$i+1][0]['nsrsbh']){ $j = 0; while( $excelData[$i][$interval-1]['nsrsbh'] === $excelData[$i+1][$j]['nsrsbh']){ $excelData[$i][] = $excelData[$i+1][$j++]; P($j); } array_splice($excelData[$i+1], 0, $j); } //输出前台展示进度 processTip( ($i/$this->excelDataCacheNum*100), 'Excel数据分片'.$i.'.....................'); //Thinkphp的F缓存方法 指定位置和名称 F('Excel_'.$excelId.'/'.$i , $excelData[$i]); } F('Excel_'.$excelId.'/'. $i , $excelData[$i]); processTip( '100%', 'Excel数据分片结束.....................'); return $this->excelDataCacheNum; } </code>
基本原理是 将数据库数据按照nsrsbh 和 id排序 读取出来,然后将数据按照分片数据 分割指定数目的文件(用于前台ajax请求,模拟多线程操作,js发送请求后端接收后处理传过来的数据分片的文件name,然后处理改文件内的数据)
系统环境 apche php 分配的memoryliimit win(win10 64bit 4G内存)为2g linux(ubuntu14.04 64位 8g内存)为4g
经过测试当数据量在3500条时分片1000正常执行。
当数据量在11770条时,linux下和win下开始出现这个问题
到978片缓存生成时就进行不下去,php程序自动跳出 效果如下 然后进程就退出了。
当分片为500的时候出现到488 就会卡 然后到了过个2分钟左右才会输出489 然后没有然后了 进程跳出了
你去php.ini
文件中看看有没有配置最大变量长度的那个设置项,找到它,修改后服务器重启即可。
我以前恰好遇到过这个问题
-
打开 error_reporting
可在脚本最上方添加<code>php</code><code>error_reporing(E_ALL); ini_set('display_errors'); </code>
Copy after login监控数据量多的情况下,是不是内存使用超出限制
如果确认是内存溢出
改造你的程序,降低内存使用
比如使用generator
具体根据你的业务,尽量不要取太多数据放入内存
保证你程序运行的时候不要随着处理数据量的增大而内存也跟着增大
理想的情况是处理数据量的增长不会导致程序运行内存的暴增
set_time_limit(0),设置超时时间,不是内存的问题可能是超时了

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics



PHP 8.4 brings several new features, security improvements, and performance improvements with healthy amounts of feature deprecations and removals. This guide explains how to install PHP 8.4 or upgrade to PHP 8.4 on Ubuntu, Debian, or their derivati

Visual Studio Code, also known as VS Code, is a free source code editor — or integrated development environment (IDE) — available for all major operating systems. With a large collection of extensions for many programming languages, VS Code can be c

If you are an experienced PHP developer, you might have the feeling that you’ve been there and done that already.You have developed a significant number of applications, debugged millions of lines of code, and tweaked a bunch of scripts to achieve op

This tutorial demonstrates how to efficiently process XML documents using PHP. XML (eXtensible Markup Language) is a versatile text-based markup language designed for both human readability and machine parsing. It's commonly used for data storage an

JWT is an open standard based on JSON, used to securely transmit information between parties, mainly for identity authentication and information exchange. 1. JWT consists of three parts: Header, Payload and Signature. 2. The working principle of JWT includes three steps: generating JWT, verifying JWT and parsing Payload. 3. When using JWT for authentication in PHP, JWT can be generated and verified, and user role and permission information can be included in advanced usage. 4. Common errors include signature verification failure, token expiration, and payload oversized. Debugging skills include using debugging tools and logging. 5. Performance optimization and best practices include using appropriate signature algorithms, setting validity periods reasonably,

A string is a sequence of characters, including letters, numbers, and symbols. This tutorial will learn how to calculate the number of vowels in a given string in PHP using different methods. The vowels in English are a, e, i, o, u, and they can be uppercase or lowercase. What is a vowel? Vowels are alphabetic characters that represent a specific pronunciation. There are five vowels in English, including uppercase and lowercase: a, e, i, o, u Example 1 Input: String = "Tutorialspoint" Output: 6 explain The vowels in the string "Tutorialspoint" are u, o, i, a, o, i. There are 6 yuan in total

Static binding (static::) implements late static binding (LSB) in PHP, allowing calling classes to be referenced in static contexts rather than defining classes. 1) The parsing process is performed at runtime, 2) Look up the call class in the inheritance relationship, 3) It may bring performance overhead.

What are the magic methods of PHP? PHP's magic methods include: 1.\_\_construct, used to initialize objects; 2.\_\_destruct, used to clean up resources; 3.\_\_call, handle non-existent method calls; 4.\_\_get, implement dynamic attribute access; 5.\_\_set, implement dynamic attribute settings. These methods are automatically called in certain situations, improving code flexibility and efficiency.
