The large file data export I have experienced (background execution, automatically generated)-PHP Tutorial-php.cn

Home

Backend Development

PHP Tutorial

The large file data export I have experienced (background execution, automatically generated)

WBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWBOYWB

Jul 30, 2016 pm 01:31 PM

excel http nbsp

1. Foreword

I will record the function of exporting statistical information in excel format in the background that I have done before. Recently, my colleagues asked me about it, but I forgot the specific details for a while, so I will record it;

Everyone As you know, the function of exporting data in excel is almost a necessary function in the background. Generally, after clicking, the file is generated and then automatically downloaded.

If the amount of data is small, the request can be completed in one go and downloaded locally;

However, if the amount of data is particularly large, the page must wait until the writing to excel is successful.

This will affect the background users' inability to operate other pages. For this reason, the excel export has been made The following functions are optimized:

excel export is divided into two parts: generating excel files and downloading excel files

excel file generation is executed in the background of the program, the front end does not have to wait, other background operations can be performed

Add a download file page to display the progress of excel file generation. After completion, the generated excel file can be downloaded. After the file is generated, click Download to download the corresponding file. 2. Generate excel file

There are many ways to generate excel files, I won’t record them one by one, just record this method;

The html format of table is used here, and the corresponding excel statement

(I vaguely remember other The method seems to be garbled when opened with office07. I tried to use the csv format file later, but it was still garbled, so I used the table format)

The beginning of the file:

<spanmicrosoft yahei><span> 1</span><span>$struserdata</span> = <<<<span>Eof
</span><span> 2</span>         <html xmlns:o="urn:schemas-microsoft-com:office:office"
<span> 3</span>         xmlns:x="urn:schemas-microsoft-com:office:excel"
<span> 4</span>         xmlns="http://www.w3.org/TR/REC-html40">
<span> 5</span><span> 6</span>         <!DOCTYPE html <span>PUBLIC</span> "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<span> 7</span>         <html>
<span> 8</span>         <head>
<span> 9</span>             <meta http-equiv="Content-type" c />
<span>10</span>         <style>
<span>11</span>         </style>
<span>12</span>         </head>
<span>13</span>         <body>
<span>14</span>         <div align=center x:publishsource="Excel">
<span>15</span><span>16</span>         <table x:str border=1 cellpadding=0 cellspacing=0 width=100%>
<span>17</span> Eof;</spanmicrosoft>

Copy after login

View Code

End of file:

<spanmicrosoft yahei><span>1</span><span>$struserdata</span> = <<<<span>Eof
</span><span>2</span>         </table>
<span>3</span>         </div>
<span>4</span>         </body>
<span>5</span>         </html>
<span>6</span> Eof;</spanmicrosoft>

Copy after login

View Code

Of course, there are some tr td tags in the middle of the file.

3. Let the program execute in the background

Scenario:

After the user clicks to generate excel, it jumps to the download page. The program is executed in the background. The user does not have to wait for the generation to be completed and can perform other operations. ;

You can see the progress of file generation and whether it can be downloaded on the download page
Idea:
Click to generate excel and display the download page ---> show_download method
Generate excel - --> create_excel method

The show_download method calls the create_excel method, and in the show_download method, I use the command line to execute the program,

Use the php command line to pass the parameters to create_excel Method
<spanmicrosoft yahei>1// $cmd = "/usr/bin/php /home/xxx/xxx.php " . $strjoin . " >/dev/null & ";
2 // $a=exec($cmd, $out, $returndata);345$command = "/usr/bin/php ".STATISTIC_EXPORT_SCRIPT_DIR."xxx.php " . "'" .$strjoin ."'". " " . $uid . " ". $action ." & ";
6$process = proc_open($command, array(),$pipes); 
7$var = proc_get_status($process); 
8proc_close($process);
9$pid = intval($var['pid'])+1;</spanmicrosoft>
Copy after login

And in the create_excel method:

You need to fill in the following code:

<spanmicrosoft yahei><span>1</span><span>set_time_limit</span>(0); <span>//</span><span>取消脚本运行时间的超时上限</span><span>2</span><span>3</span><span>ignore_user_abort</span>(<span>TRUE</span>); <span>//</span><span>后台运行,不受用户关闭浏览器的影响</span></spanmicrosoft>

Copy after login

Call the relevant api to get the data:

<spanmicrosoft yahei><span>1</span><span>$statistic</span> = <span>call_user_func</span>(<span>array</span>('shellscript','get_result'),<span>$url</span>,<span>$params</span><span>);
</span><span>2</span><span>if</span>(!<span>is_object</span>(<span>$statistic</span>) || !<span>isset</span>(<span>$statistic</span>->data-><span>items)){
</span><span>3</span><span>usleep</span>(400000);<span>//</span><span>停止400毫秒</span><span>4</span><span>$statistic</span> = <span>call_user_func</span>(<span>array</span>('shellscript','get_result'),<span>$url</span>,<span>$params</span><span>);
</span><span>5</span> }</spanmicrosoft>

Copy after login

4. Display file generation progress

But how to display the corresponding file generation progress, and how to know whether the file has been generated?

Here, the method I use is that when writing data files data.xsl, each data file generates a corresponding file progress file, temporarily called flag_data.xsl;

Ideas:

第一次请求api的时候，根据返回的total总数，以及pagesize，确定要请求的次数count;

这样便可知道要请求api的次数（分页请求api），在写入数据文件的同时，同时写入进度文件flag_data.xsl;
数据格式大约是(以逗号分割)
1,5
2,5
...

然后显示文件进度的时候，读取进度文件，这样变可知道数据文件大体的进度

前端js处理时，几秒读取一次相应的方法（如果都100%进度，可停止请求方法），从而实现动态查看文件的生成进度

查看文件的进度方法：

<spanmicrosoft yahei><span> 1</span><span>public</span><span>function</span><span> execscript_process(){
</span><span> 2</span><span>$this</span>->load->library('smarty'<span>);
</span><span> 3</span><span>$file_arr_str</span> = <span>array</span><span>();
</span><span> 4</span><span>$file_arr_process</span> = <span>array</span><span>();
</span><span> 5</span><span>$file_arr_name</span> = <span>array</span><span>();
</span><span> 6</span><span>$file_arr</span> = <span>array</span><span>();
</span><span> 7</span><span>$refresh_flag</span> = 'yes'<span>;
</span><span> 8</span><span>$uid</span> = <span>$_REQUEST</span>['uid'<span>];
</span><span> 9</span><span>$url_dir</span> = STATISTIC_EXPORT_FILE_DIR.<span>$uid</span> .'/';<span>//</span><span>@todo</span><span>10</span><span>if</span>(!<span>is_dir</span>(<span>$url_dir</span><span>)){
</span><span>11</span>             @<span>mkdir</span>(<span>$url_dir</span>,0777<span>);
</span><span>12</span><span>        }
</span><span>13</span><span>$files</span> = <span>scandir</span>(<span>$url_dir</span><span>);
</span><span>14</span><span>15</span><span>if</span>(!<span>empty</span>(<span>$files</span><span>)){
</span><span>16</span><span>foreach</span> (<span>$files</span><span>as</span><span>$key</span> => <span>$value</span><span>) {
</span><span>17</span><span>if</span>(<span>$value</span>!='.' && <span>$value</span>!='..'<span>){
</span><span>18</span><span>if</span>(<span>substr</span>(<span>$value</span>, 0 , 5)=="flag_"<span>){
</span><span>19</span><span>$file_size</span> = <span>filesize</span>(<span>$url_dir</span> . <span>$value</span><span>);
</span><span>20</span><span>if</span>(!<span>empty</span>(<span>$file_size</span><span>)){
</span><span>21</span><span>$fhandle</span> = <span>fopen</span>(<span>$url_dir</span> . <span>$value</span>, 'rb+'<span>);
</span><span>22</span><span>fseek</span>(<span>$fhandle</span>, -1,<span> SEEK_END);
</span><span>23</span><span>$fstr</span> = ''<span>;
</span><span>24</span><span>while</span>((<span>$c</span> = <span>fgetc</span>(<span>$fhandle</span>)) !== <span>false</span><span>) {
</span><span>25</span><span>if</span>(<span>$c</span> == "\n" && <span>$fstr</span>) <span>break</span><span>;
</span><span>26</span><span>$fstr</span> = <span>$c</span> . <span>$fstr</span><span>;
</span><span>27</span><span>fseek</span>(<span>$fhandle</span>, -2,<span> SEEK_CUR);
</span><span>28</span><span>                            }
</span><span>29</span><span>fclose</span>(<span>$fhandle</span><span>);
</span><span>30</span><span>$fstr</span> = <span>trim</span>(<span>$fstr</span><span>);
</span><span>31</span><span>$fstr_arr_str</span> = <span>explode</span>(',', <span>$fstr</span><span>);
</span><span>32</span><span>$file_arr_process</span>[] = 100 * <span>number_format</span>(<span>$fstr_arr_str</span>[0]/<span>$fstr_arr_str</span>[1],2).'%'<span>;
</span><span>33</span><span>$file_arr_name</span>[] = <span>substr</span>(<span>$value</span>,5<span>);
</span><span>34</span><span>                         }
</span><span>35</span><span>                    }
</span><span>36</span><span>                }
</span><span>37</span><span>            }
</span><span>38</span><span>39</span><span>foreach</span> (<span>$file_arr_process</span><span>as</span><span>$key</span> => <span>$value</span><span>) {
</span><span>40</span><span>if</span>(<span>$value</span> != '100%'<span>){
</span><span>41</span><span>$refresh_flag</span> = 'no'<span>;
</span><span>42</span><span>break</span><span>;
</span><span>43</span><span>                }
</span><span>44</span><span>            }
</span><span>45</span><span>        }
</span><span>46</span><span>47</span><span>$file_arr</span> = <span>array</span><span>(
</span><span>48</span>             'process' => <span>$file_arr_process</span>,
<span>49</span>             'name' => <span>$file_arr_name</span>,
<span>50</span>             'refresh_flag' => <span>$refresh_flag</span><span>51</span><span>            );
</span><span>52</span><span>$file_arr_json</span> = json_encode(<span>$file_arr</span><span>);
</span><span>53</span><span>echo</span><span>$file_arr_json</span><span>;
</span><span>54</span>     }</spanmicrosoft>

Copy after login

View Code

五、下载文件

文件的下载就好说了，既然已经都生成成功，下载的方法如下：

<spanmicrosoft yahei><span> 1</span><span>public</span><span>function</span><span> execscript_download(){
</span><span> 2</span><span>$filename</span> = <span>$_REQUEST</span>['filename'<span>];
</span><span> 3</span><span>$uid</span> = <span>$_REQUEST</span>['uid'<span>];
</span><span> 4</span><span>$file_dir</span> = STATISTIC_EXPORT_FILE_DIR.<span>$uid</span>.'/'.<span>$filename</span><span>;
</span><span> 5</span><span>if</span> (!<span>file_exists</span>(<span>$file_dir</span><span>)){
</span><span> 6</span><span>header</span>("Content-type: text/html; charset=utf-8"<span>);
</span><span> 7</span><span>echo</span> "File not found!"<span>;
</span><span> 8</span><span>exit</span><span>; 
</span><span> 9</span>         } <span>else</span><span> {
</span><span>10</span><span>ini_set</span>("memory_limit","500M"<span>); 
</span><span>11</span><span>header</span>('Content-Description: File Transfer'<span>);
</span><span>12</span><span>header</span>('Content-Type: application/octet-stream'<span>);
</span><span>13</span><span>header</span>('Content-Disposition: attachment; filename='.<span>basename</span>(<span>$file_dir</span><span>));
</span><span>14</span><span>header</span>('Content-Transfer-Encoding: binary'<span>);
</span><span>15</span><span>header</span>('Expires: ' . <span>gmdate</span>('D, d M Y H:i:s') . ' GMT'<span>);
</span><span>16</span><span>header</span>('Cache-Control: must-revalidate,post-check=0, pre-check=0'<span>);
</span><span>17</span><span>header</span>('Pragma: public'<span>);
</span><span>18</span><span>header</span>('Content-Length: ' . <span>filesize</span>(<span>$file_dir</span><span>));
</span><span>19</span><span>readfile</span>(<span>$file_dir</span><span>);
</span><span>20</span><span>        }
</span><span>21</span><span>22</span>     }</spanmicrosoft>

Copy after login

六、上线后出现的问题

本地本来已经测试完毕，可上线后，却出现了奇怪的问题；

现象描述：

当在后台点击生成文件，跳转到下载页的时候，因为下载页是显示文件进度的页面，
竟然出现有时候有刚刚点击的文件进度，有时候没有，就感觉没有生成相应的文件一样；

解决方法：

因为数据文件和进度文件都是生成在程序的某个文件夹file中，所以读取的时候都是读取的文件夹下的文件，从而判断显示进度；

后面才知道，由于后台程序有两台服务器，导致读取以及下载的时候找不到相应的文件夹，两个服务器相应的文件夹弄个共享目录就可以了

七、相应的后续优化

由于下载的文件多了，导致文件夹下的文件越来越多，而原来生成的文件是没有价值的，所以加了个定期删除文件的功能，只保留近七天的文件

当然可以用crontab，只不过我比较懒，是在点击生成文件的时候，判断了一下文件夹中的过期文件，从而删除

<spanmicrosoft yahei><span> 1</span><span>public</span><span>function</span><span> execscript_process_show(){
</span><span> 2</span><span>$this</span>->load->library('smarty'<span>);
</span><span> 3</span><span>$uid</span> = <span>$_REQUEST</span>['uid'<span>];
</span><span> 4</span><span>$url_dir</span> = STATISTIC_EXPORT_FILE_DIR.<span>$uid</span> .'/';<span>//</span><span>@todo</span><span> 5</span><span>if</span>(!<span>is_dir</span>(<span>$url_dir</span><span>)){
</span><span> 6</span>             @<span>mkdir</span>(<span>$url_dir</span>,0777<span>);
</span><span> 7</span><span>        }        
</span><span> 8</span><span>$files</span> = <span>scandir</span>(<span>$url_dir</span><span>);
</span><span> 9</span><span>if</span>(!<span>empty</span>(<span>$files</span><span>)){
</span><span>10</span><span>foreach</span> (<span>$files</span><span>as</span><span>$key</span> => <span>$value</span><span>) {
</span><span>11</span><span>if</span>(<span>$value</span>!='.' && <span>$value</span>!='..'<span>){
</span><span>12</span><span>foreach</span> (<span>$files</span><span>as</span><span>$key</span> => <span>$value</span><span>) {
</span><span>13</span><span>if</span>(<span>$value</span>!='.' && <span>$value</span>!='..'<span>){
</span><span>14</span><span>if</span>(<span>substr</span>(<span>$value</span>, 0 , 5)!="flag_"<span>){
</span><span>15</span><span>$filenamedate</span> = <span>substr</span>(<span>$value</span>, 0,10<span>);
</span><span>16</span><span>$today</span> = <span>date</span>('Y-m-d',<span>time</span><span>());
</span><span>17</span><span>$filenamedate</span> = <span>date</span>('Y-m-d',<span>strtotime</span>(<span>$filenamedate</span>)+(STATISTIC_FILE_EXPIRE_DAY-1)*24*3600<span>);
</span><span>18</span><span>if</span>(<span>$today</span>><span>$filenamedate</span>){<span>//</span><span>文件过期</span><span>19</span>                                     @<span>unlink</span>(<span>$url_dir</span> . <span>$value</span><span>);
</span><span>20</span>                                     @<span>unlink</span>(<span>$url_dir</span> . 'flag_' . <span>$value</span><span>);
</span><span>21</span><span>                                }
</span><span>22</span><span>                            }
</span><span>23</span><span>                        }
</span><span>24</span><span>                    }                    
</span><span>25</span><span>                }
</span><span>26</span><span>            }
</span><span>27</span><span>        }
</span><span>28</span><span>29</span><span>$this</span>->smarty->assign('uid',<span>$uid</span><span>);
</span><span>30</span><span>$this</span>->smarty->display('interact/statistic/execscript.tpl'<span>);
</span><span>31</span>     }</spanmicrosoft>

Copy after login

八、后记

大文件的导出大体就是这个样子，欢迎大家吐槽，共同交流；

当时在用命令行执行方法的时候，也参考了一下相应的资料，记录一下；

<spanmicrosoft yahei>http://blog.csdn.net/yysdsyl/article/details/4636457

http://www.codesky.net/article/201202/163385.html

http://www.cnblogs.com/zdz8207/p/3765567.html

http://blog.163.com/mojian20040228@126/blog/static/4112219320097300922992/

http://php.net/manual/en/features.commandline.php

http://blog.csdn.net/yangjun07167/article/details/5603425

http://blog.csdn.net/yunsongice/article/details/5445448

http://www.cppblog.com/amazon/archive/2011/12/01/161281.aspx

http://blog.51yip.com/tag/proc_open

http://www.justwinit.cn/post/1418/

http://limboy.me/tech/2010/12/05/php-async.html</spanmicrosoft>

Copy after login

以上就介绍了我所经历的大文件数据导出（后台执行,自动生成），包括了方面的内容，希望对PHP教程有兴趣的朋友有所帮助。

Statement of this Website

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress images for free

Clothoff.io

AI clothes remover

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)

3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

R.E.P.O. Best Graphic Settings

3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Assassin's Creed Shadows: Seashell Riddle Solution

1 weeks ago By DDD

R.E.P.O. How to Fix Audio if You Can't Hear Anyone

3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

WWE 2K25: How To Unlock Everything In MyRise

3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Where is the login entrance for gmail email?

7454

CakePHP Tutorial

1375

What is the format of the account name of steam

win11 activation key permanent

nyt connections hints and answers

Related knowledge

What should I do if the frame line disappears when printing in Excel? Mar 21, 2024 am 09:50 AM

If when opening a file that needs to be printed, we will find that the table frame line has disappeared for some reason in the print preview. When encountering such a situation, we must deal with it in time. If this also appears in your print file If you have questions like this, then join the editor to learn the following course: What should I do if the frame line disappears when printing a table in Excel? 1. Open a file that needs to be printed, as shown in the figure below. 2. Select all required content areas, as shown in the figure below. 3. Right-click the mouse and select the "Format Cells" option, as shown in the figure below. 4. Click the “Border” option at the top of the window, as shown in the figure below. 5. Select the thin solid line pattern in the line style on the left, as shown in the figure below. 6. Select "Outer Border"

How to filter more than 3 keywords at the same time in excel Mar 21, 2024 pm 03:16 PM

Excel is often used to process data in daily office work, and it is often necessary to use the "filter" function. When we choose to perform "filtering" in Excel, we can only filter up to two conditions for the same column. So, do you know how to filter more than 3 keywords at the same time in Excel? Next, let me demonstrate it to you. The first method is to gradually add the conditions to the filter. If you want to filter out three qualifying details at the same time, you first need to filter out one of them step by step. At the beginning, you can first filter out employees with the surname "Wang" based on the conditions. Then click [OK], and then check [Add current selection to filter] in the filter results. The steps are as follows. Similarly, perform filtering separately again

How to change excel table compatibility mode to normal mode Mar 20, 2024 pm 08:01 PM

In our daily work and study, we copy Excel files from others, open them to add content or re-edit them, and then save them. Sometimes a compatibility check dialog box will appear, which is very troublesome. I don’t know Excel software. , can it be changed to normal mode? So below, the editor will bring you detailed steps to solve this problem, let us learn together. Finally, be sure to remember to save it. 1. Open a worksheet and display an additional compatibility mode in the name of the worksheet, as shown in the figure. 2. In this worksheet, after modifying the content and saving it, the dialog box of the compatibility checker always pops up. It is very troublesome to see this page, as shown in the figure. 3. Click the Office button, click Save As, and then

How to set superscript in excel Mar 20, 2024 pm 04:30 PM

When processing data, sometimes we encounter data that contains various symbols such as multiples, temperatures, etc. Do you know how to set superscripts in Excel? When we use Excel to process data, if we do not set superscripts, it will make it more troublesome to enter a lot of our data. Today, the editor will bring you the specific setting method of excel superscript. 1. First, let us open the Microsoft Office Excel document on the desktop and select the text that needs to be modified into superscript, as shown in the figure. 2. Then, right-click and select the "Format Cells" option in the menu that appears after clicking, as shown in the figure. 3. Next, in the “Format Cells” dialog box that pops up automatically

How to use the iif function in excel Mar 20, 2024 pm 06:10 PM

Most users use Excel to process table data. In fact, Excel also has a VBA program. Apart from experts, not many users have used this function. The iif function is often used when writing in VBA. It is actually the same as if The functions of the functions are similar. Let me introduce to you the usage of the iif function. There are iif functions in SQL statements and VBA code in Excel. The iif function is similar to the IF function in the excel worksheet. It performs true and false value judgment and returns different results based on the logically calculated true and false values. IF function usage is (condition, yes, no). IF statement and IIF function in VBA. The former IF statement is a control statement that can execute different statements according to conditions. The latter

Where to set excel reading mode Mar 21, 2024 am 08:40 AM

In the study of software, we are accustomed to using excel, not only because it is convenient, but also because it can meet a variety of formats needed in actual work, and excel is very flexible to use, and there is a mode that is convenient for reading. Today I brought For everyone: where to set the excel reading mode. 1. Turn on the computer, then open the Excel application and find the target data. 2. There are two ways to set the reading mode in Excel. The first one: In Excel, there are a large number of convenient processing methods distributed in the Excel layout. In the lower right corner of Excel, there is a shortcut to set the reading mode. Find the pattern of the cross mark and click it to enter the reading mode. There is a small three-dimensional mark on the right side of the cross mark.

How to insert excel icons into PPT slides Mar 26, 2024 pm 05:40 PM

1. Open the PPT and turn the page to the page where you need to insert the excel icon. Click the Insert tab. 2. Click [Object]. 3. The following dialog box will pop up. 4. Click [Create from file] and click [Browse]. 5. Select the excel table to be inserted. 6. Click OK and the following page will pop up. 7. Check [Show as icon]. 8. Click OK.

How to read excel data in html Mar 27, 2024 pm 05:11 PM

How to read excel data in html: 1. Use JavaScript library to read Excel data; 2. Use server-side programming language to read Excel data.

See all articles