How to asynchronously generate excel files in php and save them to the server

Let me first talk about my current needs and difficulties:

• 1. I believe everyone is familiar with the function of exporting orders. The amount of data is large and the business is complex. Now the product needs to export a month's worth of data, about 20,000 pieces, and the query interface calls the API provided by Java.

• 2. Querying 2000 pieces of data through this interface takes 30+ seconds, and the server’s timeout is 30 seconds, so timeout is inevitable. 1w pieces of data are sent every 1000 pieces. A single curl request takes 60-78 seconds. A curl_multi request takes 45.78-52 seconds.

• 3. The business logic has not been completed at this time. The user's information needs to be queried in the database based on the uid in the returned results. Some fields also need to be judged. Therefore, you need to wait for the rest results and process the logic.

• 4. The processed data results are assembled into excel cells after traversal, which takes up a lot of memory.

Now let me talk about my own thoughts:

• 1. The client initiates the "export" action, the service sends a query operation, and generates an excel file and saves it to the server.

• 2. After the client sends an action, it is best to return immediately, without waiting for the result to be fed back to the customer, and directly tell the customer to download again half an hour later.

• 3. Download the excel file generated directly from the server.

My confusion:

How to make the client initiate a request and return it immediately, and also require the server to do business logic, process queries, field assembly and other processes? Please give me some advice, thank you!

Additional point:
I originally wanted to use cron job, but there are more and more things, and the server is not very generous. There are many services deployed on it. These things are used internally, and the efficiency is not very high.

Because many things displayed on the app are scheduled tasks, such as product refinement, etc., there are too many, so I don’t want to use cron job at the moment. Moreover, spring tasks are used in java code to perform many tasks.

Reply content:

Let me first talk about my current needs and difficulties:

• 1. I believe everyone is familiar with the function of exporting orders. The amount of data is large and the business is complex. Now the product needs to export a month's worth of data, about 20,000 pieces, and the query interface calls the API provided by Java.

• 2. Querying 2000 pieces of data through this interface takes 30+ seconds, and the server’s timeout is 30 seconds, so timeout is inevitable. 1w pieces of data are sent every 1000 pieces. A single curl request takes 60-78 seconds. A curl_multi request takes 45.78-52 seconds.

• 3. The business logic has not been completed at this time. The user's information needs to be queried in the database based on uid in the returned results. Some fields also need to be judged. Therefore, you need to wait for the rest results and process the logic.

• 4. The processed data results are assembled into excel cells after traversal, which takes up a lot of memory.

Now let me talk about my own thoughts:

• 1. The client initiates the "export" action, the service sends a query operation, and generates an excel file and saves it to the server.

• 2. After the client sends an action, it is best to return immediately, without waiting for the results to be fed back to the customer, and directly tell the customer to download again half an hour later.

• 3. Download the excel file generated directly from the server.

My confusion:

How to make the client initiate a request and return it immediately, and also require the server to do business logic, process queries, field assembly and other processes? Please give me some advice, thank you!

Additional point:
I originally wanted to use cron job, but there are more and more things, and the server is not very generous. There are many services deployed on it. These things are used internally, and the efficiency is not very high.

Because many things displayed on the app are scheduled tasks, such as product refinement, etc., there are too many, so I don’t want to use cron job at the moment. Moreover, spring tasks are used in java code to perform many tasks.

Two types, one is to use a script to run. Another option is to return directly after receiving the front-end request or let it time out. PHP can continue to execute the display setting script set_time_limit(0);. For php-fpm, fastcgi_finish_request() can be used to continue execution after returning. Apache can set ignore_user_abort( );

However, the latter method, php-fpm, will block php from receiving requests, which may cause normal users to be unable to access the website, because it is randomly assigned, and if nginx is connected to that long-term process, it will be blocked.
It is recommended to use php script to execute, you can use queue, or you can use shell to call php script to execute

Can be exported through queue
can meet your needs, the picture below is my export

Or you can choose something simpler and use Linux to schedule task execution

Submit an export request at the front desk. You add the export request to the database task table and return information to the client that the export is being performed. Then the Linux scheduled task queries the task table to find the new task and exports it. After the export is completed, the current task is modified. Now that it’s done, this shouldn’t be particularly complicated

Classmate, have you heard of cron job?

You can use distributed task queues
such as gearman, rabbitmq for asynchronous processing

Why not use the nodejs implementation that is inherently asynchronous?