Nginx retry causes HTTP request to be executed repeatedly

1. Problem background

The web site uses Nginx as a reverse proxy, and two tomcats do load balancing.

The web site has a group text message module, and the administrator can fill in the sending content and mobile phone number list. Click Send, and the group text message operation will be executed in the background. After the sending is completed, the sending result will be returned.

I encountered the problem of repeated sending a long time ago. At first, I thought the administrator acted abnormally and clicked the send button repeatedly. Therefore, restrictions are placed on front-end submissions. When clicking send, set the button to disable, get the returned data after execution, and then set it to available to limit repeated submissions from the front end.

Today, a colleague suddenly used this module again and sent two batches of text messages (8000 and 1600) respectively. The front-end prompts that the operation failed, but his mobile phone received duplicate text messages.

2. Problem analysis

Firstly, we confirmed that there are no duplicates in the mobile phone number list, and also confirmed through browser debugging that the front-end did not initiate repeated requests.

By checking the logs of the two servers, it was found that repeated sending occurred, and the sending time points were not at the same time.

Through the logs, it can be estimated that the time to send a text message is 10 milliseconds.

I also checked the processing timeout log through the log, and found the pattern. Repeated requests are processed on different machines and the processing intervals are consistent.

A log

10:03:05,878  INFO TIMEOUT.info:252 - time: 75701ms, concurrent: 1, url: 
/push/sendmessage.10:07:24,705  INFO TIMEOUT.info:252 - time: 15148ms, concurrent: 0, url: /push/sendmessage.

B log

10:03:21,599  INFO TIMEOUT.info:252 - time: 76471ms, concurrent: 1, url: 
/push/sendmessage.10:07:39,718  INFO TIMEOUT.info:252 - time: 15113ms, concurrent: 0, url: /push/sendmessage.

Based on the above information, we guessed, It should be that the request execution time exceeds the limit, causing the front-end prompt to fail. Duplicate submissions should be caused by the retry mechanism.

So I contacted the operation and maintenance for confirmation. Received relevant reply:

Nginx can configure the timeout, assuming it is configured to 15s, and a request takes 16s to be processed and returned. Nginx routes to server A for processing. When A executes to the 15th second, it does not return normally. Nginx will resend it to service B for processing. When B executes to the 15th second, it does not return normally. The front end waits for 30s and finally returns failure. A and B received corresponding requests respectively and processed them internally.

After confirming the reason with the operation and maintenance classmates, you can search by yourself based on the relevant information.

nginx’s retry mechanism

The article also points to a solution that can access services through IP and bypass nginx.

In addition, our website has a download function. Before optimization, the execution time of each download was also very long, but there was no timeout problem. Analyzed the possible differences between GET and POST. The following link is also given above to determine the general idea.

http://serverfault.com/questions/528653/how-can-i-stop-nginx-from-retrying-put-or-post-requests-on-upstream-server-timeo

3. Summary of questions

Through online information reference, Nginx can set different timeout strategies for file upload, download, GET and POST requests.

In addition, for the SMS bulk sending business, there is actually a separate module that manages configuration through rules and performs offline task execution. The existing mass sending module is only for small-scale business.