nginx about proxy_pass retry issues

Question

Dear masters, my nginx will adjust tomcat through proxy_pass, but tomcat will intermittently convulse (fullgc), causing a timeout, about 7-8 seconds

I thought of a way to set a timeout when requesting for the first time, such as 1s. If it times out, just request it again (change a machine)

Then I have the following nginx configuration. (The following is using flask's sleep to simulate tomcat's fullgc)

    upstream up {
        server 127.0.0.1:8088;
        server 127.0.0.1:8089;
    }

    server {
        listen       8087;
        server_name  localhost;
        access_log  logs/host.access.log  main;

        location / {
            proxy_connect_timeout 1s;
            proxy_send_timeout 1s;
            proxy_read_timeout   1s;

            proxy_next_upstream_timeout 1s;
            proxy_next_upstream_tries 1;
            send_timeout 1s;
#            proxy_next_upstream timeout error;
            proxy_pass      http://up;
        }
    }

The services of 8088 and 8089 are a mock service written by me
8088 will directly sleep for 2 seconds, causing nginx to timeout
8089 will directly return a "hello world",200

My expectation is
When I use the command line curl to repeatedly request 8087, "hello world" will always be returned. Although some requests are faster and some are slower (more than 1s, because they are re-executed after a timeout request) ask)

But the result is not like this

When the request exceeds 1s, 8087 will directly return a 504 error

By viewing nginx background log:
Three timeout records were printed continuously:

But 8088 (the sleep service) only received one request (some minor exceptions, written in flask)

Then I used wireshark to capture packets and found out

Can anyone help me?
Is there something wrong with my configuration??

There may be some irrelevant information provided here. If it is not useful, you can ignore it~

If there is any other good solution to the tomcat fullgc timeout problem, I would also be very grateful~

Answer 1

You asked for one try, then nginx tries that once.

try and retry are different. You try proxy_next_upstream_tries 2;. In addition, you seem to have misunderstood the meaning of proxy_next_upstream_timeout. Remove that too.