javascript - Redirection problem when superagent crawls website

Question

Use superagent to crawl the web page. When encountering a redirection, I cannot get the response body after the redirection. How to solve the problem and how to obtain the redirected webpage

I want to get the res of the 501 page, but it jumps to page 37018, causing me to get the empty res of the 501 page.

Tested web pages that could not be crawled and found two situations

1. The response code when entering the webpage is 200 at first, and after a while it refreshes to 304

2. The response code redirects from 301 to 200 upon entry, and refreshes to 304 after a while

Tested the web pages that could be crawled and found two situations

1. The response code redirects from 301 to 200 upon entry, and refreshes to 304 after a while

2. The response code when entering the webpage is 200 at first, and after a while it refreshes to 304

Ah, let me wipe it, then there will be no difference. I don’t know if this has anything to do with me not being able to crawl the content, orz

Add more

The problem is not redirection, but my regular expression matching problem

Answer 1

Don’t you even read the official documentation?

• Following redirects

By default up to 5 redirects will be followed, however you may specify
this with the res.redirects(n) method:

request
  .get('/some.png')
  .redirects(2)
  .end(callback);