Detailed explanation of the use of Python's Requests module

The Requests module is a module used for network access. In fact, there are many similar modules, such as urllib, urllib2, httplib, httplib2. They basically provide similar functions, so why can the Requests module stand out? You can open its official website and take a look. It is an http module for "human beings". So, how human is it? I believe that if you have used modules such as urllib before, you will find that it is indeed very user-friendly.

1. Import

After the download is completed, importing the module is very simple. The code is as follows:

import requests

2. Request URL

Here we list the most common The syntax for sending get or post requests.

1. Send a get request without parameters:

r=requests.get("http://php.cn/justTest")

Now, we get a response object r, we can use this object to get any information we want.

In the above example, the get request does not have any parameters. What if the request requires parameters?

2. Send a get request with parameters

payload = {'key1': 'value1', 'key2': 'value2'}
r = requests.get("http://php.cn/justTest", params=payload)

As we know from the above, our get parameters are passed as params keyword parameters.

We can print the specific URL requested to see if it is correct:

>>>print r.url
http://php.cn/justTest?key2=value2&key1=value1

You can see that the correct URL was indeed accessed.

You can also pass a list to a request parameter:

>>> payload = {'key1': 'value1', 'key2': ['value2', 'value3']}
>>> r = requests.get("http://php.cn/justTest", params=payload)
>>> print r.url
http://php.cn/justTest?key1=value1&key2=value2&key2=value3

The above is the basic form of the get request.

3. Send a post request

r = requests.post("http://php.cn/postTest", data = {"key":"value"})

As we know from the above, the post request parameters are passed as data keyword parameters.

The current data parameter passes a dictionary, we can also pass a json format data, as follows:

>>> import json
>>> import requests
>>> payload = {"key":"value"}
>>> r = requests.post("http://php.cn/postTest", data = json.dumps(payload))

Since sending json format data is too common, in the higher version of the Requests module , the keyword parameter json has been added. You can directly send json data to the post request without using the json module. See below:

>>> payload = {"key":"value"}
>>> r = requests.post("http://php.cn/postTest", json=payload)

What if we want to post a file? At this time, you need to use the files parameter:

>>> url = 'http://php.cn/postTest'
>>> files = {'file': open('report.xls', 'rb')}
>>> r = requests.post(url, files=files)
>>> r.text

We can also specify additional information such as the file name when posting the file:

>>> url = 'http://php.cn/postTest'
>>> files = {'file': ('report.xls', open('report.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})}
>>> r = requests.post(url, files=files)

tips: It is strongly recommended to use binary mode to open the file, because If opened in text file format, an error may occur due to the "Content-Length" header.

As you can see, it is easy to send requests using Requests!

3. Obtain the return information

Let’s take a look at how to obtain the return information after sending the request. Let’s continue to use the top example: In what encoding format is

>>> import requests
>>> r=requests.get('http://php.cn/justTest')
>>> r.text

r.text output?

>>> r.encoding
'utf-8'

It turned out to be output in utf-8 format. What if I want to change the output format of r.text?

>>> r.encoding = 'ISO-8859-1'

This changes the output format to "ISO-8859-1".

There is also an output statement called r.content, so what is the difference between this and r.text? r.content returns a byte stream, which can be used if we request an image address and want to save the image. Here is a code snippet as follows:

def saveImage( imgUrl,imgName ="default.jpg" ):
    r = requests.get(imgUrl, stream=True)
    image = r.content
    destDir="D:\"
    print("保存图片"+destDir+imgName+"\n")
    try:
        with open(destDir+imgName ,"wb") as jpg:
            jpg.write(image)     
            return
    except IOError:
        print("IO Error")
        return
    finally:
        jpg.close

The r.text just introduced returns String, then, if the response corresponding to the request is a json, can I directly get the data in json format? r.json() is prepared for this.

We can also get the original data returned by the server, just use r.raw.read(). However, if you really want to get the original return data, remember to add the "stream=True" option when requesting, such as:

r = requests.get('https://api.github.com/events', stream=True)。

We can also get the response status code:

>>> r = requests.get('http://php.cn/justTest')
>>> r.status_code
200

You can also use requests.codes.ok to refer to the return value of 200:

>>> r.status_code == requests.codes.ok
True

4. About headers

We can print out the response header:

>>> r= requests.get("http://php.cn/justTest")
>>> r.headers

｀r .headers｀ returns a dictionary, for example:

{
    'content-encoding': 'gzip',
    'transfer-encoding': 'chunked',
    'connection': 'close',
    'server': 'nginx/1.0.4',
    'x-runtime': '147ms',
    'etag': '"e1ca502697e5c9317743dc078f67693a"',
    'content-type': 'application/json'
}

We can use the following method to obtain part of the response headers for judgment:

r.headers['Content-Type']

or

r.headers.get('Content-Type')

If we want What should we do to get the request header (that is, the header information we send to the server)? Can be obtained directly using r.request.headers.

At the same time, we can also add custom headers (passed through headers keyword parameters) when requesting data:

>>> headers = {'user-agent': 'myagent'}
>>> r= requests.get("http://php.cn/justTest",headers=headers)

5. About Cookies

If a response If cookies are included, we can use the following method to get them:

>>> url = 'http://www.php.cn'
>>> r = requests.get(url)
>>> r.cookies['example_cookie_name']
'example_cookie_value'

We can also send our own cookies (using the cookies keyword parameters):

>>> url = 'http://php.cn/cookies'
>>> cookies={'cookies_are':'working'}
>>> r = requests.get(url, cookies=cookies)

6. About redirection

Sometimes when we request the URL, the server will automatically redirect our request. For example, github will redirect our http request to https request. We can use r.history to view redirects:

>>> r = requests.get('http://php.cn/')
>>> r.url
'http://pythontab.com/'
>>> r.history
[]

As you can see from the above example, we use the http protocol to access, but in r.url, the https protocol is printed. So what should I do if I insist that the server use the http protocol, which means that the server is prohibited from automatically redirecting? Use the allow_redirects parameter:

r = requests.get('http://php.cn', allow_redirects=False)

7. About the request time

We can use the timeout parameter to set the request timeout of the url (time unit is seconds):

requests.get('http://php.cn', timeout=1)

8 , About proxy

We can also specify a proxy in the program for http or https access (using proxies keyword parameters), as follows:

proxies = {
  "http": "http://10.10.1.10:3128",
  "https": "http://10.10.1.10:1080",
}
requests.get("http://php.cn", proxies=proxies)

九、关于session

我们有时候会有这样的情况，我们需要登录某个网站，然后才能请求相关url，这时就可以用到session了，我们可以先使用网站的登录api进行登录，然后得到session，最后就可以用这个session来请求其他url了：

s=requests.Session()
login_data={'form_email':'youremail@example.com','form_password':'yourpassword'}
s.post("http://php.cn/testLogin",login_data)
r = s.get('http://php.cn/notification/')
print r.text

其中，form_email和form_password是豆瓣登录框的相应元素的name值。

十、下载页面

使用Requests模块也可以下载网页，代码如下：

r=requests.get("http://www.php.cn")
with open("haha.html","wb") as html:
    html.write(r.content)
html.close()

The above is the detailed content of Detailed explanation of the use of Python's Requests module. For more information, please follow other related articles on the PHP Chinese website!