Share an example of using Python crawler to simulate Zhihu login

During the crawling process, some pages are prohibited from being crawled before logging in. At this time, you need to simulate the login. The following article mainly introduces how to use the Python crawler to simulate Zhihu login. The method tutorial is very detailed in the article. Friends in need can refer to it. Let’s take a look together.

Preface

Everyone who often writes crawlers knows that some pages are prohibited from crawling before logging in, such as Zhihu’s The topic page requires users to log in to access, and "login" is inseparable from the Cookie technology in HTTP.

Login Principle

The principle of Cookie is very simple, because HTTP is a stateless protocol, so in order to use the stateless HTTP protocol On top of that, the session state is maintained to let the server know which client it is currently dealing with. Cookie technology appeared. Cookie is equivalent to an identifier assigned by the server to the client.

• When the browser initiates an HTTP request for the first time, it does not carry any Cookie information

• The server The HTTP response, along with a Cookie information, is returned to the browser together

• The browser will send the Cookie information returned by the server to the server together with the second request

• The server receives the HTTP request and finds that there is a Cookie field in the request header, so it knows that it has dealt with this user before.
  

Practical application

Anyone who has used Zhihu knows that as long as you provide the user name and password and verification code to log in. Of course, this is just what we see. The technical details hidden behind need to be discovered with the help of a browser. Now let’s use Chrome to see what happens after we fill out the form?

(If you are already logged in, log out first) First enter the Zhihu login page www.zhihu.com/#signin and open the Chrome developer toolbar (press F12 ) First try to enter an incorrect verification code and observe how the browser sends the request.

Several key information can be found from the browser request

• The login URL address is https://www. zhihu.com/login/email

• There are four form data required for login: username (email), password (password), verification code (captcha), and _xsrf.

• The URL address to obtain the verification code is https://www.zhihu.com/captcha.gif?r=1490690391695&type=login
  

_xsrfWhat is it? If you are very familiar with CSRF (cross-site request forgery) attacks, then you must know its role. xsrf is a string of pseudo-random numbers, which is used to prevent cross-site request forgery. It usually exists in the form form tag of the web page. To confirm this, you can search for "xsrf" on the page. Sure enough, _xsrf is in a hidden input tag

After clarifying how to obtain the data required for browser login, you can now start writing code to simulate browser login using Python. The two third-party libraries relied on when logging in are requests and BeautifulSoup. FirstInstall

pip install beautifulsoup4==4.5.3
pip install requests==2.13.0

http.cookiejar module can be used to automatically process HTTP Cookies. The LWPCookieJar object is the encapsulation of cookies. Supports saving cookies to files and loading them from files.

The session object provides cookie persistence and connection pooling functions. Requests can be sent through the session object.

First load the cookie information from the cookies.txt file, because there is no cookie for the first run. A LoadError exception will occur.

from http import cookiejar
session = requests.session()
session.cookies = cookiejar.LWPCookieJar(filename='cookies.txt')
try:
 session.cookies.load(ignore_discard=True)
except LoadError:
 print("load cookies failed")

Get xsrf

The tag where xsrf is located has been found earlier, and the value can be obtained very conveniently by using the find method of BeatifulSoup

def get_xsrf():
 response = session.get("https://www.zhihu.com", headers=headers)
 soup = BeautifulSoup(response.content, "html.parser")
 xsrf = soup.find('input', attrs={"name": "_xsrf"}).get("value")
 return xsrf

Get the verification code

The verification code is returned through the /captcha.gif interface, here we Download and save the verification code image to the current directory for manual identification. Of course, you can use a third-party support library to automatically identify it, such as pytesser.

def get_captcha():
 """
 把验证码图片保存到当前目录，手动识别验证码
 :return:
 """
 t = str(int(time.time() * 1000))
 captcha_url = 'https://www.zhihu.com/captcha.gif?r=' + t + "&type=login"
 r = session.get(captcha_url, headers=headers)
 with open('captcha.jpg', 'wb') as f:
  f.write(r.content)
 captcha = input("验证码：")
 return captcha

Login

一切参数准备就绪之后，就可以请求登录接口了。

def login(email, password):
 login_url = 'www.zhihu.com/login/email'
 data = {
  'email': email,
  'password': password,
  '_xsrf': get_xsrf(),
  "captcha": get_captcha(),
  'remember_me': 'true'}
 response = session.post(login_url, data=data, headers=headers)
 login_code = response.json()
 print(login_code['msg'])
 for i in session.cookies:
  print(i)
 session.cookies.save()

请求成功后，session 会自动把 服务端的返回的cookie 信息填充到 session.cookies 对象中，下次请求时，客户端就可以自动携带这些cookie去访问那些需要登录的页面了。

auto_login.py 示例代码

# encoding: utf-8
# !/usr/bin/env python
"""
作者：liuzhijun
"""
import time
from http import cookiejar

import requests
from bs4 import BeautifulSoup

headers = {
  "Host": "www.zhihu.com",
  "Referer": "www.zhihu.com/",
  'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87'
}

# 使用登录cookie信息
session = requests.session()
session.cookies = cookiejar.LWPCookieJar(filename='cookies.txt')
try:
  print(session.cookies)
  session.cookies.load(ignore_discard=True)

except:
  print("还没有cookie信息")


def get_xsrf():
  response = session.get("www.zhihu.com", headers=headers)
  soup = BeautifulSoup(response.content, "html.parser")
  xsrf = soup.find('input', attrs={"name": "_xsrf"}).get("value")
  return xsrf


def get_captcha():
  """
  把验证码图片保存到当前目录，手动识别验证码
  :return:
  """
  t = str(int(time.time() * 1000))
  captcha_url = 'www.zhihu.com/captcha.gif?r=' + t + "&type=login"
  r = session.get(captcha_url, headers=headers)
  with open('captcha.jpg', 'wb') as f:
    f.write(r.content)
  captcha = input("验证码：")
  return captcha


def login(email, password):
  login_url = 'www.zhihu.com/login/email'
  data = {
    'email': email,
    'password': password,
    '_xsrf': get_xsrf(),
    "captcha": get_captcha(),
    'remember_me': 'true'}
  response = session.post(login_url, data=data, headers=headers)
  login_code = response.json()
  print(login_code['msg'])
  for i in session.cookies:
    print(i)
  session.cookies.save()


if name == 'main':
  email = "xxxx"
  password = "xxxxx"
  login(email, password)

【相关推荐】

1. python爬虫入门（4）--详解HTML文本的解析库BeautifulSoup

2. python爬虫入门（3）--利用requests构建知乎API

3. python爬虫入门（2）--HTTP库requests

4.  python爬虫入门（1）--快速理解HTTP协议

5. python爬虫入门（5）--正则表达式实例教程

The above is the detailed content of Share an example of using Python crawler to simulate Zhihu login. For more information, please follow other related articles on the PHP Chinese website!