How to automatically obtain and update cookies

This time I will bring you the method of automatic acquisition and automatic update of cookie, what are the precautions for automatic acquisition and automatic update of cookies, the following is a practical case, let's come together take a look.

A lot of information on social networking sites requires logging in to get it. Take Weibo as an example. Without logging in, you can only see the top ten Weibo posts of big Vs. To stay logged instatus, cookies must be used. Take logging in to www.weibo.cn as an example:

Analyzing the request return of the console's Headers, you will see that weibo.cn has several sets of returned cookies.

Implementation steps:

1. Use selenium to automatically log in to obtain cookies and save them to files;

2. Read the cookie and compare the validity period of the cookie. If it expires, perform step 1 again;

3. When requesting other web pages, fill in the cookie to maintain the login status.

1. Get cookies online

Use selenium PhantomJS to simulate browser login and obtain cookies;

There are usually multiple cookies, and the cookies are stored one by one in files with the .weibo suffix.

def get_cookie_from_network():
 from selenium import webdriver
 url_login = 'http://login.weibo.cn/login/' 
 driver = webdriver.PhantomJS()
 driver.get(url_login)
 driver.find_element_by_xpath('//input[@type="text"]').send_keys('your_weibo_accout') # 改成你的微博账号
 driver.find_element_by_xpath('//input[@type="password"]').send_keys('your_weibo_password') # 改成你的微博密码
 driver.find_element_by_xpath('//input[@type="submit"]').click() # 点击登录
 # 获得 cookie信息
 cookie_list = driver.get_cookies()
 print cookie_list
 cookie_dict = {}
 for cookie in cookie_list:
  #写入文件
  f = open(cookie['name']+'.weibo','w')
  pickle.dump(cookie, f)
  f.close()
  if cookie.has_key('name') and cookie.has_key('value'):
   cookie_dict[cookie['name']] = cookie['value']
 return cookie_dict

2. Get cookie

from the file Traverse files ending with .weibo, that is, cookie files, from the current directory. Use pickle to unpack it into a dict, compare the expiry value with the current time, and return empty if it expires;

def get_cookie_from_cache():
 cookie_dict = {}
 for parent, dirnames, filenames in os.walk('./'):
  for filename in filenames:
   if filename.endswith('.weibo'):
    print filename
    with open(self.dir_temp + filename, 'r') as f:
     d = pickle.load(f)
     if d.has_key('name') and d.has_key('value') and d.has_key('expiry'):
      expiry_date = int(d['expiry'])
      if expiry_date > (int)(time.time()):
       cookie_dict[d['name']] = d['value']
      else:
       return {}
 return cookie_dict

3. If the cache cookie expires, obtain the cookie from the network again

def get_cookie():
 cookie_dict = get_cookie_from_cache()
 if not cookie_dict:
  cookie_dict = get_cookie_from_network()
 return cookie_dict

4. Use cookies to request other Weibo homepages

def get_weibo_list(self, user_id):
 import requests
 from bs4 import BeautifulSoup as bs
 cookdic = get_cookie()
 url = 'http://weibo.cn/stocknews88' 
 headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36'}
 timeout = 5
 r = requests.get(url, headers=headers, cookies=cookdic,timeout=timeout)
 soup = bs(r.text, 'lxml')
 ...
 # 用BeautifulSoup 解析网页
 ...

I believe you have mastered the method after reading the case in this article. For more exciting information, please pay attention to other related articles on the PHP Chinese website!

Recommended reading:

Detailed explanation of the steps for vue to use the xe-utils function library

Vue refreshes after packaging the project How to deal with 404 display

The above is the detailed content of How to automatically obtain and update cookies. For more information, please follow other related articles on the PHP Chinese website!