简体中文(ZH-CN) English(EN) 繁体中文(ZH-TW) 日本語(JA) 한국어(KO) Melayu(MS) Français(FR) Deutsch(DE)
python - Problems encountered in crawling the csdn background article list after the crawler simulates login
ringa_lee
ringa_lee 2017-05-18 10:51:59
0
2
1015

The crawler has indeed logged in, because I can crawl personal information and it can be captured, but the URL in the picture below cannot be captured:

The URL is: http://write.blog.csdn.net/postlist, which is your csdn backend.

I’ll post the code, it’s for py2.7

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from bs4 import BeautifulSoup
import requests

class CSDN(object):
    def __init__(self, headers):
        self.session = requests.Session()
        self.headers = headers
    def get_webflow(self):
        url = 'http://passport.csdn.net/account/login'
        response = self.session.get(url=url, headers=self.headers)
        soup = BeautifulSoup(response.text, 'html.parser')
        lt = soup.find('input', {'name': 'lt'})['value']
        execution = soup.find('input', {'name': 'execution'})['value']
        soup.clear()
        return (lt, execution)
    def login(self, account, password):
        self.username = account
        self.password = password
        lt, execution = self.get_webflow()
        data = {
            'username': account,
            'password': password,
            'lt': lt,
            'execution': execution,
            '_eventId': 'submit'
        }
        url = 'http://passport.csdn.net/account/login'
        response = self.session.post(url=url, headers=self.headers, data=data)
        if (response.status_code == 200):
            print('正常')
        else:
            print('异常')
    def func(self):
        headers1={
            'Host':'write.blog.csdn.net',
            'Upgrade-Insecure-Requests':'1',
            'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36'
        }
        response=self.session.get(url='http://write.blog.csdn.net/postlist',headers=headers1,allow_redirects=False)
        print response.text
if __name__ == '__main__':
    headers = {
        'Host': 'passport.csdn.net',
        'Origin': 'http://passport.csdn.net',
        'Referer':'http://passport.csdn.net/account/login',
        'Upgrade-Insecure-Requests':'1',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36',
    }
    csdn = CSDN(headers=headers)
    account = ''
    password = ''
    csdn.login(account=account, password=password)
    csdn.func()

The output of the above code is

正常
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://passport.csdn.net?service=http://write.blog.csdn.net/postlist">here</a>.</h2>
</body></html>
ringa_lee
ringa_lee

ringa_lee

reply all(2)
小葫芦
小葫芦2017-05-18 10:53:59 2 floor

Because this address returns a 302 jump, you need to continue the request based on the Location of the returned header, and then analyze the returned content to continue processing. The browser will help you make these 302 jumps and execute the returned js and other content, and manually capture it. You need to handle it yourself.

Like +0
Add Reply
迷茫
迷茫2017-05-18 10:53:59 1 floor

Just use cookies

Like +0
Add Reply
Popular Topics
More>
Popular Articles
Popular Tutorials
More>
Related Tutorials
Popular Recommendations
Latest courses
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template
About us Disclaimer Sitemap
php.cn：Public welfare online PHP training，Help PHP learners grow quickly！