requests - 关于python模拟登陆抓取网站信息

Question

我想用requests实现在www.lu.com模拟登陆后抓取一些产品信息，通过chrome的抓包工具我抓到了登陆时的form data和header(变量header)，以及访问需要抓取的页面时的header(变量listheader),但运行完程序后我输入r.url发现返回的地址是https://list.lu.com/notFound.html,而不是我指定的具体页面，请问是怎么回事呢？

import requests,bs4
    
    s = requests.session()
    logindata={'userName':'XXXXXXX','password':'XXXXXXXX','isTrust':'Y','loginFlag':'1','validNum':'','loginagree':'on'}
    listheader={'Accept-Encoding':'gzip, deflate, sdch, br','Host':'list.lu.com','Upgrade-Insecure-Requests':'1','User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36'}
    header={'User-Agent':'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.82 Safari/537.36','X-Requested-With':'XMLHttpRequest','Referer':'https://user.lu.com/user/login?returnPostURL=https%3A%2F%2Fwww.lu.com%2F'}
    res=s.post('https://user.lu.com/user/login',logindata,headers=header)
    r=s.get('https://list.lu.com/list/productDetail/transferFa?riskFlag=false&productId=31398999',headers=listheader)

Answer 1

# coding=utf-8

__author__ = 'Rich'

import requests

session = requests.session()
session.headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36'
}
data = {
    'userName': 'XXXXXXX',
    'password': 'XXXXXXXX',
    'isTrust': 'Y',
    'loginFlag': '1',
    'validNum': '',
    'loginagree': 'on'
}

session.post('https://user.lu.com/user/login', data=data)

r = session.get('https://list.lu.com/list/productDetail/transferFa?riskFlag=false&productId=31398999')
print r.text