Web crawler - Python requests crawler, how to post the data in the payload, thank you!

Question

When I use requests to make a post request, how do I use this payload? I know how to construct
form-data data from
data data.

import requests
import json

url='http://tx3.cbg.163.com/cgi-bin/equipquery.py?act=show_overall_search'
html = requests.post(url,data=data)

Answer 1

According to the two answers above, the data has been successfully crawled out after modification

# -*- coding: utf-8 -*-

import requests
import sys
import json
reload(sys)
sys.setdefaultencoding('utf-8')

def get_tx3cbg():
    url = 'http://tx3.cbg.163.com/cgi-bin/search.py'
    headers = {
        'Content-Type':'application/x-www-form-urlencoded; charset=UTF-8',
        'Cookie':'overall_search_equips_cookie=%7C89_212365%7C23_211970%7C9_204469%7C2_156154%7C192_28326%7C176_74442%7C15_480910%7C15_477817%7C2_156122%7C10_156067%7C10_133849%7C15_481140%7C78_425212%7C12_165185%7C161_222125; usertrack=ZUcIhliu8JNB01gjA64jAg==; _ntes_nnid=f90c5442c5912e2adc252657cb12d457,1487860049616; _ntes_nuid=f90c5442c5912e2adc252657cb12d457; NTES_CMT_USER_INFO=35428995%7C%E5%BE%A1%E5%89%91%E6%8A%8A%E9%85%92%E5%90%AC%E7%A7%8B%E9%9B%A8%7Chttps%3A%2F%2Fsimg.ws.126.net%2Fe%2Fimg5.cache.netease.com%2Ftie%2Fimages%2Fyun%2Fphoto_default_62.png.39x39.100.jpg%7Cfalse%7CZmFuZ3dlaXJlbjg0M0AxNjMuY29t; UM_distinctid=15b0f71ba430-0e2166208-581f3413-1fa400-15b0f71ba4415a; vjuids=28a3aef4a.15b1fcdeaec.0.5e87609c; vjlast=1490887240.1490887240.30; vinfo_n_f_l_n3=c384af10aeab8944.1.0.1490887240441.0.1490887272552; P_INFO=fangweiren843@163.com|1491746466|1|blog|11&16|zhj&1491645222&gamebbs#zhj&330500#10#0#0|157600&0|gamebbs&tx2&mail163&blog|fangweiren843@163.com; _ga=GA1.2.69335350.1487859926; latest_views=15_466493-89_212755-1_217134-19_190513; __utma=187553192.69335350.1487859926.1492699270.1492699270.1; __utmz=187553192.1492699270.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __oc_uuid=65296ee0-25d7-11e7-be4c-213889b11650; last_login_serverid=19; area_td_id=1; overall_sid=000bEiuYYQVly-GWAHJ3BgbcS4ypMSxBavIhk6yVALj',
        'Referer':'http://tx3.cbg.163.com/cgi-bin/equipquery.py?act=show_overall_search',
        'User-Agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36'
    }
    payload = {
        'act':'overall_search_role',
        'order_by':'',
        'page':'1',
        'other_arg':'',
        'price_min':'50000',
        'price_max':'60000'
    }

    html = requests.post(url,data=payload,headers=headers).text
    html = json.loads(html)
    print html
    
get_tx3cbg()

Answer 2

Change the request payload data into dict and pass it to data

Answer 3

Check the documentation often!
http://docs.python-requests.o...

>>> payload = {'key1': 'value1', 'key2': 'value2'}

>>> r = requests.post("http://httpbin.org/post", data=payload)
>>> print(r.text)
{
  ...
  "form": {
    "key2": "value2",
    "key1": "value1"
  },
  ...
}