网页爬虫 - python爬虫翻页问题，请问各位大神我这段代码怎样翻页，还有价格要登陆后才能看到，应该怎么解决

Question

import urllib.request
import re
web=urllib.request.urlopen('https://www.gpyh.com/pricebuy/index?pageNum=1&hasStock=&goodsStandardId=1931&materialDictCode=&materialGroupCode=037001&diameter=&length=&brandId=&merchantId=')
neirong=web.read()
def getPage(self,pageIndex):
    url = self.siteURL + "?pageNum=" + str(pageIndex)
    request = urllib2.Request(url)
    response = urllib2.urlopen(request)
    return response.read().decode('gbk')
jiangrenhua=neirong.decode('UTF-8')
RegularExpression='<td>(.*)<\/td>'
Valuable=re.findall(RegularExpression,jiangrenhua)
information=[]
for i in range(173):
    print(Valuable[i]

Answer 1

?pageNum=" + str(pageIndex)

這一個不就是你的頁碼控制嗎？
登入後才看到那就用cookie或使用者名稱密碼模擬登入後取得

Answer 2

httplib2基本上應該是所有http請求的終結者了吧。

import httplib2
import urllib
http = httplib2.Http()
url='要获取的地址'
header={'Accept':'text/html',
     'Accept-Encoding':'gzip, deflate, sdch',
     'Accept-Language':'zh-CN,zh;q=0.8',
     'Cache-Control':'max-age=0',
     'Connection':'keep-alive',
     'Cookie':'cookie内容',
     'Upgrade-Insecure-Requests':'1',
     'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36'}   #要有登陆状态才能翻页就要模拟登陆后把cookie放进去
body_value={'username':'test','password':'123456'} #表单的所有内容
body_value=urllib.urlencode(body_value) #utf8编码
response, content = http.request(url, 'GET', headers=header,body=body_value)  #GET或者POST方法
response.encoding = 'utf-8'
#content就是返回内容