网页爬虫 - 在python中使用urllib2.urlopen获取的html网页不完整

Question

http://apk.hiapk.com/appinfo/...
我想要爬取这个网页中的用户评论
但是却发现使用urllib2.urlopen（request）获取的html页面不完整

代码如图

输出如图

但是实际上这个页面里面是有东西的

请问一下为什么获取到的html不全呢

Answer 1

It’s not that the retrieval is incomplete. The comments on this page are generated through javascript post-loading. The requested html is returned in urllib2.urlopen and the javascript in the page will not be executed

You can directly request http://apk.hiapk.com/web/api.... This address can return all comments, it is still json, it is easy to handle