Python crawls are you hungry?-Python Tutorial-php.cn

Python crawls are you hungry?

步履不停

Release： 2021-05-28 15:01:22

Original

5278 people have browsed it

Python crawls are you hungry?

I am learning data visualization and lack some data for practical operation, so I want to crawl some takeaway store information from Ele.me.

Mainly to obtain data, so the code is relatively simple, just go to the code:

import requests
import json
import csv
def crawler_ele(page=0):
def get_page(page):
url = &#39;https://h5.ele.me/restapi/shopping/v3/restaurants?latitude=xxxx&longitude=xxxx&offset={page}&limit=8&terminal=h5&#39;.format(page=page*8)
headers = {
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36",
    &#39;cookie&#39;: r&#39;xxxx&#39;
}
re = json.loads(requests.get(url,headers=headers).text)
return re
re = get_page(page)
if re.get(&#39;items&#39;):
with open(&#39;data.csv&#39;,&#39;a&#39;,newline=&#39;&#39;) as f:
writer = csv.DictWriter(f,fieldnames=[&#39;名称&#39;, &#39;月销售量&#39;,&#39;配送费&#39;, &#39;起送价&#39;, &#39;风味&#39;,&#39;评分&#39;, &#39;配送时长&#39;, &#39;评分统计&#39;, &#39;距离&#39;, &#39;地址&#39;])
writer.writeheader()
for item in re.get(&#39;items&#39;):
info = dict()
restaurant = item.get(&#39;restaurant&#39;)
info[&#39;地址&#39;] = restaurant.get(&#39;address&#39;)
info[&#39;配送费&#39;] = restaurant.get(&#39;float_delivery_fee&#39;)
info[&#39;名称&#39;] = restaurant.get(&#39;name&#39;)
info[&#39;配送时长&#39;] = restaurant.get(&#39;order_lead_time&#39;)
info[&#39;距离&#39;] = restaurant.get(&#39;distance&#39;)
info[&#39;起送价&#39;] = restaurant.get(&#39;float_minimum_order_amount&#39;)
info[&#39;评分&#39;] = restaurant.get(&#39;rating&#39;)
info[&#39;月销售量&#39;] = restaurant.get(&#39;recent_order_num&#39;)
info[&#39;评分统计&#39;] = restaurant.get(&#39;rating_count&#39;)
info[&#39;风味&#39;] = restaurant.get(&#39;flavors&#39;)[0].get(&#39;name&#39;)
writer.writerow(info) 
# print(info)
if re.get(&#39;has_next&#39;) == True:
crawler_page(page+1)
crawler_ele(0)

Copy after login

Here are a few simple comments:

The longitude and latitude in the url have been removed, you can query it yourself Add the latitude and longitude of the location to be crawled, or you can obtain the latitude and longitude by calling the map api;

Cookies need to be added to the headers, otherwise the login permission will limit the number of crawled pages;

The final call is recursive It is not a loop, so there will be multiple duplicate headers in the csv file that saves the results. You can open it with Excel and delete the duplicate values.

Related tutorial recommendations: Python video tutorial

The above is the detailed content of Python crawls are you hungry?. For more information, please follow other related articles on the PHP Chinese website!