此腳本將從 BoardGameGeek API 取得專案資料並將資料儲存在 CSV 檔案中。
我更新了之前的腳本。由於 API 回應採用 XML 格式，且沒有端點可以一次取得所有項目，因此前面的腳本將循環遍歷提供的 ID 範圍，對每個項目進行逐一呼叫。這不是最優的，對於更大範圍的 ID 需要很長時間（目前 BGG 上可用的最高項目（ID）數量高達 400k ），而且結果可能不可靠。因此，透過對此腳本的一些修改，更多的項目ID將作為參數值添加到單一請求url中，這樣，單一回應將傳回多個項目（〜800是單一回應傳回的最高數量。BGG稍後可能會更改它；您可以輕鬆調整batch_size以便根據需要進行調整）。
此外，此腳本將獲取所有項目，而不僅僅是與棋盤遊戲相關的數據。
為每個棋盤遊戲取得和儲存的資訊如下：
名稱、遊戲ID、類型、評級、權重、發布年份、最小玩家數、最大玩家數、最短遊戲時間、最大支付時間、最小年齡、所屬者、類別、機制、設計師、藝術家和發行商。
該腳本的更新如下；我們首先導入此腳本所需的庫：
# Import libraries
from bs4 import BeautifulSoup
from csv import DictWriter
import pandas as pd
import requests
import time
登入後複製
以下是腳本完成時根據 ID 範圍呼叫的函數。此外，如果發出請求時發生錯誤，則會呼叫此函數以儲存截至異常發生時附加到遊戲清單的所有資料。
# CSV file saving function
def save_to_csv(games):
csv_header = [
'name', 'game_id', 'type', 'rating', 'weight', 'year_published', 'min_players', 'max_players',
'min_play_time', 'max_play_time', 'min_age', 'owned_by', 'categories',
'mechanics', 'designers', 'artists', 'publishers'
]
with open('bgg.csv', 'a', encoding='UTF8') as f:
dictwriter_object = DictWriter(f, fieldnames=csv_header)
if f.tell() == 0:
dictwriter_object.writeheader()
dictwriter_object.writerows(games)
登入後複製
我們需要定義請求的標頭。請求之間的暫停可以透過 SLEEP_BETWEEN_REQUESTS 設定（我看到一些資訊說速率限制是每秒 2 個請求，但它可能是過時的訊息，因為我將暫停設定為 0 沒有遇到問題）。另外，這裡設定起始點ID（start_id_range）、最大範圍（max_id_range）和batch_size的值，batch_size是回應應該回傳的遊戲數量。基本 url 在本節中定義，但 ID 在腳本的下一部分中新增。
# Define request url headers
headers = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:85.0) Gecko/20100101 Firefox/85.0",
"Accept-Language": "en-GB, en-US, q=0.9, en"
}
# Define sleep timer value between requests
SLEEP_BETWEEN_REQUEST = 0
# Define max id range
start_id_range = 0
max_id_range = 403000
batch_size = 800
base_url = "https://boardgamegeek.com/xmlapi2/thing?id="
登入後複製
以下是這個腳本的主要邏輯。首先，根據batch size，它會產生一個ID字串，這些ID在定義的ID範圍內，但ID的數量不能超過batch_size中定義的數量，並將其附加到url的id參數中。這樣，每個回應將傳回與批量大小相同的項目數量的資料。之後，它將處理資料並將其附加到每個回應的遊戲清單中，最後附加到 CSV 檔案中。
# Main loop that will iterate between the starting and maximum range in intervals of the batch size
for batch_start in range(start_id_range, max_id_range, batch_size):
# Make sure that the batch size will not exceed the maximum ids range
batch_end = min(batch_start + batch_size - 1, max_id_range)
# Join and append to the url the IDs within batch size
ids = ",".join(map(str, range(batch_start, batch_end + 1)))
url = f"{base_url}?id={ids}&stats=1"
# If by any chance there is an error, this will throw the exception and continue on the next batch
try:
response = requests.get(url, headers=headers)
except Exception as err:
print(err)
continue
if response.status_code == 200:
soup = BeautifulSoup(response.text, features="html.parser")
items = soup.find_all("item")
games = []
for item in items:
if item:
try:
# Find values in the XML
name = item.find("name")['value'] if item.find("name") is not None else 0
year_published = item.find("yearpublished")['value'] if item.find("yearpublished") is not None else 0
min_players = item.find("minplayers")['value'] if item.find("minplayers") is not None else 0
max_players = item.find("maxplayers")['value'] if item.find("maxplayers") is not None else 0
min_play_time = item.find("minplaytime")['value'] if item.find("minplaytime") is not None else 0
max_play_time = item.find("maxplaytime")['value'] if item.find("maxplaytime") is not None else 0
min_age = item.find("minage")['value'] if item.find("minage") is not None else 0
rating = item.find("average")['value'] if item.find("average") is not None else 0
weight = item.find("averageweight")['value'] if item.find("averageweight") is not None else 0
owned = item.find("owned")['value'] if item.find("owned") is not None else 0
link_type = {'categories': [], 'mechanics': [], 'designers': [], 'artists': [], 'publishers': []}
links = item.find_all("link")
# Append value(s) for each link type
for link in links:
if link['type'] == "boardgamecategory":
link_type['categories'].append(link['value'])
if link['type'] == "boardgamemechanic":
link_type['mechanics'].append(link['value'])
if link['type'] == "boardgamedesigner":
link_type['designers'].append(link['value'])
if link['type'] == "boardgameartist":
link_type['artists'].append(link['value'])
if link['type'] == "boardgamepublisher":
link_type['publishers'].append(link['value'])
# Append 0 if there is no value for any link type
for key, ltype in link_type.items():
if not ltype:
ltype.append("0")
game = {
"name": name,
"game_id": item['id'],
"type": item['type'],
"rating": rating,
"weight": weight,
"year_published": year_published,
"min_players": min_players,
"max_players": max_players,
"min_play_time": min_play_time,
"max_play_time": max_play_time,
"min_age": min_age,
"owned_by": owned,
"categories": ', '.join(link_type['categories']),
"mechanics": ', '.join(link_type['mechanics']),
"designers": ', '.join(link_type['designers']),
"artists": ', '.join(link_type['artists']),
"publishers": ', '.join(link_type['publishers']),
}
# Append current item to games list
games.append(game)
except TypeError:
print(">>> NoneType error. Continued on the next item.")
continue
save_to_csv(games)
print(f">>> Request successful for batch {batch_start}-{batch_end}")
else:
print(f">>> FAILED batch {batch_start}-{batch_end}")
# Pause between requests
time.sleep(SLEEP_BETWEEN_REQUEST)
登入後複製
下面您可以以 pandas DataFrame 的形式預覽 CSV 檔案中的前幾行記錄。
# Preview the CSV as pandas DataFrame
df = pd.read_csv('./bgg.csv')
print(df.head(5))
登入後複製
以上是我更新了 BoardGameGeek 資料的 Python 獲取器的詳細內容。更多資訊請關注PHP中文網其他相關文章！
