Python crawler obtains websites of American dramas
This article was originally created by MaNong.com – Xiao Hao. Please read the reprint requirements at the end of the article for reprinting. Welcome to participate in our paid contribution plan!
I have always had the habit of watching American TV series. On the one hand, I can practice my English listening skills, and on the other hand, I can pass the time. It used to be possible to watch online on video websites, but since the restriction order imposed by the State Administration of Radio, Film and Television, it seems that imported American and British dramas are no longer updated simultaneously as before. However, as a nerd, how can I be willing to not follow any dramas, so I checked online and found an American drama download website [Tiantian American Dramas] that can be downloaded using Thunder. I can download various resources at will. Recently, I am obsessed with BBC’s High-definition documentary, nature is so beautiful.
Although I have found a resource website that can be downloaded, I have to open the browser every time, enter the URL, find the American drama, and then click the link to download. After a long time, the process becomes very cumbersome, and sometimes the website link cannot be opened, which is a bit troublesome. I happen to have been learning Python crawler, so today I wrote a crawler on a whim to grab all the American drama links on the website and save them in a text document. If you want any drama, just open it and copy the link to You can download it via Thunder.
In fact, I originally planned to write something that finds a URL, uses requests to open it, grabs the download link, and crawls the entire site starting from the homepage. However, there are a lot of repeated links, and the URL of the website is not as regular as I thought. After writing for a long time, I still haven't written the kind of divergent crawler I want. Maybe I am not mature enough, so keep working hard. . .
Later I discovered that the links to the TV series were all in the article, and there was a number behind the article URL, like this http://cn163.net/archives/24016/, So I cleverly used the crawler experience I wrote before. The solution is to automatically generate the URL. Can’t the number behind it be changed? And each drama is unique, so I tried to find out how many articles there are. , and then use the range function to directly generate numbers continuously to construct the url.
But many URLs do not exist, so they will hang up directly. Don’t worry, we are using requests, and its built-in status_code is used to determine the status returned by the request, so as long as it is the returned status We skip all those with code 404, and crawl the other links, which solves the URL problem.
The following is the implementation code of the above steps.
def get_urls(self): try: for i in range(2015,25000): base_url='http://cn163.net/archives/' url=base_url+str(i)+'/' if requests.get(url).status_code == 404: continue else: self.save_links(url) except Exception,e: pass
The rest went very smoothly. I found a similar crawler written by someone before on the Internet, but it only crawled one article, so I borrowed its regular expressions. I used BeautifulSoup but the effect was not as good as the regular method, so I gave up decisively. There is no limit to my learning. However, the effect is not so ideal. About half of the links cannot be crawled correctly and need to continue to be optimized.
# -*- coding:utf-8 -*- import requests import re import sys import threading import time reload(sys) sys.setdefaultencoding('utf-8') class Archives(object): def save_links(self,url): try: data=requests.get(url,timeout=3) content=data.text link_pat='"(ed2k://\|file\|[^"]+?\.(S\d+)(E\d+)[^"]+?1024X\d{3}[^"]+?)"' name_pat=re.compile(r'<h2 class="entry_title">(.*?)</h2>',re.S) links = set(re.findall(link_pat,content)) name=re.findall(name_pat,content) links_dict = {} count=len(links) except Exception,e: pass for i in links: links_dict[int(i[1][1:3]) * 100 + int(i[2][1:3])] = i#把剧集按s和e提取编号 try: with open(name[0].replace('/',' ')+'.txt','w') as f: print name[0] for i in sorted(list(links_dict.keys())):#按季数+集数排序顺序写入 f.write(links_dict[i][0] + '\n') print "Get links ... ", name[0], count except Exception,e: pass def get_urls(self): try: for i in range(2015,25000): base_url='http://cn163.net/archives/' url=base_url+str(i)+'/' if requests.get(url).status_code == 404: continue else: self.save_links(url) except Exception,e: pass def main(self): thread1=threading.Thread(target=self.get_urls()) thread1.start() thread1.join() if __name__ == '__main__': start=time.time() a=Archives() a.main() end=time.time() print end-start
The full version of the code also uses multi-threading, but it seems useless. It may be because of Python's GIL. There seem to be more than 20,000 dramas. I thought it would take a long time to complete the crawling. , but excluding URL errors and unmatched URLs, the total crawling time is less than 20 minutes. I originally wanted to use Redis to crawl on two Linux machines, but after a lot of fussing, I felt it was unnecessary, so I left it at that and will do it later when I need more data.
Another problem that tortured me during the process was the saving of file names. I must complain here. File names in txt text format can have spaces, but they cannot have slashes, backslashes, brackets etc. This is the problem. I spent the whole morning on this. At first I thought it was an error in crawling the data. After checking for a long time, I found out that the crawled drama title had a slash in it. This made me miserable. .
The above is the detailed content of Python crawler obtains websites of American dramas. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

Google AI has started to provide developers with access to extended context windows and cost-saving features, starting with the Gemini 1.5 Pro large language model (LLM). Previously available through a waitlist, the full 2 million token context windo

How to download DeepSeek Xiaomi? Search for "DeepSeek" in the Xiaomi App Store. If it is not found, continue to step 2. Identify your needs (search files, data analysis), and find the corresponding tools (such as file managers, data analysis software) that include DeepSeek functions.

The key to using DeepSeek effectively is to ask questions clearly: express the questions directly and specifically. Provide specific details and background information. For complex inquiries, multiple angles and refute opinions are included. Focus on specific aspects, such as performance bottlenecks in code. Keep a critical thinking about the answers you get and make judgments based on your expertise.

Just use the search function that comes with DeepSeek. Its powerful semantic analysis algorithm can accurately understand the search intention and provide relevant information. However, for searches that are unpopular, latest information or problems that need to be considered, it is necessary to adjust keywords or use more specific descriptions, combine them with other real-time information sources, and understand that DeepSeek is just a tool that requires active, clear and refined search strategies.

DeepSeek is not a programming language, but a deep search concept. Implementing DeepSeek requires selection based on existing languages. For different application scenarios, it is necessary to choose the appropriate language and algorithms, and combine machine learning technology. Code quality, maintainability, and testing are crucial. Only by choosing the right programming language, algorithms and tools according to your needs and writing high-quality code can DeepSeek be successfully implemented.

Question: Is DeepSeek available for accounting? Answer: No, it is a data mining and analysis tool that can be used to analyze financial data, but it does not have the accounting record and report generation functions of accounting software. Using DeepSeek to analyze financial data requires writing code to process data with knowledge of data structures, algorithms, and DeepSeek APIs to consider potential problems (e.g. programming knowledge, learning curves, data quality)

Python is an ideal programming introduction language for beginners through its ease of learning and powerful features. Its basics include: Variables: used to store data (numbers, strings, lists, etc.). Data type: Defines the type of data in the variable (integer, floating point, etc.). Operators: used for mathematical operations and comparisons. Control flow: Control the flow of code execution (conditional statements, loops).

Pythonempowersbeginnersinproblem-solving.Itsuser-friendlysyntax,extensivelibrary,andfeaturessuchasvariables,conditionalstatements,andloopsenableefficientcodedevelopment.Frommanagingdatatocontrollingprogramflowandperformingrepetitivetasks,Pythonprovid
