Write Java Zhihu crawler with zero foundation to crawl Zhihu answers
In the early stage, we crawled the title under this link:
http://www.zhihu.com/explore/recommendations
But obviously this page cannot get the answer.
A complete question page should be like this link:
http://www.zhihu.com/question/22355264
Looking carefully, aha, our encapsulation class needs to be further packaged, at least a questionDescription is needed To store the problem description:
import java.util.ArrayList;
public class Zhihu {
public String question;// 问题
public String questionDescription;// 问题描述
public String zhihuUrl;// 网页链接
public ArrayList<String> answers;// 存储所有回答的数组
// 构造方法初始化数据
public Zhihu() {
question = "";
questionDescription = "";
zhihuUrl = "";
answers = new ArrayList<String>();
}
@Override
public String toString() {
return "问题：" + question + "\n" + "描述：" + questionDescription + "\n"
+ "链接：" + zhihuUrl + "\n回答：" + answers + "\n";
}
}We add a parameter to the constructor of Zhihu to set the url value. Because the url is determined, the description and answer of the problem can be captured.
Let’s change Spider’s method of obtaining Zhihu objects and only obtain the url:
static ArrayList<Zhihu> GetZhihu(String content) {
// 预定义一个ArrayList来存储结果
ArrayList<Zhihu> results = new ArrayList<Zhihu>();
// 用来匹配url，也就是问题的链接
Pattern urlPattern = Pattern.compile("<h2 id="question-link-href">.+?question_link.+?href=\"(.+?)\".+?</h2>");
Matcher urlMatcher = urlPattern.matcher(content);
// 是否存在匹配成功的对象
boolean isFind = urlMatcher.find();
while (isFind) {
// 定义一个知乎对象来存储抓取到的信息
Zhihu zhihuTemp = new Zhihu(urlMatcher.group(1));
// 添加成功匹配的结果
results.add(zhihuTemp);
// 继续查找下一个匹配对象
isFind = urlMatcher.find();
}
return results;
}Next, in Zhihu’s construction method, obtain all detailed data through the url.
We need to process the url first, because some are for answers, its url is:
http://www.zhihu.com/question/22355264/answer/21102139
Some are for questions , its url is:
http://www.zhihu.com/question/22355264
Then we obviously need the second type, so we need to use regular expressions to cut the first type of link into the second type, Just write a function in Zhihu for this.
// 处理url
boolean getRealUrl(String url) {
// 将http://www.zhihu.com/question/22355264/answer/21102139
// 转化成http://www.zhihu.com/question/22355264
// 否则不变
Pattern pattern = Pattern.compile("question/(.*?)/");
Matcher matcher = pattern.matcher(url);
if (matcher.find()) {
zhihuUrl = "http://www.zhihu.com/question/" + matcher.group(1);
} else {
return false;
}
return true;
}The next step is to obtain each part.
Look at the title first:
Just grasp the class. The regular statement can be written as: zm-editable-content">(.+?)<
Run it and see the result:
Oh, that’s good.
Next, grab the problem description:
Ah, the same principle, grab the class, because it should be the unique identifier of this
Verification method: Right-click to view the page. Source code, ctrl+F to see if there are other strings in the page.
After verification, there is really a problem:
The class in front of the title and description content are the same.
The only way to re-crawl is by modifying the regular expression:
// 匹配标题
pattern = Pattern.compile("zh-question-title.+?<h2 id="">(.+?)</h2>");
matcher = pattern.matcher(content);
if (matcher.find()) {
question = matcher.group(1);
}
// 匹配描述
pattern = Pattern
.compile("zh-question-detail.+?<div.+?>(.*?)</div>");
matcher = pattern.matcher(content);
if (matcher.find()) {
questionDescription = matcher.group(1);
}Finally, it’s a loop to capture the answer:
Preliminary tentative regular sentence: /answer/content.+?
After changing the code, we will find that the software runs significantly slower, because it needs to visit each web page and capture the content on it
For example, editor recommendations. If there are 20 questions, then you need to visit the web page 20 times, and the speed will slow down.
Try it out, it seems to work well:
OK, that’s it for now~ Let’s continue with some details next time. Adjustments, such as multi-threading, IO stream writing to local, etc.
The above is the content of writing Java Zhihu crawler to crawl Zhihu answers. For more related content, please pay attention to the PHP Chinese website (www.php). .cn)
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1664
14
1423
52
1318
25
1268
29
1248
24
Break or return from Java 8 stream forEach?
Feb 07, 2025 pm 12:09 PM
Java 8 introduces the Stream API, providing a powerful and expressive way to process data collections. However, a common question when using Stream is: How to break or return from a forEach operation? Traditional loops allow for early interruption or return, but Stream's forEach method does not directly support this method. This article will explain the reasons and explore alternative methods for implementing premature termination in Stream processing systems. Further reading: Java Stream API improvements Understand Stream forEach The forEach method is a terminal operation that performs one operation on each element in the Stream. Its design intention is
PHP: A Key Language for Web Development
Apr 13, 2025 am 12:08 AM
PHP is a scripting language widely used on the server side, especially suitable for web development. 1.PHP can embed HTML, process HTTP requests and responses, and supports a variety of databases. 2.PHP is used to generate dynamic web content, process form data, access databases, etc., with strong community support and open source resources. 3. PHP is an interpreted language, and the execution process includes lexical analysis, grammatical analysis, compilation and execution. 4.PHP can be combined with MySQL for advanced applications such as user registration systems. 5. When debugging PHP, you can use functions such as error_reporting() and var_dump(). 6. Optimize PHP code to use caching mechanisms, optimize database queries and use built-in functions. 7
PHP vs. Python: Understanding the Differences
Apr 11, 2025 am 12:15 AM
PHP and Python each have their own advantages, and the choice should be based on project requirements. 1.PHP is suitable for web development, with simple syntax and high execution efficiency. 2. Python is suitable for data science and machine learning, with concise syntax and rich libraries.
PHP vs. Other Languages: A Comparison
Apr 13, 2025 am 12:19 AM
PHP is suitable for web development, especially in rapid development and processing dynamic content, but is not good at data science and enterprise-level applications. Compared with Python, PHP has more advantages in web development, but is not as good as Python in the field of data science; compared with Java, PHP performs worse in enterprise-level applications, but is more flexible in web development; compared with JavaScript, PHP is more concise in back-end development, but is not as good as JavaScript in front-end development.
PHP vs. Python: Core Features and Functionality
Apr 13, 2025 am 12:16 AM
PHP and Python each have their own advantages and are suitable for different scenarios. 1.PHP is suitable for web development and provides built-in web servers and rich function libraries. 2. Python is suitable for data science and machine learning, with concise syntax and a powerful standard library. When choosing, it should be decided based on project requirements.
PHP's Impact: Web Development and Beyond
Apr 18, 2025 am 12:10 AM
PHPhassignificantlyimpactedwebdevelopmentandextendsbeyondit.1)ItpowersmajorplatformslikeWordPressandexcelsindatabaseinteractions.2)PHP'sadaptabilityallowsittoscaleforlargeapplicationsusingframeworkslikeLaravel.3)Beyondweb,PHPisusedincommand-linescrip
Java Program to Find the Volume of Capsule
Feb 07, 2025 am 11:37 AM
Capsules are three-dimensional geometric figures, composed of a cylinder and a hemisphere at both ends. The volume of the capsule can be calculated by adding the volume of the cylinder and the volume of the hemisphere at both ends. This tutorial will discuss how to calculate the volume of a given capsule in Java using different methods. Capsule volume formula The formula for capsule volume is as follows: Capsule volume = Cylindrical volume Volume Two hemisphere volume in, r: The radius of the hemisphere. h: The height of the cylinder (excluding the hemisphere). Example 1 enter Radius = 5 units Height = 10 units Output Volume = 1570.8 cubic units explain Calculate volume using formula: Volume = π × r2 × h (4
PHP: The Foundation of Many Websites
Apr 13, 2025 am 12:07 AM
The reasons why PHP is the preferred technology stack for many websites include its ease of use, strong community support, and widespread use. 1) Easy to learn and use, suitable for beginners. 2) Have a huge developer community and rich resources. 3) Widely used in WordPress, Drupal and other platforms. 4) Integrate tightly with web servers to simplify development deployment.
