How does a Java crawler implement Jsoup using the dom method to traverse the Document object?-javaTutorial-php.cn

Home

Java

javaTutorial

How does a Java crawler implement Jsoup using the dom method to traverse the Document object?

王林

May 14, 2023 pm 02:19 PM

java jsoup document

先给出网页地址：

https://wall.alphacoders.com/featured.php?lang=Chinese

Copy after login

主要步骤：

利用Jsoup的connect方法获取Document对象

String html = "https://wall.alphacoders.com/featured.php?lang=Chinese";
Document doc = Jsoup.connect(html).get();

Copy after login

内容过长，就不再显示。

我们以这部分为例：

<ul class="nav nav-pills"> 
    <li><a href="https://alphacoders.com/site/about-us" rel="external nofollow"  rel="external nofollow"  >About Us</a></li> 
    <li><a href="https://alphacoders.com/site/faq" rel="external nofollow"  rel="external nofollow"  >FAQ</a></li> 
    <li><a href="https://alphacoders.com/site/privacy" rel="external nofollow"  rel="external nofollow"  >Privacy Policy</a></li> 
    <li><a href="https://alphacoders.com/site/tos" rel="external nofollow"  rel="external nofollow"  >Terms Of Service</a></li> 
    <li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow"  rel="external nofollow"  >Acceptable Use</a></li> 
    <li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow"  rel="external nofollow"  >Etiquette</a></li> 
    <li><a href="https://alphacoders.com/site/advertising" rel="external nofollow"  rel="external nofollow"  >Advertise With Us</a></li> 
    <li><a id="change_consent">Change Consent</a></li> 
</ul>

Copy after login

我们先找到所有的ul：

Elements elements = doc.getElementsByTag("ul");

Copy after login

输出如下：

<ul class="nav navbar-nav center"> 
 <li> <a title="Submit Wallpapers" href="https://alphacoders.com/site/submit-wallpaper" rel="external nofollow" ><i class="el el-circle-arrow-up"></i> 提交</a> </li> 
 <li> <a href="https://alphacoders.com/contest" rel="external nofollow" ><i class="el el-gift"></i> 精美奖品</a> </li> 
</ul>
<ul class="nav navbar-nav navbar-right center"> 
 <li> <a href="language.php?lang=Chinese" rel="external nofollow" > <img src="https://cache.yisu.com/upload/information/20220517/112/2427.png" alt="Chinese-flag">   中文    </a> </li> 
 <li> <a  href="https://alphacoders.com/users/login" rel="external nofollow" ><i class="el el-user"></i> 登录</a> </li> 
 <li> <a href="https://alphacoders.com/users/register" rel="external nofollow" ><i class="el el-edit"></i> 注册</a> </li> 
</ul>
<ul class="pagination"> 
 <li class="active"><a id="prev_page" href="#" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >&lt; 上一页</a></li> 
 <li class="active"><a>1</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >2</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=3" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >3</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=4" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >4</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=5" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >5</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=6" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >6</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=7" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >7</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=8" rel="external nofollow" >8</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=9" rel="external nofollow" >9</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=10" rel="external nofollow" >10</a></li> 
 <li><a>...</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=319" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >319</a></li> 
 <li><a id="next_page" href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >下一页 &gt;</a></li> 
</ul>
<ul class="pagination"> 
 <li class="active"><a href="#" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >&lt; 上一页</a></li> 
 <li class="active"><a>1</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >2</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=3" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >3</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=4" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >4</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=5" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >5</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=6" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >6</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=7" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >7</a></li> 
 <li><a>...</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=319" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >319</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >下一页 &gt;</a></li> 
</ul>
<ul class="pagination"> 
 <li class="active"><a href="#" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >&lt;&lt; </a></li> 
 <li class="active"><a href="#" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >&lt; 上一页</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >下一页 &gt;</a></li> 
 <li><a title="末页 (319)" href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=319" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" > &gt;&gt;</a></li> 
</ul>
<ul class="pagination"> 
 <li class="active"><a href="#" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >1</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >2</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=3" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >3</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=4" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >4</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=5" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >5</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=6" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >6</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=7" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >7</a></li> 
</ul>
<ul class="nav nav-pills"> 
 <li><a href="https://alphacoders.com/site/about-us" rel="external nofollow"  rel="external nofollow"  >About Us</a></li> 
 <li><a href="https://alphacoders.com/site/faq" rel="external nofollow"  rel="external nofollow"  >FAQ</a></li> 
 <li><a href="https://alphacoders.com/site/privacy" rel="external nofollow"  rel="external nofollow"  >Privacy Policy</a></li> 
 <li><a href="https://alphacoders.com/site/tos" rel="external nofollow"  rel="external nofollow"  >Terms Of Service</a></li> 
 <li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow"  rel="external nofollow"  >Acceptable Use</a></li> 
 <li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow"  rel="external nofollow"  >Etiquette</a></li> 
 <li><a href="https://alphacoders.com/site/advertising" rel="external nofollow"  rel="external nofollow"  >Advertise With Us</a></li> 
 <li><a id="change_consent">Change Consent</a></li> 
</ul>

Copy after login

可以发现class为"nav nav-pills"的只有一个，我们找到它：

Elements elements = doc.getElementsByTag("ul");
        //System.out.println(elements);
Element tempElement = null;
for(Element element : elements) {
    if (element.className().equals("nav nav-pills")) {
        tempElement = element;
        //System.out.println(element.className());
        break;
    }
}

Copy after login

循环遍历这个ul，输出其中每一个li里每一个a的href和rel属性：

Elements li = tempElement.getElementsByTag("li");
for(Element element : li) {
    Elements element2 = element.getElementsByTag("a");
    for(Element element3 : element2) {
        String hrefString = element3.attr("href");
        String relString = element3.attr("rel");
        if(hrefString != "" && relString != "") {
            System.out.println("href=" + hrefString + " rel="external nofollow"  rel="external nofollow"  " + "rel=" + relString);
        }
    }
}

Copy after login

最终结果：

href=https://alphacoders.com/site/about-us rel=nofollow
href=https://alphacoders.com/site/faq rel=nofollow
href=https://alphacoders.com/site/privacy rel=nofollow
href=https://alphacoders.com/site/tos rel=nofollow
href=https://alphacoders.com/site/acceptable_use rel=nofollow
href=https://alphacoders.com/site/etiquette rel=nofollow
href=https://alphacoders.com/site/advertising rel=nofollow

完整代码：

import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
import org.jsoup.Jsoup;


/** 
 * @ClassName: Jsoup_Test
 * @description: 
 * @author: KI
 * @Date: 2020年8月17日 下午8:15:14
 */

public class Jsoup_Test {

    public static void main(String[] args) throws IOException {
        // TODO 自动生成的方法存根
        
        String html = "https://wall.alphacoders.com/featured.php?lang=Chinese";
        Document doc = Jsoup.connect(html).get();
        
        System.out.println(doc);
        Elements elements = doc.getElementsByTag("ul");
        //System.out.println(elements);
        Element tempElement = null;
        for(Element element : elements) {
            if (element.className().equals("nav nav-pills")) {
                tempElement = element;
                //System.out.println(element.className());
                break;
            }
        }
        System.out.println(tempElement);
        Elements li = tempElement.getElementsByTag("li");
        for(Element element : li) {
            Elements element2 = element.getElementsByTag("a");
            for(Element element3 : element2) {
                String hrefString = element3.attr("href");
                String relString = element3.attr("rel");
                if(hrefString != "" && relString != "") {
                    System.out.println("href=" + hrefString + " rel="external nofollow"  rel="external nofollow"  " + "rel=" + relString);
                }
            }
        }        

    }

}

Copy after login

The above is the detailed content of How does a Java crawler implement Jsoup using the dom method to traverse the Document object?. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress images for free

Clothoff.io

AI clothes remover

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)

4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

R.E.P.O. Best Graphic Settings

4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Assassin's Creed Shadows: Seashell Riddle Solution

2 weeks ago By DDD

R.E.P.O. How to Fix Audio if You Can't Hear Anyone

4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

WWE 2K25: How To Unlock Everything In MyRise

1 months ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Where is the login entrance for gmail email?

7509

CakePHP Tutorial

1378

What is the format of the account name of steam

win11 activation key permanent

nyt connections hints and answers

Related knowledge

Perfect Number in Java Aug 30, 2024 pm 04:28 PM

Guide to Perfect Number in Java. Here we discuss the Definition, How to check Perfect number in Java?, examples with code implementation.

Random Number Generator in Java Aug 30, 2024 pm 04:27 PM

Guide to Random Number Generator in Java. Here we discuss Functions in Java with examples and two different Generators with ther examples.

Weka in Java Aug 30, 2024 pm 04:28 PM

Guide to Weka in Java. Here we discuss the Introduction, how to use weka java, the type of platform, and advantages with examples.

Smith Number in Java Aug 30, 2024 pm 04:28 PM

Guide to Smith Number in Java. Here we discuss the Definition, How to check smith number in Java? example with code implementation.

Java Spring Interview Questions Aug 30, 2024 pm 04:29 PM

In this article, we have kept the most asked Java Spring Interview Questions with their detailed answers. So that you can crack the interview.

GTA 6 document gives a detailed overview of all leaks Sep 08, 2024 am 06:37 AM

On September 3, version 1.5 of the Grand Theft Auto VI document which provides a detailed overview of everything that is known about the game to date. The updated version of the document was announced in a trailer published on X (formerly Twitter).??

Break or return from Java 8 stream forEach? Feb 07, 2025 pm 12:09 PM

Java 8 introduces the Stream API, providing a powerful and expressive way to process data collections. However, a common question when using Stream is: How to break or return from a forEach operation? Traditional loops allow for early interruption or return, but Stream's forEach method does not directly support this method. This article will explain the reasons and explore alternative methods for implementing premature termination in Stream processing systems. Further reading: Java Stream API improvements Understand Stream forEach The forEach method is a terminal operation that performs one operation on each element in the Stream. Its design intention is

TimeStamp to Date in Java Aug 30, 2024 pm 04:28 PM

Guide to TimeStamp to Date in Java. Here we also discuss the introduction and how to convert timestamp to date in java along with examples.

See all articles