Home > Java > javaTutorial > How does a Java crawler implement Jsoup using the dom method to traverse the Document object?

How does a Java crawler implement Jsoup using the dom method to traverse the Document object?

王林
Release: 2023-05-14 14:19:12
forward
1555 people have browsed it

先给出网页地址:

https://wall.alphacoders.com/featured.php?lang=Chinese
Copy after login

主要步骤:

利用Jsoup的connect方法获取Document对象

String html = "https://wall.alphacoders.com/featured.php?lang=Chinese";
Document doc = Jsoup.connect(html).get();
Copy after login

内容过长,就不再显示。

我们以这部分为例:

<ul class="nav nav-pills"> 
    <li><a href="https://alphacoders.com/site/about-us" rel="external nofollow"  rel="external nofollow"  >About Us</a></li> 
    <li><a href="https://alphacoders.com/site/faq" rel="external nofollow"  rel="external nofollow"  >FAQ</a></li> 
    <li><a href="https://alphacoders.com/site/privacy" rel="external nofollow"  rel="external nofollow"  >Privacy Policy</a></li> 
    <li><a href="https://alphacoders.com/site/tos" rel="external nofollow"  rel="external nofollow"  >Terms Of Service</a></li> 
    <li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow"  rel="external nofollow"  >Acceptable Use</a></li> 
    <li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow"  rel="external nofollow"  >Etiquette</a></li> 
    <li><a href="https://alphacoders.com/site/advertising" rel="external nofollow"  rel="external nofollow"  >Advertise With Us</a></li> 
    <li><a id="change_consent">Change Consent</a></li> 
</ul>
Copy after login

我们先找到所有的ul:

Elements elements = doc.getElementsByTag("ul");
Copy after login

输出如下:

<ul class="nav navbar-nav center"> 
 <li> <a title="Submit Wallpapers" href="https://alphacoders.com/site/submit-wallpaper" rel="external nofollow" ><i class="el el-circle-arrow-up"></i> 提交</a> </li> 
 <li> <a href="https://alphacoders.com/contest" rel="external nofollow" ><i class="el el-gift"></i> 精美奖品</a> </li> 
</ul>
<ul class="nav navbar-nav navbar-right center"> 
 <li> <a href="language.php?lang=Chinese" rel="external nofollow" > <img src="https://cache.yisu.com/upload/information/20220517/112/2427.png" alt="Chinese-flag">   中文    </a> </li> 
 <li> <a  href="https://alphacoders.com/users/login" rel="external nofollow" ><i class="el el-user"></i> 登录</a> </li> 
 <li> <a href="https://alphacoders.com/users/register" rel="external nofollow" ><i class="el el-edit"></i> 注册</a> </li> 
</ul>
<ul class="pagination"> 
 <li class="active"><a id="prev_page" href="#" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >&lt; 上一页</a></li> 
 <li class="active"><a>1</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >2</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=3" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >3</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=4" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >4</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=5" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >5</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=6" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >6</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=7" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >7</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=8" rel="external nofollow" >8</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=9" rel="external nofollow" >9</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=10" rel="external nofollow" >10</a></li> 
 <li><a>...</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=319" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >319</a></li> 
 <li><a id="next_page" href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >下一页 &gt;</a></li> 
</ul>
<ul class="pagination"> 
 <li class="active"><a href="#" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >&lt; 上一页</a></li> 
 <li class="active"><a>1</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >2</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=3" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >3</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=4" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >4</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=5" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >5</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=6" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >6</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=7" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >7</a></li> 
 <li><a>...</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=319" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >319</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >下一页 &gt;</a></li> 
</ul>
<ul class="pagination"> 
 <li class="active"><a href="#" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >&lt;&lt; </a></li> 
 <li class="active"><a href="#" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >&lt; 上一页</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >下一页 &gt;</a></li> 
 <li><a title="末页 (319)" href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=319" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" > &gt;&gt;</a></li> 
</ul>
<ul class="pagination"> 
 <li class="active"><a href="#" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >1</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=2" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >2</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=3" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >3</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=4" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >4</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=5" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >5</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=6" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >6</a></li> 
 <li><a href="https://wall.alphacoders.com/featured.php?lang=Chinese&amp;page=7" rel="external nofollow"  rel="external nofollow"  rel="external nofollow" >7</a></li> 
</ul>
<ul class="nav nav-pills"> 
 <li><a href="https://alphacoders.com/site/about-us" rel="external nofollow"  rel="external nofollow"  >About Us</a></li> 
 <li><a href="https://alphacoders.com/site/faq" rel="external nofollow"  rel="external nofollow"  >FAQ</a></li> 
 <li><a href="https://alphacoders.com/site/privacy" rel="external nofollow"  rel="external nofollow"  >Privacy Policy</a></li> 
 <li><a href="https://alphacoders.com/site/tos" rel="external nofollow"  rel="external nofollow"  >Terms Of Service</a></li> 
 <li><a href="https://alphacoders.com/site/acceptable_use" rel="external nofollow"  rel="external nofollow"  >Acceptable Use</a></li> 
 <li><a href="https://alphacoders.com/site/etiquette" rel="external nofollow"  rel="external nofollow"  >Etiquette</a></li> 
 <li><a href="https://alphacoders.com/site/advertising" rel="external nofollow"  rel="external nofollow"  >Advertise With Us</a></li> 
 <li><a id="change_consent">Change Consent</a></li> 
</ul>
Copy after login

可以发现class为"nav nav-pills"的只有一个,我们找到它:

Elements elements = doc.getElementsByTag("ul");
        //System.out.println(elements);
Element tempElement = null;
for(Element element : elements) {
    if (element.className().equals("nav nav-pills")) {
        tempElement = element;
        //System.out.println(element.className());
        break;
    }
}
Copy after login

循环遍历这个ul,输出其中每一个li里每一个a的href和rel属性:

Elements li = tempElement.getElementsByTag("li");
for(Element element : li) {
    Elements element2 = element.getElementsByTag("a");
    for(Element element3 : element2) {
        String hrefString = element3.attr("href");
        String relString = element3.attr("rel");
        if(hrefString != "" && relString != "") {
            System.out.println("href=" + hrefString + " rel="external nofollow"  rel="external nofollow"  " + "rel=" + relString);
        }
    }
}
Copy after login

最终结果:

href=https://alphacoders.com/site/about-us rel=nofollow
href=https://alphacoders.com/site/faq rel=nofollow
href=https://alphacoders.com/site/privacy rel=nofollow
href=https://alphacoders.com/site/tos rel=nofollow
href=https://alphacoders.com/site/acceptable_use rel=nofollow
href=https://alphacoders.com/site/etiquette rel=nofollow
href=https://alphacoders.com/site/advertising rel=nofollow

完整代码:

import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import java.io.IOException;
import org.jsoup.Jsoup;


/** 
 * @ClassName: Jsoup_Test
 * @description: 
 * @author: KI
 * @Date: 2020年8月17日 下午8:15:14
 */

public class Jsoup_Test {

    public static void main(String[] args) throws IOException {
        // TODO 自动生成的方法存根
        
        String html = "https://wall.alphacoders.com/featured.php?lang=Chinese";
        Document doc = Jsoup.connect(html).get();
        
        System.out.println(doc);
        Elements elements = doc.getElementsByTag("ul");
        //System.out.println(elements);
        Element tempElement = null;
        for(Element element : elements) {
            if (element.className().equals("nav nav-pills")) {
                tempElement = element;
                //System.out.println(element.className());
                break;
            }
        }
        System.out.println(tempElement);
        Elements li = tempElement.getElementsByTag("li");
        for(Element element : li) {
            Elements element2 = element.getElementsByTag("a");
            for(Element element3 : element2) {
                String hrefString = element3.attr("href");
                String relString = element3.attr("rel");
                if(hrefString != "" && relString != "") {
                    System.out.println("href=" + hrefString + " rel="external nofollow"  rel="external nofollow"  " + "rel=" + relString);
                }
            }
        }        

    }

}
Copy after login

The above is the detailed content of How does a Java crawler implement Jsoup using the dom method to traverse the Document object?. For more information, please follow other related articles on the PHP Chinese website!

Related labels:
source:yisu.com
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Popular Tutorials
More>
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template