Home Backend Development Python Tutorial Save crawled data to mysql

Save crawled data to mysql

May 28, 2017 am 09:58 AM
mysql

Python and mysql use pymysql for data connection

<span style="color: #0000ff">import<span style="color: #000000"> pymysql



conn =pymysql.connect(host=<span style="color: #800000">'<span style="color: #800000">127.0.0.1<span style="color: #800000">',user=<span style="color: #800000">'<span style="color: #800000">root<span style="color: #800000">',passw<a href="http://www.php.cn/wiki/1360.html" target="_blank">ord</a>=<span style="color: #800000">'<span style="color: #800000">123456<span style="color: #800000">',db=<span style="color: #800000">'<span style="color: #800000">company<span style="color: #800000">',char<a href="http://www.php.cn/code/8209.html" target="_blank">set</a>=<span style="color: #800000">"<span style="color: #800000">utf8<span style="color: #800000">"<span style="color: #000000">)

cur=<span style="color: #000000">conn.cursor()
sql=‘’‘<br><br>’‘’<span style="color: #000000">employee=<span style="color: #000000">cur.execute(sql)
conn.commit()
cur.close()
conn.close()</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>
Copy after login

The basic operation is probably like this
This time we crawled the Taobao product page

The process is omitted and the code is omitted


import requestsimport reimport pymysqldef getHTMLtext(url):    try:
       r=requests.get(url,timeout=100)
       r.raise_for_status()
       r.encoding=r.apparent_encoding       return r.text    except:        return ""def getpage(itl,html):    try:
        plt=re.findall(r'"view_price":"[\d.]*"',html)
        nlt=re.findall(r'"raw_title":".*?"',html)        for i in range(len(plt)):
            price = eval(plt[i].split(':')[1])
            title = eval(nlt[i].split(':')[1])
            itl.append([price, title])    except:       print("")def printgoods(itl):
    tplt = "{:2}\t{:8}\t{:16}"
    print(tplt.format("序号", "价格", "商品名称"))

    count = 0
    conn = pymysql.connect(host='127.0.0.1', user='root', password='123456', db='company',charset="utf8")

    cur = conn.cursor()

    sqlc = '''
                create table coffee(
                id int(11) not null auto_increment primary key,
                name varchar(255) not null,
                price float not null)DEFAULT CHARSET=utf8;                '''

    try:
        A = cur.execute(sqlc)
        conn.commit()        print('成功')    except:        print("错误")    for g in itl:
        count = count + 1
        b=tplt.format(count, g[0], g[1])



        sqla = '''
        insert into  coffee(name,price)
        values(%s,%s);       '''
        try:
            B = cur.execute(sqla,(g[1],g[0]))
            conn.commit()            print('成功')        except:            print("错误")        # save_path = 'D:/taobao.txt'
        # f=open(save_path,'a')
        #        # f.write(b+'\n')
        # f.close()
    conn.commit()
    cur.close()
    conn.close()def main():
    goods="咖啡"
    depth =2
    start_url='https://s.taobao.com/search?q='+goods
    List =[]    for i in range(depth):        try:
            url =start_url +"&amp;s="+ str(i*44)
            html=getHTMLtext(url)
            getpage(List,html)        except:           continue


    print(printgoods(List))    # savefiles(data)main()
Copy after login


##You can see that the required data has been Saved to database

The above is the detailed content of Save crawled data to mysql. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Article Tags

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

PHP's big data structure processing skills PHP's big data structure processing skills May 08, 2024 am 10:24 AM

PHP's big data structure processing skills

How to optimize MySQL query performance in PHP? How to optimize MySQL query performance in PHP? Jun 03, 2024 pm 08:11 PM

How to optimize MySQL query performance in PHP?

How to use MySQL backup and restore in PHP? How to use MySQL backup and restore in PHP? Jun 03, 2024 pm 12:19 PM

How to use MySQL backup and restore in PHP?

How to insert data into a MySQL table using PHP? How to insert data into a MySQL table using PHP? Jun 02, 2024 pm 02:26 PM

How to insert data into a MySQL table using PHP?

What are the application scenarios of Java enumeration types in databases? What are the application scenarios of Java enumeration types in databases? May 05, 2024 am 09:06 AM

What are the application scenarios of Java enumeration types in databases?

How to fix mysql_native_password not loaded errors on MySQL 8.4 How to fix mysql_native_password not loaded errors on MySQL 8.4 Dec 09, 2024 am 11:42 AM

How to fix mysql_native_password not loaded errors on MySQL 8.4

How to use MySQL stored procedures in PHP? How to use MySQL stored procedures in PHP? Jun 02, 2024 pm 02:13 PM

How to use MySQL stored procedures in PHP?

How to create a MySQL table using PHP? How to create a MySQL table using PHP? Jun 04, 2024 pm 01:57 PM

How to create a MySQL table using PHP?

See all articles