In-depth analysis of python data mining Json structure analysis

This article analyzes and summarizes the relevant knowledge points of Python data mining and Json structure analysis through examples. Friends who are interested in this can refer to it.

json is a lightweight data exchange format, which can also be said to be a configuration file format

Files in this format are what we often encounter in data processing

Python provides a built-in module json, which only needs to be imported before use

You can view the json help document through the help function

The commonly used methods of json include load, loads, dump and dumps. These are all beginners in python and I will not do it. Too many explanations

json can be used in conjunction with a database, which is very useful when processing large amounts of data in the future

Now we will formally use data mining to process json files

Many websites now use Ajax, so generally many of them are XHR files

Here I want to use a map website to demonstrate

We use the browser The debugging obtained the relevant url

https://ditu.amap.com/service/poiInfo?id=B001B0IZY1&query_type=IDQ

Next we simulate the browser through the get method in the requests module Issue an http request and return the result object

The code is as follows

# coding=utf-8
__Author__ = "susmote"

import requests
url = "https://ditu.amap.com/service/poiInfo?id=B001B0IZY1&query_type=IDQ"

resp = requests.get(url)
print(resp.text[0:200])

The result of running in the terminal is as follows

The data has been obtained, but in order to use the data next, we need to use the json module to analyze the data

The code is as follows

import requests
import json

url = "https://ditu.amap.com/service/poiInfo?id=B001B0IZY1&query_type=IDQ"

resp = requests.get(url)

json_dict = json.loads(resp.text)

print(type(json_dict))

print(json_dict.keys())

Let’s briefly talk about the above code:

Import the json module, then call the loads method, and pass the returned text as the parameter of the method

In The running result in the terminal is as follows

It can be seen that the result of the conversion is a dictionary corresponding to the json string, because type (json_dict) returns

Because the object is a dictionary, we can call the dictionary method. Here we call the keys method

The result returns three keys, namely status, searcOpt, data

Let’s check the data in the data key

import requests
import json

url = "https://ditu.amap.com/service/poiInfo?id=B001B0IZY1&query_type=IDQ"

resp = requests.get(url)

json_dict = json.loads(resp.text)

print(json_dict['data'])

Run this code in the terminal

You can see that there is a lot of data we need, such as

, which are not marked one by one. By comparing with what is displayed on the web page, we can know which ones It’s useful

Let’s get useful information through the code and output it clearly

# coding=utf-8
__Author__ = "susmote"

import requests
import json

url = "https://ditu.amap.com/service/poiInfo?id=B001B0IZY1&query_type=IDQ"

resp = requests.get(url)

json_dict = json.loads(resp.text)

data_dict = json_dict['data']

data_list = data_dict['poi_list']

dis_data = data_list[0]

print('城市： ', dis_data['cityname'])
print('名称： ', dis_data['name'])
print('电话： ', dis_data['tel'])
print('区号： ', dis_data['areacode'])
print('地址： ', dis_data['address'])
print('经度： ', dis_data['longitude'])
print('纬度： ', dis_data['latitude'])

Because what is returned is a dictionary, through the study of the file structure, the dictionary is nested with lists, and the lists are nested with dictionaries. Through layer-by-layer unnesting, the data is successfully obtained.

Here are the steps Listed separately, so you will see it more clearly

Let’s run the program through the terminal to get the information we want

Isn’t it very simple? Yes, this program can be used as a template. When obtaining information from other places, you only need to change a URL.

For example, the following examples

Peking University

Or Tencent Building

Data mining is endless. I hope everyone can analyze the data more and find the data you want.

Related recommendations:

How to process Python data numpy.median

##

The above is the detailed content of In-depth analysis of python data mining Json structure analysis. For more information, please follow other related articles on the PHP Chinese website!