How to do big data analysis in python

Data acquisition: public data, Python crawler

There are two main ways to obtain external data. (Recommended learning: Python video tutorial)

The first is to obtain external public data sets. Some scientific research institutions, enterprises, and governments will open some data. You need to go to a specific website. Download this data. These data sets are usually complete and of relatively high quality.

Another way to obtain external data is through crawlers.

For example, you can use a crawler to obtain the recruitment information for a certain position on the recruitment website, crawl the rental information of a certain city on the rental website, crawl the list of movies with the highest ratings on Douban, and obtain the likes ranking on Zhihu, NetEase Cloud music review ranking list. Based on the data crawled from the Internet, you can analyze a certain industry and a certain group of people.

Before crawling, you need to understand some basic knowledge of Python: elements (lists, dictionaries, tuples, etc.), variables, loops, functions...

And, how to use Python libraries (urllib, BeautifulSoup, requests, scrapy) implement web crawlers.

After mastering the basic crawlers, you still need some advanced skills, such as regular expressions, using cookie information, simulated user login, packet capture analysis, building a proxy pool, etc., to deal with the anti-crawler restrictions of different websites. .

Data access: SQL language

When dealing with data within 10,000, Excel has no problem with general analysis. Once the amount of data is large, it will be unable to cope with it. The database can solve this problem very well. And most companies store data in the form of SQL.

SQL, as the most classic database tool, provides the possibility for the storage and management of massive data, and greatly improves the efficiency of data extraction. You need to master the following skills:

Extracting data under specific circumstances

Add, delete, query, and modify database

Grouping and aggregation of data, how to create multiple tables The connection between

Data preprocessing: Python (pandas)

Many times the data we get is not clean, with duplicates, missing data, outliers, etc. Wait, at this time it is necessary to clean the data and process the data that affects the analysis, in order to obtain more accurate analysis results.

For data preprocessing, if you learn how to use pandas (Python package), you will have no problem dealing with general data cleaning. The knowledge points that need to be mastered are as follows:

Selection: Data Access

Missing Value Processing: Delete or fill in missing data rows

Duplicate Value Processing: Duplicate Value Judgment and Delete

Outlier processing: clear unnecessary spaces and extreme, abnormal data

Related operations: descriptive statistics, Apply, histograms, etc.

Merge: conform to various Merge operations of logical relationships

Grouping: data division, executing functions separately, data reorganization

Reshaping: quickly generating pivot tables

Probability theory and statistical knowledge

The knowledge points that need to be mastered are as follows:

Basic statistics: mean, median, mode, percentile, extreme value, etc.

Others Descriptive statistics: skewness, variance, standard deviation, significance, etc.

Other statistical knowledge: population and sample, parameters and statistics, ErrorBar

Probability distribution and hypothesis testing: various Distribution, hypothesis testing process

Other probability theory knowledge: conditional probability, Bayes, etc.

With the basic knowledge of statistics, you can use these statistics to do basic analysis. You can use Seaborn, matplotlib, etc. (python package) to do some visual analysis, through various visual statistical charts, and obtain instructive results.

Python Data Analysis

Master the method of regression analysis. Through linear regression and logistic regression, you can actually perform regression analysis on most data and obtain A relatively accurate conclusion. The knowledge points that need to be mastered in this part are as follows:

Regression analysis: linear regression, logistic regression

Basic classification algorithm: decision tree, random forest...

Basic clustering Class algorithm: k-means...

Feature Engineering Basics: How to use feature selection to optimize the model

Parameter adjustment method: How to adjust parameters to optimize the model

Python data analysis package: scipy, numpy, scikit-learn, etc.

At this stage of data analysis, focus on understanding the regression analysis method. Most problems can be solved. Using descriptive statistical analysis and regression analysis, you can completely get A good analytical conclusion.

Of course, as your practice increases, you may encounter some complex problems, and you may need to understand some more advanced algorithms: classification and clustering.

Then you will know which algorithm model is more suitable for different types of problems. For model optimization, you need to understand how to improve prediction accuracy through feature extraction and parameter adjustment.

You can realize the entire process of data analysis, data mining modeling and analysis through the scikit-learn library in Python.

For more Python related technical articles, please visit the Python Tutorial column to learn!

The above is the detailed content of How to do big data analysis in python. For more information, please follow other related articles on the PHP Chinese website!