Data Analysis and Mining

Baidu MTC is the industry's leading mobile application testing service platform, providing solutions to the cost, technology and efficiency issues faced by developers in mobile application testing. At the same time, industry-leading Baidu technology is shared, and the authors come from Baidu employees and industry leaders.

1. Overview

1.1 Overview of User Research

The key to the success of mobile apps lies in marketing and product design. The core of data analysis and mining solutions is the process of marketing. Customer positioning and user experience improvement during product design. Providing the required products and services to target users is the secret to the success of any mobile APP. How to find target customers and understand users' product needs requires the power of data analysis and mining. Whether it is customer positioning or user experience, in the final analysis it is still user research. At this point, the success of mobile APP products is no different from any other type of product.

User research can be carried out from two different dimensions: qualitative analysis and quantitative analysis: Qualitative analysis is a method of discovering new things from small-scale data samples, mainly used in user experience surveys; quantitative analysis uses big data A method of testing and proving something with a large amount of samples, mainly used in user behavior data analysis.

1.2 Data analysis and mining process specifications

The construction of data analysis and mining systems is different from the construction of traditional business operation systems and has its own characteristics and rules. Data analysis and mining are an important part of Knowledge-Discovery in Databases (KDD). KDD is a non-trivial process of identifying effective, novel, potentially useful, and ultimately understandable patterns from data sets. process.
Cross-industry standard process for data mining (CRISP-DM: cross-industry standard process for data mining) is a data analysis and mining process that occupies a leading position in the KDD process model, with an adoption rate of nearly 60%, and was jointly drafted by EU agencies Model. CRISP-DM includes 6 different links, as shown in the figure below:

1. Business Understanding:
The initial stage focuses on understanding the project goals and understanding the requirements from a business perspective. Translate this knowledge into a definition of the data mining problem and a preliminary plan for accomplishing the goals.
2. Data Understanding:
The data understanding stage starts from the initial data collection and goes through some active processing. The purpose is to become familiar with the data, identify data quality problems, discover the internal properties of the data for the first time, or It is to detect interesting subsets to form hypotheses of implicit information.
3. Data Preparation:
The data preparation phase includes all activities to construct the final data set from unprocessed data. These data will be the input values to the model tool. The tasks in this phase can be performed multiple times without any prescribed order. Tasks include selection of tables, records, and attributes, as well as transforming and cleaning data for model tools.
4. Data modeling (Modeling):
At this stage, different model technologies can be selected and applied, and the model parameters are adjusted to optimal values. Generally, there are techniques that can solve the same type of data mining problems. Some technologies have special requirements for data formation, so they often need to jump back to the data preparation stage.
5. Model Evaluation:
At this stage, you have established a high-quality display model from the perspective of data analysis. Before starting the final deployment of the model, it is important to thoroughly evaluate the model, review the steps to construct the model, and ensure that the model can accomplish the business goals. The key purpose of this stage is to determine whether there are important business issues that have not been adequately considered. At the end of this phase, a decision on the use of the data mining results must be reached.
6. Model Deployment:
Usually, the creation of a model is not the end of the project. The role of the model is to find knowledge from the data, and the acquired knowledge needs to be reorganized and presented in a way that is user-friendly. Depending on the needs, this stage can produce simple reports or implement a more complex and repeatable data mining process. In many cases, this phase is undertaken by the customer rather than the data analyst.

2. User behavior data analysis

2.1 Objectives

User behavior data refers to the interactive behavior information between users and mobile APP applications. It is the quantitative analysis part of the user research dimension. Through Analyze the user's login and operation logs to obtain the user's usage information of mobile APP products, user equipment, network environment and other information.

2.2 Method

Obtaining user behavior data is usually carried out by burying data. By recording the user’s detailed operation log, we can understand the detailed interaction behavior between the user and the product, as well as the device and network environment when the user accesses the mobile APP. information. The traditional data burying method requires enterprises to develop their own information collection programs and log processing programs. The implementation cost and development workload are specific. If it is compatible with platform differences, the cost will be greater, so it is not suitable for emerging mobile APPs. The analysis of user behavior data can be carried out using a mature data statistical analysis platform.

2.3 Tools

Baidu Mobile Statistics Platform is a professional mobile APP statistical analysis tool launched by Baidu, supporting ios and android platforms. Developers can easily embed the statistics SDK to achieve comprehensive monitoring of mobile applications, grasp product performance in real time, and gain accurate insight into user behavior.
Baidu mobile statistics platform provides powerful application statistical analysis functions for mobile APPs, including:
1. Traffic sources: channel traffic comparison, segmented channel analysis, accurate monitoring of different promotion position data, and real-time knowledge of channel contribution;
2. Audience insights: Based on Baidu’s massive data accumulation, multi-dimensional analysis and presentation of user portrait information;
3. Terminal analysis: Device distribution is clear at a glance (device model, brand, operating system, resolution, networking method, operation Business, etc.);
Baidu mobile statistics function interface is shown in the figure below:

2.4 Output

The result of user behavior data analysis is user role portrait, building user label model, user label The acquisition of data mainly relies on data mining algorithms. The composition of the label system is different for different industries, different businesses, and different users. It requires a more professional industry user portrait model, which will not be discussed too much here. An example of user portrait output results is shown in the figure below:

3. User experience data analysis

3.1 Objectives

For a mobile APP to be successful, in addition to satisfying In addition to user functional needs, a good user experience must also be provided. User experience refers to how a product connects and functions with the outside world, that is, how people "contact" and "use" the product. User experience forms the user's overall impression of the company or product, defines the difference between the company or product and its competitors, and determines whether the user will come back again. High-quality user experience is an important asset of a company or product, and can bring an increase in return on investment (ROI) and user conversion rate (conversion rate) to the company.

3.2 Method

The prerequisite for improving user experience is to obtain user experience data. User experience data can be obtained through traditional direct contact with users, or through remote remote online surveys through the Internet mode. The two complement each other and complement each other. The direct contact user model is conducted through user interviews and on-site surveys, with sufficient communication and significant results. However, the selection of target research objects, communication costs and sample size are limited by time and financial investment. The Internet remote remote online survey model realizes the onlineization of offline questions. Through online Q&A, costs can be saved and the sample size can be expanded. It is a useful supplement to the direct contact with users model. A comparison of the main features of the two is shown in the figure below:

3.3 Tools

Baidu crowd testing platform is an extension and typical application of the crowdsourcing model developed by Baidu in software and product testing. It will The relevant testing work of enterprise products is completed by the public in the online community. It is a task crowdsourcing platform that not only serves Baidu's own products, but also provides services to the public. The purpose of the Baidu public testing platform is to use the public's testing capabilities and testing resources to complete a large workload of product experience in a short period of time, and to ensure quality. The experience results will be fed back to the platform as soon as possible, and then the platform managers will report the information Collect and hand it over to developers, so that product quality and user experience can be improved from the user's perspective.
Baidu public testing platform mainly provides the following types of test tasks:

Quick judgment tasks: Generally, they are simple multiple-choice questions, and users can quickly complete the judgment.
Questionnaire task: Users only need to complete the online questionnaire to get the corresponding gift certificate reward;
Product fault finding task: experience a new product, submit a BUG of the product or propose the product suggestions for improvements.
Special tasks: Enterprises can set special tasks based on specific purposes, such as the current creative solicitation task for Suntech Educational Institutions.
Field research task: Research object recruitment project, by initiating field research tasks, recruit qualified research objects and participate in on-site communication with users.

The operation interface on the homepage of Baidu Public Testing Platform is as shown below: