What does data preprocessing include?

Data preprocessing content: 1. Data review, which can be divided into four aspects: accuracy review, applicability review, timeliness review and consistency review; 2. Data screening, which analyzes the issues found during the review process. Errors should be corrected as much as possible; 3. Data sorting, arrange the data in a certain order.

The operating environment of this tutorial: Windows 7 system, Dell G3 computer.

Data preprocessing refers to some processing of data before the main processing. For example, before most geophysical area observation data are converted or enhanced, the irregularly distributed measurement network is first converted into a regular network through interpolation to facilitate computer calculations. In addition, for some profile measurement data, such as seismic data preprocessing includes vertical stacking, rearrangement, trace addition, editing, resampling, multi-channel editing, etc.

Data preprocessing refers to the necessary processing such as review, screening, sorting, etc. before classifying or grouping the collected data.

Preprocessing content

1. Data review

Statistical data obtained from different channels , differing in the content and methods of review.

The original data should be reviewed mainly from two aspects: completeness and accuracy. The completeness audit mainly checks whether there are any omissions in the units or individuals that should be investigated, and whether all investigation items or indicators are completed completely. Accuracy review mainly includes two aspects: first, checking whether the data materials truly reflect the objective actual situation and whether the content is consistent with reality; second, checking whether the data has errors and whether the calculations are correct, etc. The main methods for reviewing data accuracy include logical checks and calculation checks. Logical inspection is mainly to review whether the data is logical, whether the content is reasonable, and whether there are any conflicts between items or figures. This method is mainly suitable for reviewing qualitative (quality) data. Calculation check is to check whether there are any errors in the calculation results and calculation methods of each data in the questionnaire. It is mainly used for the review of quantitative (numeric) data.

For secondary information obtained through other channels, in addition to reviewing its completeness and accuracy, we should also focus on reviewing the applicability and timeliness of the data. Secondary data can come from a variety of sources, and some data may have been obtained through special surveys for specific purposes, or have been processed according to the needs of specific purposes. For users, they should first clarify the source of the data, the caliber of the data, and the relevant background information in order to determine whether the data meets the needs of their own analysis and research, whether it needs to be reprocessed, etc., and they cannot blindly copy it. In addition, the timeliness of the data must be reviewed. For some time-sensitive issues, if the data obtained is too late, the significance of the research may be lost. In general, the most recent statistics should be used whenever possible. After the data is reviewed and confirmed to be suitable for actual needs, further processing is necessary.

The content of data review mainly includes the following four aspects:

• Accuracy review. It mainly checks the data from the perspective of authenticity and accuracy of the data. The focus of the review is to check the errors that occurred during the investigation process.

• Suitability review. Mainly based on the purpose of the data, check the extent to which the data explanation explains the problem. Specifically, it includes whether the data matches the survey topic, the definition of the overall target, and the explanation of the survey items.

• Timely review. The main purpose is to check whether the data is submitted according to the prescribed time. If it is not submitted according to the prescribed time, it is necessary to check the reason for not submitting it in time.

• Consistency review. The main purpose is to check whether the data is comparable in different regions or countries and in different time periods.

2. Data filtering

Errors found during the review process should be corrected as much as possible. After the investigation, when the errors found in the data cannot be corrected, or some data do not meet the requirements of the investigation and cannot be made up, the data needs to be screened. Data screening includes two aspects: one is to remove some data that does not meet the requirements or data with obvious errors; the other is to screen out the data that meets certain specific conditions and remove the data that does not meet the specific conditions. Data screening is very important in market research, economic analysis, and management decision-making.

3. Data sorting

Data sorting is to arrange the data in a certain order so that researchers can find some obvious characteristics or trends and find solutions to problems by browsing the data. clues. In addition, sorting can also help to check and correct errors in data and provide a basis for reclassification or grouping. In some cases, sorting itself is one of the purposes of analysis. Sorting can be easily accomplished with the help of a computer.

For categorical data, if it is alphabetic data, sorting can be divided into ascending order and descending order, but ascending order is more commonly used because ascending order is the same as the natural arrangement of letters; if it is Chinese character data, there are many sorting methods. , for example, sorting by the first pinyin letter of Chinese characters, which is exactly the same as sorting letter-type data, or sorting by strokes, in which there are also ascending and descending orders according to the number of strokes. Alternately using different sorting methods is very useful in the process of checking and correcting Chinese character data.

For numerical data, there are only two kinds of sorting, namely ascending and descending. Sorted data are also called ordinal statistics.

For more related knowledge, please visit the FAQ column!

The above is the detailed content of What does data preprocessing include?. For more information, please follow other related articles on the PHP Chinese website!