Learn to Split in Training and Testing Data from a Dataset Using Python

Summary

This article teaches you how to divide a dataset into training and testing data and save this division in a .pkl file, essential for training and evaluating Machine Learning models in an organized way. The process uses the sklearn and pickle libraries, allowing you to reuse the processed data in future projects. This article is the next step in a series of tutorials on data preprocessing.

Main Topics Covered:

• Notebook preparation on Google Colab
• Division of the dataset into training and testing data
• Detailed explanation of Python code for division
• Saving the split to a .pkl file using pickle
• Advantages of saving processed data for future use

Important: To follow this article, first read the articles below in the suggested sequence. Each article provides the foundation you need to understand the next, ensuring you understand the entire workflow up to this point.

Article 1: Applying Machine Learning: A Guide to Getting Started as Models in Classification

Article 2: Exploring Classification in Machine Learning: Types of Variables

Article 3: Exploring Google Colab: Your Ally for Coding Machine Learning Models

Article 4: Exploring Data with Python on Google Colab: A Practical Guide Using the adult.csv Dataset

Article 5: Demystifying Predictor and Class Division and Categorical Attribute Handling with LabelEncoder and OneHotEncoder

Article 6: Data Scaling: The Foundation for Efficient Models

Introduction

In this article, you will learn how to divide a dataset into training and testing, as well as saving this division in a .pkl file. This process is essential to ensure a clean separation between the data that will be used to train the model and that that will be used to evaluate its performance.

Starting the process in Google Colab

First of all, access this notebook link and select File > Save a copy to Drive. Remember that the dataset (adult.csv) needs to be loaded again with each new post (more information in Article 4 above), as each tutorial creates a new notebook, adding only the necessary code presented in this article, but the notebook is with all the code generated so far. A copy of the notebook will be saved on Google Drive, within the Colab Notebooks folder, keeping the process organized and continuous.

Why split the dataset into training and testing?

Dividing the dataset is a fundamental step in any Machine Learning project, as it allows the model to "learn" from a part of the data (training) and then be evaluated on new data, never seen before (testing). This practice is essential to measure the generalization of the model. To facilitate monitoring, we will use the following variables:

• X_adult_treinamento: training predictor variables
• X_adult_teste: test predictor variables
• y_adult_treinamento: training target variable
• y_adult_teste: test target variable

Python code to split the dataset

Below is the Python code to perform the split between training and testing data:

from sklearn.model_selection import train_test_split

X_adult_treinamento, X_adult_teste, y_adult_treinamento, y_adult_teste = train_test_split(X_adult, y_adult, test_size=0.2, random_state=0)

# Dados para o treinamento
X_adult_treinamento.shape, y_adult_treinamento.shape

# Dados para o teste
X_adult_teste.shape, y_adult_teste.shape

The figure below shows the previous code with its outputs after execution.

Explanation of the Code:

train_test_split: Function from the sklearn library that splits the dataset.
test_size=0.2: Indicates that 20% of the data will be reserved for testing, and the remaining 80% for training.
random_state=0: Ensures that the division is always the same, generating consistent results for each run.
shape: Checks the shape of the data after splitting to confirm that the splitting occurred correctly.

Saving the split to a .pkl file

To make work easier and ensure consistency between different runs, we will save the training and testing variables in a .pkl file. This makes it possible to reuse the data whenever necessary, without having to do the division again.

Code to save variables using pickle:

import pickle
with open('adult.pkl', mode='wb') as fl:
  pickle.dump([X_adult_treinamento, y_adult_treinamento, X_adult_teste, y_adult_teste], fl)

To view the adult.pkl file on the notebook, simply click on the folder icon on the left side as shown in the figure below.

Explanation of the Code:

pickle: Python library used to serialize objects, allowing you to save complex variables in files.
dump: Saves the variables in a file called adult.pkl. This file will be read in the future to load the dataset divided into training and testing, optimizing the workflow.

Conclusion

In this article, you learned how to split a dataset into training and testing data and save it in a .pkl file. This process is fundamental in Machine Learning projects, ensuring an organized and efficient structure. In the next article, we will cover the creation of models, starting with the Naive Bayes algorithm, using the adult.pkl file to continue development.

Books I recommend

1. Practical Statistics for Data Scientists
2. Introduction to Computing Using Python
3. 2041: How Artificial Intelligence Will Change Your Life in the Next Decades
4. Intensive Python Course
5. Understanding Algorithms. An Illustrated Guide for Programmers and Others Who Are Curious
6. Artificial Intelligence - Kai-Fu Lee
7. Introduction to Artificial Intelligence - A Non-Technical Approach - Tom Taulli

New Kindles

I did a detailed analysis of the new Kindles launched this year, highlighting their main innovations and benefits for digital readers. Check out the full text at the following link: The Fascinating World of Digital Reading: Advantages of Having a Kindle.

Amazon Prime

Joining Amazon Prime offers a series of advantages, including unlimited access to thousands of films, series and music, as well as free shipping on millions of products with fast delivery. Members also enjoy exclusive offers, early access to promotions and benefits on services such as Prime Video, Prime Music and Prime Reading, making the shopping and entertainment experience much more convenient and rich.

If you are interested, use the following link: AMAZON PRIME, which helps me continue to promote artificial intelligence and computer programming.

The above is the detailed content of Learn to Split in Training and Testing Data from a Dataset Using Python. For more information, please follow other related articles on the PHP Chinese website!