Home > Backend Development > Python Tutorial > Intro to Data Analysis using PySpark

Intro to Data Analysis using PySpark

DDD
Release: 2025-01-12 12:14:43
Original
1010 people have browsed it

This tutorial demonstrates PySpark functionality using a World Population dataset.

Preliminary Setup

First, ensure Python is installed. Check your terminal using:

python --version
Copy after login

If not installed, download Python from the official website, selecting the appropriate version for your operating system.

Install Jupyter Notebook (instructions available online). Alternatively, install Anaconda, which includes Python and Jupyter Notebook along with many scientific libraries.

Launch Jupyter Notebook from your terminal:

jupyter notebook
Copy after login

Create a new Python 3 notebook. Install required libraries:

!pip install pandas
!pip install pyspark
!pip install findspark
!pip install pyspark_dist_explore
Copy after login

Download the population dataset (CSV format) from datahub.io and note its location.

Import Libraries and Initialize Spark

Import necessary libraries:

import pandas as pd
import matplotlib.pyplot as plt
import findspark
findspark.init()
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, IntegerType, FloatType, StringType, StructField
from pyspark_dist_explore import hist
Copy after login

Before initializing the Spark session, verify Java is installed:

java -version
Copy after login

If not, install the Java Development Kit (JDK).

Initialize the Spark session:

spark = SparkSession \
    .builder \
    .appName("World Population Analysis") \
    .config("spark.sql.execution.arrow.pyspark.enabled", "true") \
    .getOrCreate()
Copy after login

Verify the session:

spark
Copy after login

If a warning about hostname resolution appears, set SPARK_LOCAL_IP in local-spark-env.sh or spark-env.sh to an IP address other than 127.0.0.1 (e.g., export SPARK_LOCAL_IP="10.0.0.19") before re-initializing.

Data Loading and Manipulation

Load data into a Pandas DataFrame:

pd_dataframe = pd.read_csv('population.csv')
pd_dataframe.head()
Copy after login

Load data into a Spark DataFrame:

sdf = spark.createDataFrame(pd_dataframe)
sdf.printSchema()
Copy after login

Rename columns for easier processing:

sdf_new = sdf.withColumnRenamed("Country Name", "Country_Name").withColumnRenamed("Country Code", "Country_Code")
sdf_new.head(5)
Copy after login

Create a temporary view:

sdf_new.createTempView('population_table')
Copy after login

Data Exploration with SQL Queries

Run SQL queries:

spark.sql("SELECT * FROM population_table").show()
spark.sql("SELECT Country_Name FROM population_table").show()
Copy after login

Data Visualization

Plot a histogram of Aruba's population:

sdf_population = sdf_new.filter(sdf_new.Country_Name == 'Aruba')
fig, ax = plt.subplots()
hist(ax, sdf_population.select('Value'), bins=20, color=['red'])
Copy after login

Intro to Data Analysis using PySpark

This revised response maintains the original structure and content while using slightly different wording and phrasing for a more natural flow and improved clarity. The image remains in its original format and location.

The above is the detailed content of Intro to Data Analysis using PySpark. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Popular Tutorials
More>
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template