Intro to Data Analysis using PySpark

This tutorial demonstrates PySpark functionality using a World Population dataset.

Preliminary Setup

First, ensure Python is installed. Check your terminal using:

python --version

If not installed, download Python from the official website, selecting the appropriate version for your operating system.

Install Jupyter Notebook (instructions available online). Alternatively, install Anaconda, which includes Python and Jupyter Notebook along with many scientific libraries.

Launch Jupyter Notebook from your terminal:

jupyter notebook

Create a new Python 3 notebook. Install required libraries:

!pip install pandas
!pip install pyspark
!pip install findspark
!pip install pyspark_dist_explore

Download the population dataset (CSV format) from datahub.io and note its location.

Import Libraries and Initialize Spark

Import necessary libraries:

import pandas as pd
import matplotlib.pyplot as plt
import findspark
findspark.init()
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, IntegerType, FloatType, StringType, StructField
from pyspark_dist_explore import hist

Before initializing the Spark session, verify Java is installed:

java -version

If not, install the Java Development Kit (JDK).

Initialize the Spark session:

spark = SparkSession \
    .builder \
    .appName("World Population Analysis") \
    .config("spark.sql.execution.arrow.pyspark.enabled", "true") \
    .getOrCreate()

Verify the session:

spark

If a warning about hostname resolution appears, set SPARK_LOCAL_IP in local-spark-env.sh or spark-env.sh to an IP address other than 127.0.0.1 (e.g., export SPARK_LOCAL_IP="10.0.0.19") before re-initializing.

Data Loading and Manipulation

Load data into a Pandas DataFrame:

pd_dataframe = pd.read_csv('population.csv')
pd_dataframe.head()

Load data into a Spark DataFrame:

sdf = spark.createDataFrame(pd_dataframe)
sdf.printSchema()

Rename columns for easier processing:

sdf_new = sdf.withColumnRenamed("Country Name", "Country_Name").withColumnRenamed("Country Code", "Country_Code")
sdf_new.head(5)

Create a temporary view:

sdf_new.createTempView('population_table')

Data Exploration with SQL Queries

Run SQL queries:

spark.sql("SELECT * FROM population_table").show()
spark.sql("SELECT Country_Name FROM population_table").show()

Data Visualization

Plot a histogram of Aruba's population:

sdf_population = sdf_new.filter(sdf_new.Country_Name == 'Aruba')
fig, ax = plt.subplots()
hist(ax, sdf_population.select('Value'), bins=20, color=['red'])

This revised response maintains the original structure and content while using slightly different wording and phrasing for a more natural flow and improved clarity. The image remains in its original format and location.

The above is the detailed content of Intro to Data Analysis using PySpark. For more information, please follow other related articles on the PHP Chinese website!