This page presents the worked solutions for the statistics tasks, showing how to perform the same analyses in both Python and R.
WarningBefore you proceed…
These are worked solutions. We strongly recommend attempting to debug the Python and R problem scripts in the Tasks section before reviewing these solutions. Learning to debug is a crucial data science skill!
NoteHigh-Performance Alternatives: Polars and DuckDB
The Python solutions below use pandas for seamless integration with seaborn and statsmodels. When working with larger-than-RAM datasets or complex data engineering pipelines, modern alternatives to explore include:
Polars: A fast, multi-threaded DataFrame library built in Rust.
DuckDB: An in-process SQL analytical engine for querying files directly without database overhead.
Task 1: Reproducing the Excel exercise
This exercise involves reading data from an Excel file (ParkRunPerformanceData.xlsx), cleaning it, calculating summaries, and visualizing the distribution of run times.
import pandas as pdimport seaborn as snsimport matplotlib.pyplot as pltimport matplotlib.dates as mdatessns.set_theme(style="whitegrid")# Assigning the path to a variablepath_to_file ="../00_data/ParkRunPerformanceData.xlsx"# Corrected path# Reading data from an Excel filedata = pd.read_excel(path_to_file, sheet_name="Sheet1")# Exploring the dataprint("---\n--- Head of Data ---\n---")
---
--- Head of Data ---
---
Code
print(data.head())
Event Date Run Time (minutes)
0 2015-08-22 20.633333
1 2015-08-15 22.383333
2 2015-08-08 26.966667
3 2015-08-01 22.250000
4 2015-07-25 24.816667
Code
print("\n--- Info ---")
--- Info ---
Code
data.info()
<class 'pandas.DataFrame'>
RangeIndex: 120 entries, 0 to 119
Data columns (total 2 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Event Date 120 non-null datetime64[us]
1 Run Time (minutes) 120 non-null float64
dtypes: datetime64[us](1), float64(1)
memory usage: 2.0 KB
Code
print("\n--- Summary Statistics ---")
--- Summary Statistics ---
Code
print(data.describe())
Event Date Run Time (minutes)
count 120 120.000000
mean 2014-04-07 21:48:00 23.420278
min 2012-08-04 00:00:00 20.633333
25% 2013-09-07 00:00:00 21.783333
50% 2014-04-08 12:00:00 23.075000
75% 2014-11-23 18:00:00 24.820833
max 2015-08-22 00:00:00 31.133333
std NaN 2.059807
Code
# Rename the columnsiflen(data.columns) >=2: data.columns = ["date", "runtime"]# Sorting the datadata_sorted = data.sort_values(by="runtime")print("\n--- Sorted Data (Ascending) ---")
# Nicer Histogramplt.figure(figsize=(10, 6))sns.histplot( data=data, x="runtime", binwidth=1, color="steelblue", edgecolor="white", alpha=0.7)plt.title("Park Run Times Distribution\nRecords from Aug 2012 to Aug 2015", fontsize=15)plt.xlabel("Run time in seconds")plt.ylabel("frequency")plt.xlim(14, 33)
(14.0, 33.0)
Code
plt.axhline(0, color="grey", linewidth=1)plt.figtext(0.8, 0.01, "Source: Andrew Tomlinson", wrap=True, horizontalalignment='center', fontsize=10)plt.show()
Various Outputs from the Excel Exercise (Python)
Code
# Have the run times improved?plt.figure(figsize=(10, 6))data_reg = data.copy()data_reg["date_num"] = mdates.date2num(data_reg["date"])sns.regplot( data=data_reg, x='date_num', y="runtime", scatter_kws={'s':10}, line_kws={'color':'blue'})# Fix x-axis to show datesax = plt.gca()ax.xaxis.set_major_formatter(mdates.DateFormatter('%Y-%m'))plt.title("Run Times over Date")plt.show()
Various Outputs from the Excel Exercise (Python)
Code
# Loading the necessary librarieslibrary(tidyverse)library(readxl)# Assigning the path to a variablepath_to_file <-"../00_data/ParkRunPerformanceData.xlsx"# Corrected path# Reading data from an Excel filedata <-read_excel(path = path_to_file,sheet ="Sheet1",col_types =c("date", "numeric"))# Exploring the datacat("---\n--- Head of Data ---\n---")
tibble [120 × 2] (S3: tbl_df/tbl/data.frame)
$ Event Date : POSIXct[1:120], format: "2015-08-22" "2015-08-15" ...
$ Run Time (minutes): num [1:120] 20.6 22.4 27 22.2 24.8 ...
Code
cat("\n--- Summary Statistics ---")
--- Summary Statistics ---
Code
summary(data)
Event Date Run Time (minutes)
Min. :2012-08-04 00:00:00 Min. :20.63
1st Qu.:2013-09-07 00:00:00 1st Qu.:21.78
Median :2014-04-08 12:00:00 Median :23.07
Mean :2014-04-07 21:48:00 Mean :23.42
3rd Qu.:2014-11-23 18:00:00 3rd Qu.:24.82
Max. :2015-08-22 00:00:00 Max. :31.13
Code
# Rename the columns to ensure consistency and ease manipulationnames(data) <-c("date","runtime")# Sorting the datadata_sorted <- data |>arrange(runtime)cat("\n--- Sorted Data (Ascending) ---")
# Rounding the run times to the nearest minutedata_rounded <- data |>mutate(runtime_mins =round(x = runtime, digits =0))cat("\n--- Data with Rounded Runtimes ---")
# A quick histogramhist(data_rounded$runtime_mins,breaks =14:33, main ="Quick Histogram of Run Times")
Various Outputs from the Excel Exercise (R)
Code
# A nicer histogramdata |>ggplot(aes(x = runtime))+geom_histogram(binwidth =1, col ="white",fill ="steelblue", alpha =0.7)+labs (x ="Run time in seconds",y ="frequency",title ="Park Run Times Distribution",subtitle ="Records from Aug 2012 to Aug 2015",caption ="Source: Andrew Tomlinson")+scale_x_continuous(breaks =14:33)+geom_hline(yintercept =0,linewidth =1,col ="grey30")+theme_minimal()+theme(title =element_text(size =15))
Various Outputs from the Excel Exercise (R)
Code
# Have the run times improved?# A different explorationdata |>ggplot(aes(x = date, y = runtime))+geom_point() +geom_smooth(method ="lm")
Various Outputs from the Excel Exercise (R)
Task 2: Reproducing the SPSS exercise
This exercise reproduces in code an analysis originally carried out in SPSS (RunningData.xlsx), including filtering, grouping, and statistical tests.
import pandas as pdimport seaborn as snsimport matplotlib.pyplot as pltimport scipy.stats as statsimport statsmodels.formula.api as smfsns.set_theme(style="whitegrid")path_to_file ="../00_data/RunningData.xlsx"data = pd.read_excel(path_to_file, sheet_name="Sheet1")# Rename columnsiflen(data.columns) >=5: data.columns = ["position", "time", "age_cat", "gender", "prev_runs"]print("---\n--- Head ---")
---
--- Head ---
Code
print(data.head())
position time age_cat gender prev_runs
0 1 16.900000 30-34 M 60
1 2 17.166667 20-24 M 91
2 3 17.300000 25-29 M 26
3 4 17.366667 40-44 M 14
4 5 17.383333 40-44 M 4
# Comparing timesplt.figure(figsize=(10, 6))sns.boxplot(data=data_adults, x="time", hue="pr_gr")plt.title("Time by Previous Runs Group")plt.show()
Various Outputs from the SPSS Exercise (Python)
Code
# Linear Modeldata_adults["age"] = data_adults["age_cat"].str.extract(r"^(\d{2})").astype(float)model = smf.ols(formula="time ~ age + gender + prev_runs", data=data_adults)results = model.fit()print("\n--- Linear Model Summary ---\n")
# var.equal = TRUE matches the Python version above; R's default is Welch.t.test(times_male_adults, times_female_adults, var.equal =TRUE)
Two Sample t-test
data: times_male_adults and times_female_adults
t = -14.628, df = 522, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-6.622996 -5.054669
sample estimates:
mean of x mean of y
25.15512 30.99395
Code
# Previous runs vs timesdata_adults |>ggplot(aes(x = prev_runs, y = time))+geom_point()+geom_smooth(method ="lm")
Various Outputs from the SPSS Exercise (R)
Code
data_adults |>ggplot(aes(x = prev_runs, y =time, col = gender))+geom_point() +geom_smooth(method ="lm")
Various Outputs from the SPSS Exercise (R)
Code
cat("\n--- Correlation Test (Time vs Previous Runs) ---")
--- Correlation Test (Time vs Previous Runs) ---
Code
cor.test(data_adults$time, data_adults$prev_runs)
Pearson's product-moment correlation
data: data_adults$time and data_adults$prev_runs
t = -5.4024, df = 522, p-value = 1.001e-07
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
-0.3096642 -0.1473580
sample estimates:
cor
-0.2301107
Code
# Finding the median of prev runsmedian_prev_runs <-median(data_adults$prev_runs)cat("\n--- Median Previous Runs ---")
--- Median Previous Runs ---
Code
print(median_prev_runs)
[1] 18
Code
data_adults_pr_gr <- data_adults |>mutate(pr_gr = prev_runs>=median_prev_runs)cat("\n--- Data with pr_gr group ---")
--- Data with pr_gr group ---
Code
print(head(data_adults_pr_gr))
# A tibble: 6 × 6
position time age_cat gender prev_runs pr_gr
<dbl> <dbl> <chr> <chr> <dbl> <lgl>
1 1 16.9 30-34 M 60 TRUE
2 2 17.2 20-24 M 91 TRUE
3 3 17.3 25-29 M 26 TRUE
4 4 17.4 40-44 M 14 FALSE
5 5 17.4 40-44 M 4 FALSE
6 7 18.2 35-39 M 19 TRUE
Code
# a quick visual checkdata_adults_pr_gr |>ggplot(aes(prev_runs,fill = pr_gr))+geom_histogram()
Various Outputs from the SPSS Exercise (R)
Code
# Comparing timesdata_adults_pr_gr |>ggplot(aes(x = time, col = pr_gr))+geom_boxplot()
Various Outputs from the SPSS Exercise (R)
Code
# Linear Model# Extract first two digits of age category for numeric age proxydata_adults_lm <- data_adults |>mutate(age =str_extract(age_cat, '^\\d{2}') |>as.numeric()) my_linear_model <-lm(formula ="time ~ age + gender + prev_runs", data = data_adults_lm) cat("\n--- Linear Model Summary ---")
--- Linear Model Summary ---
Code
summary(my_linear_model)
Call:
lm(formula = "time ~ age + gender + prev_runs", data = data_adults_lm)
Residuals:
Min 1Q Median 3Q Max
-10.3003 -3.3456 -0.3801 2.5654 15.2136
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 30.198335 0.723994 41.711 < 2e-16 ***
age 0.043466 0.018523 2.347 0.0193 *
genderM -5.657628 0.395852 -14.292 < 2e-16 ***
prev_runs -0.035480 0.007473 -4.748 2.66e-06 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 4.473 on 520 degrees of freedom
Multiple R-squared: 0.322, Adjusted R-squared: 0.3181
F-statistic: 82.31 on 3 and 520 DF, p-value: < 2.2e-16