pySparkGuide.com

pySparkGuide.com

Reference and tutorials for busy Big Data Professionals.
The finest pySpark resource on the internet!

misc

theory - SQL Expressions vs Dataframe API expressions in pySpark functions
guide - Single node spark notebook setup using docker
guide - Style guide - Chaining functions in python

performance

spark - Spark Adaptive Query Engine (AQE) - all the details you need to know
spark - Spark join strategies
guide - Cache and persist - why and how

pyspark api

guide - Cast string to Date or DateTime using to_date() and to_timestamp()

read and write

guide - JSON - read, set schema and write with pySpark
guide - Delta - read, set schema and write with pySpark
theory - Read and write modes in pySpark
guide - CSV - read, set schema and write with pySpark
theory - Understanding partition discovery and partition read optimization in spark

sql

theory - Note on window restrictions for window functions
guide - GROUP BY in pySpark
guide - CASE in pySpark
guide - Filter rows in pySpark
guide - Window functions with pySpark