Cleaning and Tidying Data in Pandas

on YouTube ↗01h 27m

" summary

Shows concrete cleaning steps using pandas: loading files with read_csv, inspecting df.head/df.info, reshaping with pd.melt() to turn wide tables into long ones, splitting combined labels with str.split() to make separate variables, and turning rows back into columns with pd.pivot_table()/reset_index. Demonstrations use Pew, Billboard, Ebola, Weather and Gapminder datasets, converting 317 songs × 81 columns into roughly 22,000 rows, deduplicating with drop_duplicates, merging on keys, and saving results to CSV.

AI-written summary · about

" chapters (53)

1. 0:00Introduction
2. 0:18Setup: Github Repo, Jupyter Setup
3. 5:35Loading Datasets - panda.read_csv()
4. 7:43Dataset / Dataframe At A Glance
5. 7:53Get First Rows: df.head()
6. 8:58Get Columns: df.columns
7. 9:15Get Index: df.index
8. 9:37Get Body: df.values
9. 10:46Get Shape: df.shape
10. 12:04Get Summarizing Statistics: df.info()
11. 13:12Filtering, Slicing a Dataset / Dataframe
12. 13:25Extract a Single Column: df['col_name']
13. 14:12Dataframe vs Series
14. 14:41Extract N Columns: df[['col1_name', 'col2_name']]
15. 15:51Panda's Version: df.version
16. 16:26Extract Rows: df.iloc
17. 17:30Extract Rows: df.loc vs df.iloc vs df.idx
18. 18:45Extract Rows: df.iloc
19. 19:37Extract Rows: df.ix - Deprecated
20. 20:38Extract Multiple Rows and Columns
21. 22:00Extract Rows using Boolean Subsetting
22. 23:24Extract Rows using Multiple Boolean Subsetting
23. 24:55Cleaning a Dataset / Dataframe
24. 25:38General Issues according to a "Tidy Data" Research Paper
25. 29:45Issue 1: Column Headers are Values and not Variables Names
26. 30:19Load Pew Dataset
27. 32:55Transform Columns into Rows: pd.melt()
28. 36:59Load Billboard Dataset
29. 42:00Issue 2: Multiple Variables are Stored in 1 Column
30. 43:06Load Ebola Dataset
31. 46:22Transform Columns into Rows: pd.melt()
32. 47:14Split Column using String Manipulation through Accessors
33. 51:19Extract Column / Series from Accessor Split: accessor.get()
34. 53:13Add Column to Dataframe
35. 54:13Contracted Form for pd.melt() and Accessor String Manipulation: pd.merge()
36. 56:10Issue 3: Variables Stored in Rows And Columns
37. 56:25Load Weather Dataset
38. 58:30Transform Columns into Rows: pd.melt()
39. 1:01:00Transform Rows into Columns
40. 1:02:00Transform Rows into Columns: pd.pivot() vs pd.pivot_table()
41. 1:04:30Transform Rows into Columns: pd.pivot_table()
42. 1:06:19Flatten nested / hierarchical table: pd.reset_index()
43. 1:07:42Issue 4: Multiple Types of Observational Unit in Same Table (i.e De-nomalized Table)
44. 1:09:43Extract Type Observational Unit in new Dataframe, Drop Duplicates
45. 1:11:30Create "key" for extracted observational unit dataframe
46. 1:12:11Save new dataframe: pd.to_csv()
47. 1:13:22Merge / Join dataframe on common columns
48. 1:16:25Randomly Sample a dataframe
49. 1:17:15Note on Memory Consumption between all 3 dataframes
50. 1:18:25Summary from "Tidy Data" Research Paper
51. 1:20:06Q&A
52. 1:21:21Q&A 1: Simulating R's Chaining in Python
53. 1:24:49Q&A 2: Best Practices on Braquet Notation vs Chaining

" related talks

• 01h 24mIntroduction to Data-Analysis with PandasAlexander CS Hendorf · PyData Berlin 2017
• 01h 31mPandas, Data Wrangling & Data ScienceKrishna Sankar · PyData San Francisco 2016
• 44mIntroduction to PandasJason Myers · PyOhio 2014
• 01h 52mIntroduction to data analytics with pandasQuentin Caudron · PyData Seattle 2017
• 42mUsing Python pandas for scientific ResearchUwe Ziegenhagen · FrOSCon 2016
• 21mHow to use Pandas EfficientlyHossein Mortazavi · PyData Yerevan 2022
• 30mMessy Sensor Data: A Programmer’s Cleaning GuideXavier Ho · PyCon AU 2017
• 35mIdiomatic PandasMatt Harrison · Global Artificial Intelligence Virtual Conference 2022