Data Manipulation with Pandas

Pandas is the cornerstone library for data manipulation in Python. As a data engineer, you will use it daily to read, filter, transform, aggregate, and write data. This lesson covers DataFrames, reading and writing multiple formats, filtering, grouping, joins, and method chaining.

---

DataFrames: The Core Abstraction

A DataFrame is a two-dimensional, labelled data structure — think of it as a spreadsheet or SQL table in memory.

import pandas as pd

# Create a DataFrame from a dictionary
df = pd.DataFrame({
    "name": ["Alice", "Bob", "Charlie", "Diana"],
    "department": ["Engineering", "Marketing", "Engineering", "Marketing"],
    "salary": [95000, 72000, 105000, 68000],
    "start_date": ["2021-03-15", "2020-07-01", "2019-11-20", "2022-01-10"],
})

print(df)
#       name   department  salary  start_date
# 0    Alice  Engineering   95000  2021-03-15
# 1      Bob    Marketing   72000  2020-07-01
# 2  Charlie  Engineering  105000  2019-11-20
# 3    Diana    Marketing   68000  2022-01-10

Key DataFrame Attributes