Exploratory Data Analysis

Exploratory Data Analysis (EDA) is the process of investigating a dataset to discover patterns, spot anomalies, test hypotheses, and check assumptions — primarily through statistical summaries and visualisations. Coined by the statistician John Tukey in the 1970s, EDA is the bridge between raw data and formal modelling.

---

Why EDA Matters

Before building any model, you need to understand your data:

• What does the data look like? What are the shapes and types?
• Are there missing values, duplicates, or outliers?
• What are the distributions of individual variables?
• Are there relationships between variables?
• What patterns and trends exist?
• What features are most relevant to the problem?

Skipping EDA is like prescribing medicine without diagnosing the patient.

---

The EDA Process

Step 1: Understand the Data Structure

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load the dataset
df = pd.read_csv('dataset.csv')

# Basic information
print(f"Shape: {df.shape}")       # (rows, columns)
print(f"Columns: {df.columns.tolist()}")
print(f"Data types:\n{df.dtypes}")

# First few rows
df.head()

# Detailed info
df.info()

Step 2: Statistical Summary

# Numeric summary
df.describe()

# Include non-numeric columns
df.describe(include='all')