Data Quality & Validation

Bad data is worse than no data — it leads to wrong decisions, broken dashboards, and lost trust. Data quality must be validated at every stage of your pipeline. This lesson covers schema validation with Pydantic, data quality checks with Great Expectations, data contracts, and monitoring.

---

The Cost of Bad Data

┌──────────────────────────────────────────────────┐
│ Bad data entered at the source                   │
│         ↓                                        │
│ Propagates through the pipeline                  │
│         ↓                                        │
│ Loads into the warehouse                         │
│         ↓                                        │
│ Feeds into dashboards and ML models              │
│         ↓                                        │
│ Business makes wrong decisions                   │
│         ↓                                        │
│ Hours or days to find and fix the root cause     │
└──────────────────────────────────────────────────┘

Rule of thumb: Fix data quality issues as early as possible — the cost of fixing grows exponentially as data moves downstream.

---

Schema Validation with Pydantic

Pydantic validates data at runtime, catching type errors, missing fields, and constraint violations.

from pydantic import BaseModel, Field, field_validator, EmailStr
from datetime import date
from typing import Optional