Testing and Validation (pyATS, Batfish)

Automation without testing is dangerous — you can misconfigure thousands of devices as quickly as one. This lesson covers testing and validation tools that ensure your automation changes are correct before they are deployed and verified after deployment.

---

Why Test Network Automation?

Risk	Consequence
Untested config push	Outage affecting thousands of users
Typo in a template	Wrong IP address, broken routing
Missing rollback plan	Cannot recover from a failed change
No pre-change validation	Deploy a change that conflicts with existing config
No post-change validation	Assume the change worked without verification

The Testing Pyramid for Network Automation

          ┌─────────────┐
          │  Integration │   Test against real/virtual devices
          │    Tests     │   (pyATS, robot framework)
          ├─────────────┤
          │  Simulation  │   Model the network and analyse
          │    Tests     │   (Batfish, GNS3)
          ├─────────────┤
          │    Unit      │   Test templates, data, logic
          │    Tests     │   (pytest, yamllint, j2lint)
          └─────────────┘

---

Unit Testing

Linting Data Files

# Validate YAML syntax
pip install yamllint
yamllint host_vars/

# Validate Jinja2 templates
pip install j2lint
j2lint templates/

Testing Jinja2 Templates with pytest

# test_templates.py
import pytest
from jinja2 import Environment, FileSystemLoader
import yaml

@pytest.fixture
def env():
    return Environment(
        loader=FileSystemLoader("templates"),
        trim_blocks=True,
        lstrip_blocks=True,
    )

def test_router_template_renders_hostname(env):
    template = env.get_template("router.j2")
    variables = {"hostname": "test-rtr-01", "interfaces": [], "ntp_servers": []}
    result = template.render(variables)
    assert "hostname test-rtr-01" in result

def test_router_template_renders_interfaces(env):
    template = env.get_template("router.j2")
    variables = {
        "hostname": "test-rtr-01",
        "interfaces": [
            {
                "name": "GigabitEthernet0/0",
                "description": "Test",
                "ip": "10.0.0.1",
                "mask": "255.255.255.252",
                "enabled": True,
            }
        ],
        "ntp_servers": [],
    }
    result = template.render(variables)
    assert "interface GigabitEthernet0/0" in result
    assert "ip address 10.0.0.1 255.255.255.252" in result
    assert "no shutdown" in result

# Run tests
pytest test_templates.py -v

---

Batfish — Offline Network Analysis

Batfish is an open-source network configuration analysis tool. It reads your device configs and models the network without connecting to any live devices:

Feature	Description
Offline analysis	No need to connect to live devices
Pre-deployment validation	Test changes before they are applied
Reachability analysis	Can host A reach host B?
ACL analysis	What traffic does an ACL permit or deny?
Routing analysis	What path does traffic take?
Compliance checks	Verify configs meet policies
Diff analysis	Compare two config snapshots

Setting Up Batfish

# Run Batfish server in Docker
docker run -d -p 9997:9997 -p 9996:9996 batfish/batfish

# Install Python client
pip install pybatfish

Using Batfish

from pybatfish.client.session import Session

# Connect to Batfish
bf = Session(host="localhost")

# Initialise a network snapshot from config files
bf.set_network("my-network")
bf.init_snapshot("configs/", name="current", overwrite=True)

# Analyse undefined references
undefined = bf.q.undefinedReferences().answer().frame()
print(undefined)

# Check reachability
reachability = bf.q.reachability(
    pathConstraints={"startLocation": "core-rtr-01"},
    headers={"dstIps": "10.0.2.0/24"},
    actions="SUCCESS",
).answer().frame()
print(reachability)

# Analyse ACLs
acl_results = bf.q.searchFilters(
    headers={"srcIps": "0.0.0.0/0", "dstIps": "10.0.1.0/24", "dstPorts": "22"},
    action="PERMIT",
).answer().frame()
print(acl_results)

---

pyATS — Cisco Testing Framework