Levels 1–5 of Agentic Engineering

Chapter 38 Evaluating and Directing AI-Generated Code

This is the most important chapter in Phase 9. Without these skills, every subsequent chapter produces a reader who uses AI tools without the judgment to evaluate their output.

The Core Problem: Non-Determinism

Every programming concept you've learned so far is deterministic. Run 2 + 2 a million times, you get 4 every time. Call sorted([3,1,2]) and the result is always [1,2,3]. Functions have inputs and outputs. Tests pass or fail. This predictability is the entire foundation of software engineering.

Large language models (LLMs) are fundamentally non-deterministic. Ask the same question twice and you may get different answers. Ask it to write a function and the variable names, structure, edge case handling, and even the algorithm choice can vary between runs. This isn't a bug—it's how the technology works. LLMs sample from probability distributions over tokens; small floating-point differences cascade into different outputs.

This creates a paradox: you want reliable software from an unreliable generator. The only way to make a non-deterministic system behave as if it were deterministic is to surround it with extremely robust guardrails:

• Tests — the code either passes or it doesn't, regardless of how it was generated
• Specifications — precise instructions reduce the space of possible outputs
• Context documents (CLAUDE.md, rules files) — constrain the model's choices to match your project's patterns
• Review pipelines — diffs, linting, type checks catch what the model missed
• Deterministic validation — CI/CD, automated checks, acceptance criteria that don't depend on judgment

Every technique in Phases 5 and 6—context engineering, compounding rules, harness engineering, Docker isolation—exists to constrain non-deterministic output into deterministic boundaries. The guardrails aren't optional extras. They are the entire point. Without them, you're shipping lottery tickets.

Reading Diffs

A diff shows line-by-line changes: + = added, - = removed. When Claude Code makes changes, you review the diff before accepting. Check: does the change match what I asked? Any unrelated changes? Any wrong deletions?

Worked Example: Reviewing a Real Diff

Here's how an experienced developer reviews a diff. Suppose you asked Claude to "add error handling to the load_tasks function." The diff looks like this:

 def load_tasks(filepath):
-    with open(filepath) as f:
-        data = json.load(f)
-    return data
+    if not isinstance(filepath, str) or not filepath.strip():
+        raise ValueError("filepath must be a non-empty string")
+    try:
+        with open(filepath) as f:
+            data = json.load(f)
+    except:
+        return []
+    if not isinstance(data, list):
+        raise TypeError(f"Expected list, got {type(data).__name__}")
-    import os
     return data

Now the annotation—line by line:

• Lines 1-2 (added): Good—added input validation for filepath. Prevents cryptic errors from None or empty strings reaching open().
• Line 5 (except:): Problem—this is a bare except that catches all exceptions, including KeyboardInterrupt and SystemExit. It should be except (FileNotFoundError, json.JSONDecodeError): to catch only the expected failures.
• Line 6 (return []): Question—silently returning an empty list hides errors. Is that intentional, or should it log a warning? Depends on the project's error philosophy.
• Lines 8-9 (type check on data): Good—validates the JSON structure before returning. Catches corrupted files early.
• Line 10 (import os removed): Question—why was this import removed? If nothing else uses os, this is a fine cleanup. But check the rest of the file first.

Verdict: Revise. The bare except is a real bug—ask Claude to catch specific exceptions. The rest is solid.

Writing Implementation Specs

The difference between getting mediocre AI output and great AI output is the quality of your specification:

Recognizing AI Failures

TaskForge Connection

In the exercises below, you'll use TaskForge as the subject for AI code generation. You have enough context about the project to judge whether AI output is correct.

Micro-Exercises

Interactive Exercises

Chapter 39 Levels 1 & 2 — Tab Complete and the Agent IDE

Level 1: Tab Complete

Concept: Inline code suggestions as you type. Press Tab to accept.

Level 2: Agent IDE

Concept: Chat connected to your codebase. Describe in natural language, AI produces multi-file edits.

Plan Mode

Describe → AI plans → you review → approve → execute → review diff → test. Boris Cherny (Claude Code creator) starts 80% of tasks in plan mode. With each model generation, one-shot success climbs. Plan mode is the right entry point.

TaskForge Connection

You'll use Claude Code on TaskForge for the first time in this chapter's exercise.

Micro-Exercises

npm install -g @anthropic-ai/claude-code
claude --version

Expected output:

$ npm install -g @anthropic-ai/claude-code

added 1 package in 8s

Verify the installation:

$ claude --version
claude v1.x.x

If npm is not found, return to Phase 1 Chapter 3 and install Node.js. If you get a permissions error, try sudo npm install -g @anthropic-ai/claude-code (macOS/Linux) or run your terminal as Administrator (Windows).

Interactive Exercises

Chapter 40 Level 3 — Context Engineering

Context engineering is controlling what the model sees so every token does useful work. "Every token needs to fight for its place in the prompt."

The Context Window

Think of the context window as the model's working memory budget. Everything the model can consider at once—your prompt, the system instructions, file contents, conversation history—must fit in this budget. When it fills up, older information gets pushed out. Managing this budget is the core skill of context engineering.

The Four Surfaces

CLAUDE.md Example for TaskForge

# TaskForge
Command-line task manager with JSON persistence and Flask API.

## Commands
- `python3 -m pytest` — run tests
- `python3 -m flask run` — start API server
- `python3 src/taskforge/main.py` — run CLI

## Architecture
- src/taskforge/ — core logic (main.py, models.py, api.py)
- tests/ — pytest test files
- data/ — JSON storage (gitignored)

## Code Style
- Type hints on all function signatures
- Docstrings on all public functions
- No bare try/except — always catch specific exceptions

## Gotchas
- Task IDs are auto-incrementing integers, NOT UUIDs
- JSON file is the single source of truth
- Flask API mirrors CLI functionality exactly

Before/After: Context Quality

The same model with different context produces dramatically different results:

Building Instruction Scaffolds

A CLAUDE.md for a small project is straightforward. But what if your goal is bigger—like telling Claude Code how to set up an entire environment from scratch? This is where you build an instruction scaffold: a document so precise that a non-deterministic model produces the same correct output nearly every time.

Remember the core problem from Chapter 38: LLMs are non-deterministic. Ask Claude to "set up Docker for this project" five times and you'll get five different Dockerfiles, different compose configs, different auth approaches. Some will work. Some won't. The variation is the enemy.

The solution is a scaffold that eliminates ambiguity at every decision point:

In Chapter 24 (Harness Engineering), you'll see a complete scaffold that applies every one of these principles to set up a Docker environment for Claude Code. It's over 300 lines—and every line exists because leaving it out caused a failure in practice.

Micro-Exercises

Interactive Exercises

Chapter 41 Level 4 — Compounding Engineering

Compounding engineering improves every session after the current one. The loop: Plan → Delegate → Assess → Codify. The Codify step is what makes it compound.

Where to Codify

TaskForge Connection

You'll create a docs/architecture.md for TaskForge and link to it from CLAUDE.md. This is compounding: future Claude sessions will find the architecture doc and produce better code.

Micro-Exercises

Interactive Exercises

The Spec-Driven Workflow

The single most effective workflow for AI-assisted development. Instead of jumping straight to code, you go: brainstorm → spec → plan → execute.

Step 1: Brainstorm

Start with a conversation, not code. Ask the AI to ask you questions:

I want to build [feature]. Before we write any code, ask me clarifying questions
one at a time to understand: what problem this solves, who uses it, what the core
features are, what it should NOT do, and what edge cases matter.

The AI asks clarifying questions. You answer. Output: a shared understanding of the problem.

Step 2: Write a Spec (spec.md)

Turn the brainstorm into a written specification:

• What the system does (requirements)
• What it does NOT do (explicit scope boundaries)
• Data model (what gets stored)
• API or interface design (inputs, outputs, errors)
• Edge cases and error handling
• Acceptance criteria (how you know it's done)

Step 3: Generate a Plan (prompt_plan.md)

Break the spec into ordered implementation steps. Each step is a prompt you'll give the AI, small enough to review in under 5 minutes.

Step 4: Execute Step by Step

Give each prompt from the plan one at a time. Review the output. Run tests. Fix issues before moving on. Never skip review. Never batch multiple steps.

Why this works: The spec is your source of truth, not the AI's memory. The plan creates checkpoints. If something goes wrong, you know exactly where. And it compounds — your spec quality improves with every project.

Chapter 42 Level 5 — MCP, Skills, and Capabilities

MCP (Model Context Protocol)

Concept: Standardized connectors letting Claude Code interact with external tools and services.

// .mcp.json (project root)
{
  "mcpServers": {
    "postgres": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-postgres"],
      "env": { "DATABASE_URL": "postgresql://..." }
    }
  }
}

Custom Skills

Concept: Reusable instruction sets loaded on demand. A skill defines a workflow, tool restrictions, and model choice.

# .claude/skills/pr-review/SKILL.md
---
name: pr-review
description: Reviews PRs for bugs, security, performance, style.
tools: Read, Glob, Grep, Bash
model: opus
---
You are a senior code reviewer. Check:
## Security — hardcoded secrets, injection, XSS
## Performance — N+1 queries, missing indexes
## Output — BLOCKING, SUGGESTION, or APPROVED

Custom Subagents

Concept: Specialized roles with restricted tools. Each subagent only has access to the tools it needs.

# .claude/agents/test-writer.md
---
name: test-writer
description: Writes tests. Can read source and write test files only.
tools: Read, Glob, Write
model: sonnet
---
Write comprehensive tests. Cover: happy path, edge cases, error cases.

Hooks

Automated scripts at lifecycle points:

// .claude/settings.json
{ "hooks": { "preCommit": [{ "command": "npm run lint && npm test" }] } }

TaskForge Connection

You'll create a skill for TaskForge that adds a new feature with tests, and a subagent that reviews the result.

Micro-Exercises

Interactive Exercises

Phase 9 Gate Checkpoint & TaskForge AI Integration

TaskForge Checkpoint

TaskForge now has AI-readable configuration, custom tooling, and at least one AI-implemented feature. Ready for multi-agent work.

What You Can Now Do