AI Agents and Tool Use

Programme overview

An AI agent doesn't just answer; it acts, by calling tools in a loop. That makes it useful and dangerous. In this course you take apart two versions of Paystream's support agent, recorded step by step on 150 real requests that a support lead labelled. You'll design tools scoped to the logged-in customer, move business rules out of the prompt into tested code, and write a loop with limits, repeated-call detection and a trace. You'll find the refunds v1 issued without approval and the other customers' transfers it read, evaluate both versions on outcomes and on the paths they took, and see how instructions hidden in transfer narrations hijacked v1. Finally, you'll work out cost per correct resolution, design monitoring and a gradual rollout, and write the design for a safer v3. Every number comes from running the code on the recorded runs; live model calls are optional and shown with the Anthropic SDK.

Curriculum

1. 01From Chatbot to Agent15 min

   • From chatbot to agentWhat an AI agent is (a model that chooses tools in a loop, while your code runs them), how it differs from a fixed workflow, and how to read the step-by-step record of an agent run.
2. 02Designing Tools25 min

   • Designing toolsWrite the tools an agent can call, with clear names, descriptions and input schemas, scoped to the customer being served, and returning errors as data the model can act on.
3. 03The Agent Loop15 min

   • The agent loopWrite the loop that runs an agent (send, check for tool calls, run them, send results back) with a step limit, a stop on repeated failures, and a full trace of every step.
4. 04Rules in Code, Not Prompts25 min

   • Rules in code, not promptsMove business rules out of the prompt and into deterministic tools the agent must call, check them against labelled decisions, and measure how often an agent that judged the rule itself got it wrong.
5. 05Permissions and Approvals25 min

   • Permissions and approvalsClassify an agent's tools by the harm they can do, give it only the ones it needs, require a person's approval for risky actions, and measure the damage v1's broad permissions would have done.
6. 06Evaluating Agents25 min

   • Evaluating agentsMeasure an agent on outcomes (did it take the right final action?) and on trajectories (did it get there safely?), break the results down by request type, and read the failures behind the numbers.
7. 07Injection Through Tools25 min

   • Injection through toolsSee how instructions hidden in data an agent reads (here, transfer narrations) can hijack it, measure how often v1 was hijacked, and build the layered defences that stopped v2.
8. 08Cost, Latency and Limits15 min

   • Cost, latency and limitsWork out what an agent costs per request and per correct resolution, why each extra step costs more than the last, how long customers wait, and which limits keep both under control.
9. 09Monitoring Agents in Production20 min

   • Monitoring agents in productionDecide what to log, which numbers to watch every day, when to alert a person, and how to roll an agent out gradually, so that problems are found by your dashboard, not your customers.
10. 10Final Project20 min

    • Final project: Paystream's support agentPlan your final project, a support agent designed, tested on recorded runs, and made safe, with an evaluation and operating plan the head of support can sign off.

Assessment and certificate

You are assessed with a final assessment (pass mark 60%) and a final project. Your certificate is awarded when you meet those requirements. Paying for the programme does not award it on its own.