AI Engineer Capstone: From Prototype to Production

Programme overview

The capstone of the AI Engineer track. Shieldline Insurance, from the Business Analyst Capstone, wants a WhatsApp assistant that turns a customer's first message into a structured claim, asks for exactly the documents still needed, answers policy questions, and hands serious cases to a person. Working from recorded outputs of three configurations on 600 real messages, you'll write the validator that repairs plate numbers and catches invented ones, put escalation rules in code and measure their recall, test retrieval against a keyword baseline, and find out why an LLM judge that agrees with people 90% of the time still can't run alone. Then you'll red-team two guardrails and discover one blocks Pidgin speakers and angry customers far more often, price and time each configuration, pass one through a release gate, and watch it in production as an unannounced voice-note feature quietly breaks it. No API key needed; every number comes from running the code.

Curriculum

1. 01The Brief and the Design25 min

   • The brief and the designMeet Shieldline's WhatsApp claims assistant, decide what the model does and what code decides, define what it must never do, and get to know the recorded test data.
2. 02Extraction and Validation30 min

   • Extraction and validationParse and validate the model's JSON in code, repair what can safely be repaired, catch invented plate numbers without gold labels, and measure each configuration field by field and by language.
3. 03Rules in Code25 min

   • Rules in codeDecide escalation by rule as well as by the model's judgement, measure what each catches, and build the document checklist in code from the policy, so the assistant asks for exactly what's missing.
4. 04Retrieval over the Policy25 min

   • Retrieval over the policyMeasure whether the right policy section reaches the model for each customer question, compare the production retriever with a keyword baseline, and find the questions retrieval fails.
5. 05Grounded Answers and Judging25 min

   • Grounded answers and judgingCompare answers with and without retrieval, see how answer quality depends on retrieval hits, and test whether an LLM judge can replace human grading, including the one kind of error it misses.
6. 06Red-Teaming and Fair Guardrails25 min

   • Red-teaming and fair guardrailsAttack the assistant on purpose, compare two guardrails by how many attacks get through, and check the price: genuine customers wrongly blocked, and whether that price falls on Pidgin speakers and angry customers.
7. 07Cost, Latency and the Release Gate30 min

   • Cost, latency and the release gatePrice each configuration per message and per month, measure its 95th-percentile latency, and put every quality, safety, speed and cost requirement into one release gate that a configuration must pass in full.
8. 08Launch, Monitoring and the Presentation30 min

   • Launch, monitoring and the presentationWatch the assistant's first four weeks in production with control limits set from its first two weeks, catch a change nobody announced, trace it to its cause, and present the project to Shieldline's leadership.

Assessment and certificate

You are assessed with a final assessment (pass mark 60%) and a final project. Your certificate is awarded when you meet those requirements. Paying for the programme does not award it on its own.