Build a cost model that breaks down LLM inference spend by model choice, token volume and context length.
Digital Transformation & Artificial Intelligence
Reducing LLM Inference Costs with Model Routing, Caching and Quantisation
Cut generative AI running costs by combining model routing, caching and quantisation without degrading the output quality users depend on.
🗓️Duration5 training days
📚Content4 modules · 8 sessions
🎓On completionAccredited attendance certificate
About the programme
Course Overview
As generative AI features move from pilot to production, inference cost becomes one of the largest and least predictable lines in a technology budget, driven by token volume, context length and the choice of model for every request. This course teaches practical techniques for controlling that cost without simply downgrading the user experience: routing requests to the smallest model capable of handling them, caching repeated or semantically similar prompts, and quantising models to reduce memory and compute per inference. Participants work through cost modelling exercises using real token-pricing structures, build a routing policy that sends easy queries to smaller models and hard queries to larger ones, and evaluate the accuracy trade-offs of different quantisation levels. The course also covers batching, context-length management and caching architecture, since cost reduction usually comes from combining several techniques rather than any single one. Participants leave with a cost model for their own application, a routing and caching design ready for implementation, and a method for tracking savings against output-quality metrics over time.
Expected Learning Outcomes
Design a model-routing policy that sends each request to the smallest model capable of an acceptable answer.
Implement prompt and response caching, including semantic caching, to avoid recomputation of repeated queries.
Evaluate quantisation levels for accuracy loss against the compute and memory savings they deliver.
Apply batching and context-length management techniques that reduce cost per request under real traffic patterns.
Set output-quality guardrails so cost-reduction techniques are rejected automatically if accuracy drops below an agreed threshold.
Track inference cost and quality together over time and report savings in terms finance stakeholders can verify.
Who Should Attend
Machine learning engineers responsible for the running cost of a production generative AI feature.
Platform teams operating shared LLM infrastructure across multiple internal applications.
Product managers under pressure to reduce the unit economics of an AI-powered feature.
FinOps and cloud-cost analysts extending cost governance practices to AI workloads.
Solution architects designing new generative AI applications that must meet a cost ceiling.
Technical leads evaluating whether to replace a single large model with a mixed-model architecture.
Course Modules
Select any module to see its sessions and points.
01Understanding and Modelling LLM Inference Cost
2 sessions · 8 points
Session 1Where Inference Cost Actually Comes From
- Break down a token-pricing structure into input tokens, output tokens and context-window charges for a given model.
- Trace how prompt design, system messages and retrieved context inflate token counts beyond the user's visible question.
- Identify which application features drive the highest inference volume through usage logs and request sampling.
- Separate fixed infrastructure cost from variable per-request cost when comparing self-hosted and API-based models.
Session 2Building a Cost Model for a Production Application
- Construct a spreadsheet or dashboard model that projects monthly inference cost from expected traffic and token volume.
- Model how cost scales under peak load, seasonal demand and feature growth rather than average daily usage alone.
- Set a cost-per-request target and trace which components of the current architecture exceed it.
- Present the cost model to stakeholders in a form that ties technical choices directly to budget impact.
02Model Routing Strategies
2 sessions · 8 points
Session 1Designing a Routing Policy Across Model Tiers
- Classify incoming requests by difficulty so simple queries can be routed to smaller, cheaper models.
- Build a lightweight classifier or rule set that decides which model tier handles each request before the main call is made.
- Design an escalation path that reroutes a request to a larger model when a smaller model's confidence is low.
- Measure the accuracy and cost trade-off of a routing policy against always using the largest available model.
Session 2Using Small and Specialised Models Alongside Large Ones
- Evaluate small language models and task-specific fine-tuned models as lower-cost substitutes for general-purpose large models.
- Combine multiple specialised models in a pipeline so each step uses the cheapest model capable of that step's task.
- Benchmark small-model accuracy against large-model accuracy on the specific tasks the application actually performs.
- Plan a fallback path so specialised models can hand off to a larger model when they encounter unfamiliar input.
03Caching and Batching Techniques
2 sessions · 8 points
Session 1Prompt, Response and Semantic Caching
- Implement exact-match caching for repeated prompts, such as common support questions or template-based requests.
- Build semantic caching that recognises differently worded but equivalent requests using embedding similarity.
- Set cache expiry and invalidation rules so cached responses do not go stale when underlying data changes.
- Measure cache hit rate and the resulting cost saving separately from other optimisation techniques.
Session 2Batching and Context-Length Management
- Batch compatible requests together to improve throughput and reduce per-request overhead on self-hosted infrastructure.
- Trim retrieved context and conversation history to the minimum needed for an accurate response.
- Apply summarisation or truncation strategies to long conversation histories before they are resent to the model.
- Balance batching latency against user experience requirements for interactive, real-time features.
04Quantisation and Ongoing Cost Governance
2 sessions · 8 points
Session 1Quantisation and Compression for Self-Hosted Models
- Compare quantisation levels, such as 8-bit and 4-bit, on the memory and compute savings they deliver for a self-hosted model.
- Test quantised models against an accuracy benchmark to confirm degradation stays within an agreed tolerance.
- Apply distillation to train a smaller model that approximates a larger model's behaviour at lower inference cost.
- Select hardware and serving frameworks that take advantage of quantised model formats efficiently.
Session 2Tracking Savings and Sustaining Cost Discipline
- Set up dashboards that track cost per request, cache hit rate and routing distribution alongside output-quality metrics.
- Define an approval process for any change that could increase cost, such as raising context length or model tier.
- Report realised savings against the original cost model in terms that finance stakeholders can independently verify.
- Review routing and caching rules on a fixed cycle as new, cheaper models become available.
What the participant receives
4 course modules
A structured syllabus
8 training sessions
across 5 days
32 detailed points
Applied, detailed content
Accredited attendance certificate
On completing the programme
Complete your registration
We will contact you within one business day to confirm.
Ready to start?
Reserve your seat and start building the skill.