Cost per task, model tiering, prompt caching, routing, batch processing, and hard caps: how to run LLM features cheaply without a quality regression.
The pilot cost almost nothing. Then the feature shipped, usage grew, an agent loop retried a few thousand times over a weekend, and the invoice became a board-level conversation. LLM features have a cost profile unlike the software most companies run, and the teams that keep them affordable treat cost as an engineering metric from the first week rather than a finance surprise in month three.
Software has traditionally been bought per licence or per seat, and that is a bad unit for LLM features: usage, not head count, drives the bill. Per-token pricing is the vendor's unit, but it is not yours either, because a single business outcome may take several model calls, a couple of retries, some tool calls, and a minute of human review.
The unit that maps to your P&L is cost per task: what it costs, end to end, to resolve one support ticket, classify one document, draft one proposal, or complete one agent run. Define the task precisely, measure everything that contributes to it, and compare it to the baseline of doing the same work without the model. Every lever below is judged by one question: does it lower cost per task without lowering the task's quality?
To measure it you need instrumentation on every call: input and output tokens, cache reads, model used, retries, tool invocations, latency, and the task ID that ties them together. Without that, "we should use a cheaper model" is an opinion.
Vendors now ship in tiers, and the gap between them is where most of the savings live. Anthropic's Claude Opus 5, released on July 24, 2026, is described as coming close to the frontier intelligence of Claude Fable 5 at half the price. OpenAI's GPT-5.6, released July 9, 2026, ships in three variants ranked from least to most capable: Luna, Terra, Sol. Tiered models are a standard cost lever, not an exotic optimisation.
A useful mental model:
The mistake is choosing one tier for the whole product, or choosing by reputation rather than by running your own evaluation set on each candidate.
Most production prompts are mostly constant: the system prompt, tool definitions, style guides, reference documents, few-shot examples. Prompt caching lets the provider reuse the processed form of that stable prefix across calls instead of paying for it every time. It is one of the two standard cost levers, alongside tiering, and it usually requires only restructuring the prompt so that the stable content comes first and the variable content (the user's message, the retrieved chunks) comes last.
Measure your cache hit rate. A low hit rate usually means something dynamic, such as a timestamp or a per-user greeting, has crept into the prefix and is invalidating it.
Routing sends each request to the cheapest tier that can handle it. Three patterns work in practice:
Routing itself costs a call, so keep the router cheap and confirm that escalations are actually rare.
Much LLM work does not need an answer in seconds: nightly classification, backfills over historical records, weekly reports, enrichment of new CRM entries. Separate those workloads from interactive ones. Latency-tolerant jobs can run on slower or cheaper tiers, can be scheduled off-peak, and, where your provider offers a batch tier, can take advantage of it. Only interactive features should pay for low latency.
The largest overspends we see come from loops, not steady usage: an agent retries a failing tool, a queue redelivers the same job, a test suite runs against production keys. Guardrails are cheap and non-negotiable:
This is the design behind our My Cloud Company managed agents: approval gates before consequential actions, hard budget caps that stop a run rather than warn about it, and an audit trail that shows exactly where each unit of spend went.
Every lever above can quietly reduce quality, so cost work without an evaluation set is gambling. Build a set of real tasks with known-good outcomes, score every configuration on it, and put cost and quality on the same dashboard. Then make trade-offs explicitly: a configuration that is meaningfully cheaper and slightly worse may be the right choice for internal summaries and the wrong one for customer-facing drafts, but that should be a decision someone made, not a regression nobody noticed. Set thresholds so a change that drops quality below the line fails the build.
We price the outcome, not the seat: cost per task, not cost per licence. On generative AI and LLM application engagements we baseline the workflow first, so there is a real number to beat, then instrument every call before optimising anything. Model tiering, caching, and routing decisions are made against an evaluation set that runs before launch and on every change afterwards. Human-in-the-loop is designed in where the cost of a wrong answer is high, and we operate the feature after launch because usage patterns, models, and prices all move. For agentic workloads our AI agent development practice ships with hard caps and approval gates from day one, and AI workflow automation projects move batch-able work off the realtime path as a matter of course. If you need an outside view on where your current spend is going, start with AI consulting or contact us directly.
Answers
Use the cheapest tier that passes your evaluation set for that specific task. Many tasks pass on a workhorse model, some genuinely need frontier capability, and the only reliable way to tell them apart is to score each candidate on real examples rather than on reputation.
Usually prompt caching combined with hard spend caps. Restructuring the prompt so stable content comes first often takes a day and cuts repeated processing, while caps and iteration limits prevent the retry loops that cause the largest surprise bills.
Score every configuration change on an evaluation set built from real tasks, and put cost and quality side by side. Set a regression threshold so a change that drops quality below it fails automatically, the same way a failing test blocks a release.
Put it into practice
The XISLABS services closest to what this article covers.
Keep reading
Stop picking one winner. A three-tier model strategy (frontier, workhorse, self-hosted) with routing by task, data residency, and eval-driven switching.
Why RAG demos impress and production RAG fails, and the engineering that closes the gap: permission-aware retrieval, hybrid search, citations, and evals.
What the 2026-07-28 MCP spec changes mean operationally, what the adoption numbers say, and how to expose internal systems as MCP servers safely.
We build the AI agents, automation, and software behind ideas like these — scoped to a metric, shipped in weeks, operated after launch.