What happens when you promise a customer that an AI feature will be “fast and reliable,” and then someone asks what that means in numbers? That question is where most LLM projects either grow up or quietly fall apart. Traditional web services have spent two decades turning vague promises into measurable targets. Large language models (LLMs) break several of the assumptions behind those targets, and teams that copy their old monitoring playbook end up with dashboards full of green lights while users complain.
This article walks through what service level objectives (SLOs) mean for LLM-powered services, why the classic vocabulary needs adjusting, and what actually works in practice.
The Vocabulary You Need First
Three terms do most of the work, and they are routinely confused.
An SLI (service level indicator) is a number you measure. “The share of requests that finished under two seconds” is an SLI. An SLO (service level objective) is a target range for that indicator, such as “99% of requests under two seconds over a rolling 28 days.” An SLA (service level agreement) is a contract with consequences attached if the target is missed. (Planning capacity for a new cloud service) (Service-level objective (Wikipedia))
Google’s Site Reliability Engineering book, published free online in 2016, is the canonical explainer of this vocabulary and of error budgets, the idea that if your SLO is 99%, the remaining 1% is a budget you can spend on risky releases. That framework predates modern LLMs by roughly a decade, which is exactly why it fits awkwardly at first: it was built for systems where the same input reliably produced the same output.
Why LLM Services Break the Classic Model
Three properties of LLM endpoints make naive SLOs misleading.
Non-determinism
A deterministic service can promise that identical requests get identical responses. An LLM endpoint cannot. The same prompt can produce different text on two consecutive calls, and quality varies from request to request. That means a quality SLO has to be framed statistically: “at least 92% of items in eval set Y pass rubric Z,” not “every response will be correct.” Teams that skip this framing end up arguing about single bad outputs instead of tracking a rate.
Latency is several numbers, not one
For a traditional API, latency is one distribution. For a generative endpoint, practitioners track at least three: time to first token (how long until output starts streaming), inter-token latency (how fast tokens arrive once started), and end-to-end latency (total time to completion). End-to-end latency also scales with output length, so a long answer is not “slower” in the same sense a slow database query is. If your SLO only tracks averages, streaming interfaces can feel broken while the average looks fine.
Cost is a first-class reliability signal
Per-token pricing makes cost per successful task a meaningful indicator alongside latency and availability. A retry loop that quietly triples token spend to hit a latency target is not a reliability win. Emerging practice treats throughput and spend as SLIs in their own right, though no formal standard defines them yet.
Building an SLO Set That Holds Up
A workable LLM SLO set usually covers four families. The table below shows typical indicator choices and what each one catches.
| Family | Example SLI | What it catches |
|---|---|---|
| Availability | Share of requests returning a valid response (excluding client errors) | Outages, provider incidents |
| Latency | Share of requests with time to first token under 1.5s | Perceived sluggishness in chat UIs |
| Quality | Share of eval items passing a graded rubric | Prompt regressions, model swaps |
| Cost | Median tokens per completed task | Silent retry loops, prompt bloat |
A few practical rules make these targets survive contact with reality.
Start with one quality SLO, not five. Pick the eval set that most closely mirrors real traffic and track a single pass rate. Expanding later is easy; defending five half-measured quality targets in a review is not.
Set windows deliberately. A 28-day rolling window smooths daily noise and matches how most teams do release planning. Weekly windows are useful for fast-moving products but invite arguments about whether a bad Tuesday counts.
Separate rate limits from SLOs. Rate limits and quotas cap load; they do not measure it. A 429 response is a capacity decision, not a reliability failure, and mixing the two into one availability number hides real incidents. Most provider terms also exclude rate-limit responses and scheduled maintenance from their published availability commitments, which is worth reading carefully before you mirror those numbers internally.
Treat provider commitments as inputs, not your SLO. Major API providers publish monthly availability commitments in their terms. Those are contractual SLAs, not SLOs in the reliability-engineering sense, and they say nothing about quality or time to first token. Your own SLOs should describe what your users experience end to end, including your retrieval layer, prompt assembly, and fallback logic.
Where Evals and Monitoring Meet
The hardest part of LLM reliability is that quality is a distribution, and distributions need sampling. A quality SLI is only as good as the eval set behind it, and eval sets rot. Prompts drift, user behavior shifts, and a rubric that graded well in March may grade poorly in September.
What works in practice is a two-tier loop. Offline, a fixed eval set runs on every prompt or model change, producing a pass rate that feeds the quality SLO. Online, a small sample of production traffic gets graded by humans or by a model-based rubric, and that sample catches the drift the offline set misses. Neither tier alone is sufficient, and teams that rely only on offline evals tend to discover regressions from user complaints rather than dashboards.
There is a governance question hiding here too. When a model provider updates a hosted model, quality can shift without any change on your side. Owning the measurement layer, rather than renting it, matters more than it first appears; the same logic applies to deciding who owns an AI service after the pilot, a question that shapes everything from on-call rotation to vendor contracts.
Common Mistakes and What Actually Works
Several failure patterns show up again and again.
Copying web-service SLOs wholesale. A 99.99% availability target borrowed from an internal API makes no sense for a feature that depends on an external model provider with its own published commitment. Set targets you can actually control, and document the dependency chain.
Measuring latency as a single average. Averages hide the tail, and the tail is what users remember. Track percentiles for time to first token, and watch the streaming experience specifically.
Ignoring capacity planning until launch week. Token throughput per instance, concurrency limits, and queue behavior under load all determine whether your latency SLO survives a marketing push. Working through planning capacity for a new cloud service before the traffic arrives is cheaper than renegotiating targets afterward.
Letting the error budget go unspent or unowned. If the budget is never spent, the SLO is probably too loose. If it is always exhausted, releases stall and teams start ignoring the dashboard. Someone needs to own the number and the decision it drives.
Forgetting the fallback path. When the primary model degrades, does the service return a cached answer, a smaller model’s output, or an honest error? Each choice produces a different SLI, and the choice should be written down before the incident, not during it.
Standards bodies offer useful scaffolding here. The ISO/IEC 25010 software quality model defines reliability and performance efficiency as distinct characteristics, which helps when mapping LLM SLOs onto existing quality documentation. The NIST AI Risk Management Framework (AI RMF 1.0, published in 2023) frames AI reliability and measurement in public-sector terms and is a reasonable reference when SLOs need to survive an audit or a procurement review.
A Worked Example
Suppose a support assistant answers customer questions with retrieval-augmented generation. A first SLO set might look like this:
- Availability: 99.5% of requests return a valid response over 28 days, excluding client errors and rate-limit responses.
- Latency: 95% of requests begin streaming within 1.5 seconds; 90% complete within 12 seconds for answers under 400 tokens.
- Quality: 90% of items in the support eval set pass the graded rubric, measured weekly.
- Cost: median tokens per resolved question stays under the agreed threshold.
Each number has a named owner, a dashboard, and a documented response when the error budget burns too fast. That last part is what separates an SLO from a wish. For teams watching how these debates play out across the industry, following the news on service-level practice is a reasonable way to keep targets calibrated. (News)
Conclusion
LLM service level objectives are not a new religion; they are the old reliability discipline applied to systems that are probabilistic, expensive per call, and partly dependent on someone else’s infrastructure. The teams that get this right keep the vocabulary simple, measure latency as a distribution rather than a number, treat quality as a sampled rate instead of a per-request guarantee, and give every target an owner and a consequence. Start with one availability SLO and one quality SLO, add latency percentiles once streaming is live, and let the error budget tell you when to slow down. The models will keep changing. The measurement layer is what makes that change survivable.
Frequently Asked Questions
What is the difference between an SLO and an SLA for an LLM service?
An SLO is an internal target for a measured indicator, such as 99% of requests under two seconds. An SLA is a contract with consequences if a target is missed. Provider terms usually publish availability commitments as SLAs; your internal SLOs should describe the end-to-end user experience, including your own retrieval and fallback layers.
How do you measure quality when LLM outputs vary?
Quality is tracked as a pass rate over an eval set rather than a per-request guarantee. A fixed offline eval set runs on every prompt or model change, and a sampled slice of production traffic gets graded by humans or a rubric to catch drift. The SLO is then a target on that rate, such as 90% of items passing weekly.
What latency metrics matter most for streaming LLM apps?
Time to first token matters most for perceived responsiveness, followed by inter-token latency and end-to-end completion time. End-to-end latency scales with output length, so it should be measured separately for short and long answers. Averages hide the tail, so track percentiles.
Do provider uptime commitments count as my SLO?
No. Published provider commitments are contractual availability figures, typically monthly, and they usually exclude rate-limit responses and scheduled maintenance. They say nothing about quality or time to first token. Use them as an input when setting your own targets, not as a substitute.
How many SLOs should a new LLM feature start with?
Two is a sensible start: one availability SLO and one quality SLO tied to a representative eval set. Add latency percentiles once streaming is live and cost indicators once token spend is visible. Expanding a small, well-measured set later is far easier than defending five half-measured targets.
What is an error budget and how is it used with LLMs?
The error budget is what remains when you subtract the SLO from 100%. If the availability target is 99.5%, the 0.5% shortfall is budget you can spend on risky releases. When the budget burns fast, releases slow and reliability work takes priority. A budget that is never spent usually means the target is too loose.
