Running a Large Language Model on your own infrastructure sounds like the ultimate move for data privacy and cost control. But once the model is up and serving traffic, a new problem emerges: how do you know if it's actually working? Traditional server monitoring tells you if the CPU is hot or memory is low, but it doesn't tell you if the AI is hallucinating, stuck in an infinite loop, or silently dropping requests. This gap is where Site Reliability Engineering (SRE) meets the specific quirks of generative AI.
The reality is that self-hosted LLMs are not just another microservice. They have unique failure modes, resource bottlenecks, and performance characteristics that standard DevOps tools often miss. If you are managing these systems, you need a specialized approach to observability. You aren't just watching servers; you're watching intelligence. Here is how to build a robust operational stack for your self-hosted models without drowning in noise.
Why Standard Monitoring Fails for LLMs
Most teams start with the same playbook they use for web apps: check HTTP status codes and response times. For an LLM, this is dangerously incomplete. A request might return a 200 OK status code, but the generated text could be gibberish, cut off mid-sentence, or take forty seconds to arrive. Conversely, a request might fail quickly, but the root cause could be a subtle GPU memory leak rather than a network timeout.
The core issue is that LLM inference is stateful and resource-intensive in ways that stateless APIs are not. When you host a model, you are dealing with massive tensor computations on GPUs. The "health" of the system depends on three distinct layers:
- Infrastructure Layer: Is the GPU available? Is there enough VRAM?
- Serving Layer: How many requests are queued? What is the throughput?
- Output Layer: Is the quality consistent? Are tokens generating at the expected speed?
If you only monitor the first layer, you will think everything is fine until users complain about slow responses. If you ignore the third layer, you might ship a broken model update without realizing it. Effective SRE practice for LLMs requires instrumenting all three layers simultaneously.
The Core Metrics You Must Track
To get actionable insights, you need to look beyond generic system stats. The most popular open-source serving framework, vLLM, exposes a set of Prometheus metrics that are essential for any serious deployment. These metrics give you a real-time view of what is happening inside the engine.
| Metric Name | What It Measures | Why It Matters |
|---|---|---|
| vllm_num_requests_running | Active requests being processed | Indicates current load capacity |
| vllm_num_requests_waiting | Requests in the queue | High values signal bottleneck or insufficient replicas |
| vllm_gpu_cache_usage_perc | GPU memory cache utilization (%) | Critical for preventing Out-of-Memory errors |
| vllm_avg_generation_throughput_toks_per_s | Average token generation speed | Directly impacts user-perceived latency |
Keep an eye on vllm_gpu_cache_usage_perc specifically. In continuous batching systems, this metric determines how many concurrent sequences can be processed. If this percentage stays consistently above 80-90%, you are likely hitting the ceiling of your hardware capability. Users will experience increased latency as the scheduler struggles to fit new requests into the available memory blocks. Setting alerts here allows you to scale out before the system degrades gracefully into a crawl.
Setting Up Observability in Kubernetes
Most self-hosted LLMs run on Kubernetes because of its ability to manage GPU resources and handle complex scaling logic. However, getting data from pods to your dashboards requires proper configuration. You don't want to manually scrape every pod; you want automation.
The standard approach involves using the Prometheus Operator. By creating a ServiceMonitor resource, you can instruct Prometheus to automatically discover and scrape metrics from your vLLM pods. This ensures that as you scale your inference cluster up or down, your monitoring coverage scales with it.
Here is a practical workflow for setting this up:
- Label Your Pods: Ensure your vLLM deployments have consistent labels (e.g.,
app: llm-inference) so the ServiceMonitor can target them. - Define the ServiceMonitor: Create a YAML manifest that points to the port exposing the Prometheus metrics endpoint (usually 8000/metrics).
- Configure Scrape Interval: For LLMs, a 15-second scrape interval is often sufficient. Faster intervals add overhead without much benefit for long-running generation tasks.
- Visualize in Grafana: Import community-maintained dashboards for vLLM to get pre-built views of throughput, queue depth, and GPU usage.
This setup gives you the foundation. But raw metrics are just numbers. To make them useful, you need to correlate them with business outcomes.
LLMs as Assistants, Not Replacements
There is a growing trend to use AI to fix AI. Some teams try to deploy autonomous agents that analyze logs and restart failed pods without human intervention. While exciting, recent evaluations suggest this is still premature. A comprehensive 2026 evaluation by ClickHouse tested several advanced LLMs on their ability to perform root cause analysis (RCA) in production environments. The results were telling: even top-tier models struggled to consistently outperform experienced SREs when tasked with identifying issues from raw observability data.
The lesson here isn't that AI is useless for operations; it's that context matters. An LLM lacks the institutional knowledge of your specific architecture. It doesn't know that a spike in latency at 3 PM is normal because of a scheduled batch job. Instead of full automation, the effective pattern is augmented investigation.
Use your self-hosted LLMs to assist the SRE team in these ways:
- Log Summarization: Feed large chunks of error logs to the model to get a concise summary of potential causes.
- Drafting Updates: Have the model draft incident updates for stakeholders based on the timeline of events.
- Hypothesis Generation: Ask the model to suggest three possible root causes given a set of symptoms, which the engineer can then verify.
This keeps the human in the loop, ensuring that decisions are grounded in reality rather than statistical probability alone.
Beyond Infrastructure: Monitoring Quality
Tracking GPU usage is table stakes. The harder challenge is monitoring the actual output quality. If your model starts producing repetitive text or ignoring system prompts after an update, your infrastructure metrics will look perfect, but your product is broken.
This is where LLMOps diverges from traditional MLOps. You need to implement sampling-based quality checks. Every hour, send a set of known test prompts to your production endpoint. Compare the outputs against expected baselines. If the divergence exceeds a certain threshold, trigger an alert.
Additionally, track time-to-first-token (TTFT) separately from total generation time. TTFT is highly sensitive to queue depth and pre-processing overhead. If TTFT spikes while throughput remains stable, you likely have a scheduling issue. If both spike, you likely have a hardware or network bottleneck.
Common Pitfalls and How to Avoid Them
Even with the right tools, teams often stumble on a few recurring issues. Being aware of these can save you weeks of debugging.
Ignoring Cold Starts: Loading a large model into GPU memory takes time. If you scale from zero replicas, users will face significant delays during the initial load. Monitor the startup duration and consider keeping a minimum number of warm replicas for critical services.
Over-Scaling Based on CPU: LLM inference is GPU-bound. Watching CPU usage to decide scaling decisions is misleading. Focus on GPU utilization and request queue length instead.
Alert Fatigue from Transient Spikes: LLM workloads can be bursty. A single heavy prompt might cause a momentary dip in throughput. Use intelligent alerting rules that require sustained deviations over a period (e.g., 5 minutes) before triggering a page.
Frequently Asked Questions
What is the difference between LLMOps and MLOps?
While MLOps focuses on the lifecycle of machine learning models including training and deployment, LLMOps specifically addresses the unique challenges of large language models. This includes managing non-deterministic outputs, handling massive context windows, optimizing for token throughput rather than just accuracy, and integrating retrieval-augmented generation pipelines. LLMOps places a heavier emphasis on runtime observability and quality assurance of generated content.
Which metrics are most important for vLLM monitoring?
The four most critical metrics are vllm_num_requests_running, vllm_num_requests_waiting, vllm_gpu_cache_usage_perc, and vllm_avg_generation_throughput_toks_per_s. Together, these provide a complete picture of load, queuing, memory pressure, and processing speed. Monitoring GPU cache usage is particularly vital for preventing out-of-memory crashes in continuous batching scenarios.
Can LLMs fully automate Site Reliability Engineering tasks?
Currently, no. Recent evaluations show that autonomous root cause analysis by LLMs is not yet reliable enough to replace experienced SREs. However, LLMs are highly effective as assistants that summarize logs, suggest hypotheses, and draft communications. The best practice is to use them to augment human decision-making rather than to act autonomously in production environments.
How do I monitor the quality of LLM outputs in production?
Implement periodic sampling tests where known prompts are sent to the production endpoint. Compare the resulting outputs against baseline expectations for consistency, format, and factual accuracy. Set alerts if the deviation score exceeds a predefined threshold. Additionally, track time-to-first-token to detect early signs of performance degradation that may affect output coherence.
What is the recommended scrape interval for Prometheus when monitoring LLMs?
A 15-second scrape interval is generally recommended for LLM inference endpoints. This frequency provides sufficient granularity to catch transient issues without placing excessive load on the application or the monitoring stack. Since LLM generation tasks are relatively long compared to typical API calls, sub-second precision is rarely necessary for operational health checks.
