Imagine you launch a new AI feature. Traffic spikes, costs triple overnight, and your dashboard shows thousands of requests from a single IP address. Was it a viral moment or an attacker trying to drain your bank account? This isn't hypothetical; it's the daily reality for anyone exposing Large Language Model (LLM) endpoints to the public. Traditional rate limiting, designed for simple web pages, fails miserably against AI workloads where one request can cost ten times more than another. If you treat every request equally, you're either blocking legitimate users or letting attackers bleed you dry.
You need a defense that understands tokens, not just HTTP calls. You need to detect abuse before it hits your inference engine. This guide breaks down how to build a robust system that balances availability, cost control, and security, using strategies proven by major providers like OpenAI, Anthropic, and AWS.
Why Traditional Rate Limiting Fails on LLMs
Most developers start with Requests Per Minute (RPM). It’s easy to implement and works fine for fetching a user profile. But LLMs are different. A request asking "What is the capital of France?" consumes negligible compute. A request pasting a 50-page PDF and asking for a summary consumes massive resources. If your limit is 100 RPM, an attacker can send 100 complex, long-context requests per minute, crushing your GPU capacity while a legitimate user sending short queries gets throttled unfairly.
This mismatch creates two distinct problems: Denial of Service (DoS), where the server becomes unresponsive due to resource exhaustion, and Denial of Wallet (DoW), where the service stays up but the bill explodes. OWASP identifies this as Model Denial of Service (LLM04), highlighting that attacks often use unusually long inputs or context-window overflows. To fix this, you must shift from counting requests to counting resources. You need to track input tokens, output tokens, and concurrency simultaneously.
The Core Mechanisms: Token-Aware Admission Control
Effective protection starts at the edge, before the request ever reaches the model provider. Think of your gateway as a bouncer who checks not just ID, but also how heavy the person's backpack is. You cannot simply let everyone in based on their ticket count. You must estimate the cost of each entry.
Here is how modern admission control works:
- Token Estimation: Before dispatching, calculate the estimated input tokens plus the maximum requested output tokens (
max_tokens). Reserve this amount against the user's quota. - Concurrency Limits: Restrict the number of simultaneous generations per tenant. This prevents a few heavy jobs from starving interactive traffic.
- Spend Caps: Set hard monetary limits per hour or billing period. Unlike alerts, which just notify you, hard caps stop traffic when the budget is exhausted.
- Bounded Queues: If a user exceeds their burst allowance, queue them rather than rejecting immediately, but keep the queue size finite to prevent memory overflow.
Amazon Bedrock illustrates this well. Its mantle endpoint reserves input tokens plus max_tokens upon admission. As the model streams output, actual usage is reconciled. If the output quota is hit mid-generation, the stream stops with a cutoff reason. This prevents attackers from bypassing limits by issuing a small number of requests with huge output allowances.
| Dimension | What It Measures | Primary Risk Mitigated | Implementation Complexity |
|---|---|---|---|
| Requests Per Minute (RPM) | Number of API calls | Basic spam, simple DoS | Low |
| Tokens Per Minute (TPM) | Total input + output tokens | Resource exhaustion, high-cost queries | Medium |
| Concurrency | Simultaneous active requests | GPU saturation, latency spikes | High |
| Spend Cap | Monetary cost accrued | Denial of Wallet (DoW) | Medium |
Detecting Abuse: Beyond Simple Thresholds
Rate limiting answers "how much," but abuse detection answers "who" and "why." A steady stream of requests might be a bot farm. A sudden spike from a new account might be a trial abuser. You need signals that differentiate malicious automation from legitimate viral growth.
Start with behavioral anomalies. Track request velocity over multiple time windows-1 minute, 15 minutes, and 24 hours. A sudden fivefold increase in token consumption is a red flag. Look for patterns like repeated prompts (indicating scraping or extraction attempts), excessive tool calls in agentic workflows, or unusual model selection (e.g., a free-tier user suddenly requesting the most expensive reasoning model).
Identity signals matter too. Newly created accounts with disposable emails, credential reuse across IPs, and geographic mismatches are strong indicators of Sybil attacks. However, don't rely solely on IP reputation. Corporate NATs and mobile carriers share IPs, leading to false positives if you block entire subnets. The best approach combines identity, device fingerprinting, and behavioral scoring.
Graduated responses are crucial. Don't just block. Start by adding friction: require reauthentication, reduce quotas temporarily, or downgrade the user to a cheaper model. Hard blocks should be reserved for clear-cut cases like stolen keys or persistent violations after warnings.
Provider-Specific Nuances You Must Know
Each major provider handles limits differently, and ignoring these details leads to broken integrations. Here is what you need to know about the current landscape as of late 2026.
OpenAI applies limits at the organization and project level, not just per key. They expose detailed headers like x-ratelimit-remaining-tokens. Crucially, they distinguish between rate limits (temporary bursts) and usage limits (monthly spend). Honoring the Retry-After header is mandatory; ignoring it causes retry storms that worsen outages.
Anthropic uses usage tiers (Start, Build, Scale) that consolidate limits across models like Claude Sonnet and Haiku. Their recent updates allow programmatic inspection of workspace quotas via a dedicated Rate Limits API, giving you finer control over team-based spending.
Google Gemini measures RPM, TPM, and Requests Per Day (RPD). Quotas apply per project, not per API key. Values vary significantly by model and tier, so hard-coding generic limits is a mistake. Always query the Google AI Studio for current project-specific quotas.
Amazon Bedrock assigns quotas based on account history, region, and specific model endpoints. AWS warns that quotas are upper bounds, not guarantees. Sudden bursts can trigger ThrottlingExceptions even if average traffic is low. Use bounded concurrency and exponential backoff with jitter to smooth traffic.
Handling Retries and Client-Side Resilience
Your backend logic is only half the battle. Your clients must behave correctly when they get rejected. Treating HTTP 429 (Too Many Requests) as a fatal error is common, but treating it as a signal to hammer the server harder is worse.
Implement capped exponential backoff with random jitter. When a client receives a 429, it should wait for the duration specified in the Retry-After header. If that header is missing, wait for a base delay multiplied by an exponent, plus a random value to prevent synchronized retries from all clients hitting the server at once. Set a maximum retry count and a total deadline. If the deadline passes, fail gracefully.
Idempotency keys are your friend here. If a network glitch causes a timeout, the client might retry. Without an idempotency key, you generate two responses and pay twice. With it, the server recognizes the duplicate request and returns the cached result without re-running inference.
Building Your Defense-in-Depth Strategy
No single control is enough. IP blocking fails behind NATs. Account limits fail against stolen keys. Content filters miss economic attacks. You need layers.
Layer 1 is the Edge Gateway. Validate authentication, check IP reputation, and enforce basic RPM/TPM limits. Layer 2 is the Application Logic. Enforce per-user budgets, concurrency semaphores, and tool-call limits. Layer 3 is Anomaly Detection. Run near-real-time analysis to spot unusual patterns and trigger automated actions like quarantine or human review. Layer 4 is Observability. Log everything: tenant ID, token counts, latency, status codes, and policy decisions. But be careful with privacy-redact sensitive prompt content in logs.
A practical rollout takes weeks, not days. Start by inventorying your models and costs. Then, add authentication and basic token estimation. Next, implement concurrency controls and 429 handling. Finally, layer on anomaly detection and dashboards. Test with simulated abuse scenarios to tune false positives. You want to catch the bots without annoying the humans.
Why is RPM alone insufficient for LLM APIs?
RPM treats all requests as equal, but LLM costs vary wildly based on token count and complexity. A single request with a large context window can consume 10x the resources of a short query. Relying on RPM allows attackers to exhaust resources with fewer, heavier requests, leading to uneven load distribution and potential service degradation.
What is 'Denial of Wallet' in the context of AI?
Denial of Wallet (DoW) occurs when an attacker forces the service provider to incur significant financial costs, potentially bankrupting the operation, even if the service remains technically available. Unlike traditional DoS, which aims to crash the server, DoW aims to maximize the bill through expensive, high-volume inference requests.
How should I handle HTTP 429 errors in my application?
Do not retry immediately. Honor the Retry-After header if present. If absent, use exponential backoff with jitter to spread out retries. Implement a maximum retry limit and a total timeout to prevent infinite loops. Ensure your client logic accounts for the fact that retries still count toward your quota.
Can I rely on IP addresses to block abusive users?
Not exclusively. IP-based blocking often results in false positives because corporate networks, mobile carriers, and VPNs share IP addresses among many users. Attackers can also rotate IPs easily. Use IP reputation as one signal among many, combined with identity verification and behavioral analysis.
What is the role of concurrency limits in LLM infrastructure?
Concurrency limits restrict the number of simultaneous generation processes. This protects GPU memory and processing power from being saturated by a few heavy tasks. It ensures fair access for other users and maintains acceptable latency levels during peak traffic.
