Rate Limiting and Proxies: How Websites Throttle Traffic and How to Respect It

What rate limiting is

Rate limiting caps how many requests a client can make in a time window. Websites use it to protect their servers, ensure fair access and stop abuse. When you exceed a limit, you typically see:

• HTTP 429 Too Many Requests, often with a Retry-After header.
• HTTP 503 Service Unavailable or custom error pages.
• Silent throttling: slower responses or degraded content.
• Escalation: CAPTCHAs or blocks if you keep going.

What websites rate-limit on

Proxies solve per-IP limits, and help with subnet limits if your pool is diverse. They don't solve the rest. See IP reputation and proxies and proxy pool size.

How many IPs do you need?

IPs in rotation ≈ target requests per minute ÷ safe requests per minute per IP

Find the safe rate empirically: start slow on one IP and increase until you see 429s or challenges, then use a comfortable fraction of that.

Building good throttling

Per-domain concurrency limits

Even with thousands of IPs, cap concurrent requests to each domain. A spike that looks like a DDoS will trigger domain-wide defences.

Token bucket per IP and per domain

import time, threading

class TokenBucket:
    def __init__(self, rate_per_sec, burst):
        self.rate, self.capacity = rate_per_sec, burst
        self.tokens, self.last = burst, time.monotonic()
        self.lock = threading.Lock()

    def take(self):
        while True:
            with self.lock:
                now = time.monotonic()
                self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
                self.last = now
                if self.tokens >= 1:
                    self.tokens -= 1
                    return
            time.sleep(0.05)

domain_bucket = TokenBucket(rate_per_sec=5, burst=10)  # 5 req/s to this domain overall

Respect Retry-After

import time
def wait_retry_after(resp, default=10):
    ra = resp.headers.get("Retry-After")
    try:
        time.sleep(int(ra))
    except (TypeError, ValueError):
        time.sleep(default)

Exponential backoff with jitter

On 429 or 503, wait base × 2^attempt + random jitter, and switch IP if the problem is per-IP.

Adaptive throttling

Tools like Scrapy's AutoThrottle adjust speed based on response latency. See Scrapy proxies.

Rate limits and rotating gateways

With rotating residential gateways, you control rotation via sessions. Remember:

• Per-request rotation spreads load well for stateless pages.
• Sticky sessions concentrate requests on one IP; keep request counts per session modest.
• HTTP keep-alive can keep you on one exit longer than you think.

See proxy rotation strategies.

Being a good citizen

Rate limits exist for a reason. Scraping at a pace that degrades a site for real users is harmful and invites legal and technical escalation. Scrape during off-peak hours where possible, cache aggressively, and fetch only what you need. See ethical web scraping.

Discovering a site's limits safely

1. Start with one IP at a very low rate (for example, one request every few seconds).
2. Gradually increase, watching for 429s, challenges or slower responses.
3. Note the rate at which problems start; operate well below it.
4. Repeat occasionally; limits change.

Don't run discovery aggressively on small sites. If a site publishes API rate limits, use those.

Rate limiting in Scrapy, Python and Node

See Scrapy proxies, Python requests proxies and Node.js proxies.

Distributed rate limiting

When several workers or servers hit the same domain, coordinate limits centrally (for example, a shared Redis-based token bucket). Otherwise each worker obeys its own limit and the total overwhelms the target.

Interpreting rate-limit responses

See avoiding IP bans when scraping and proxy rotation strategies.

Reading rate-limit headers

Many sites and most APIs tell you their limits, if you look at the response headers. Common ones:

A small helper makes your client respect them:

import time

def wait_if_limited(r):
    if r.status_code in (429, 503) and "Retry-After" in r.headers:
        value = r.headers["Retry-After"]
        time.sleep(int(value) if value.isdigit() else 60)
    elif r.headers.get("X-RateLimit-Remaining") == "0":
        reset = r.headers.get("X-RateLimit-Reset", "")
        if reset.isdigit():
            time.sleep(max(0, int(reset) - time.time()))

The date form of Retry-After falls back to a minute here; parse it properly if a target uses it. Remember that limits are usually per IP, per account or per API key. Rotating proxies spread per-IP limits, but they don't reset per-account ones, and dodging a stated limit is exactly what sites object to. MDN documents Retry-After and the 429 status. See also ethical web scraping and Scrapy proxies.

Key takeaways

• Websites rate-limit by IP, subnet, session, account, fingerprint and overall domain load; proxies only address the first two.
• Estimate IPs needed as target requests per minute divided by safe requests per minute per IP.
• Always cap per-domain concurrency, add jitter, honour Retry-After and use exponential backoff.
• Coordinate limits centrally when multiple workers hit the same domain.
• Rate limits protect real users; scraping at a pace that degrades a site is harmful and invites escalation. See avoiding IP bans when scraping.

FAQ

Can proxies bypass rate limits completely?

They bypass per-IP limits. Domain-wide, session and account limits still apply.

What does a 429 response mean?

You've exceeded a rate limit. Slow down, honour Retry-After, and consider rotating IPs.

How fast can I scrape a site?

It varies by site. Measure, start slow and scale carefully.