Proxy Rotation Strategies for Scraping and Automation

Rotation is a design decision

"Use rotating proxies" is the start of a strategy, not the whole thing. How you choose which IP to use for each request affects success rates, costs and how quickly you get blocked. If your provider rotates for you (a backconnect gateway), your strategy is mostly about sessions and retries. If you manage a list of static proxies, you decide everything. For the basics, see what is proxy rotation.

Strategy 1: Round-robin

Cycle through proxies in order. Simple and spreads load evenly.

import itertools
pool = itertools.cycle(proxies)
proxy = next(pool)

Weakness: keeps using bad proxies; predictable patterns.

Strategy 2: Random

Pick a random proxy each time. Avoids patterns, but load is uneven in small pools.

Strategy 3: Weighted by health

Track success and failures per proxy and prefer healthy ones.

import random
from collections import defaultdict

stats = defaultdict(lambda: {"ok": 1, "fail": 1})

def pick():
    weights = [stats[p]["ok"] / (stats[p]["ok"] + stats[p]["fail"]) for p in proxies]
    return random.choices(proxies, weights=weights, k=1)[0]

def record(p, ok):
    stats[p]["ok" if ok else "fail"] += 1

Strategy 4: Cooldowns

After a proxy hits a rate limit or block, rest it for a period before reuse.

import time
cooldown_until = {}

def available():
    now = time.time()
    return [p for p in proxies if cooldown_until.get(p, 0) <= now]

def penalise(p, seconds=300):
    cooldown_until[p] = time.time() + seconds

Strategy 5: Subnet-aware rotation

Websites often apply limits per /24 subnet. Avoid sending consecutive requests from neighbouring IPs, and spread load across subnets:

from collections import defaultdict
by_subnet = defaultdict(list)
for p in proxies:
    ip = p.split("@")[-1].split(":")[0]
    by_subnet[ip.rsplit(".", 1)[0]].append(p)
subnet_cycle = itertools.cycle(list(by_subnet))
def pick_subnet_aware():
    return random.choice(by_subnet[next(subnet_cycle)])

Strategy 6: Sticky sessions per task

For flows that need the same IP (logins, pagination, checkouts), assign one proxy or session ID per task and keep it until the task finishes or fails. See sticky vs rotating sessions.

Strategy 7: Per-domain pools

Keep separate health stats per target domain. A proxy blocked on one site may be fine on another.

Strategy 8: Multi-tier waterfall

Route each request through the cheapest tier first and escalate on failure:

TIERS = ["datacenter", "residential", "unblocker"]

def fetch_waterfall(url):
    for tier in TIERS:
        r = fetch_with_tier(url, tier)
        if r is not None and not looks_blocked(r):
            return r, tier
    return None, None

Track which tier each domain ends up needing, and start future requests for that domain at that tier. This can dramatically reduce costs compared with sending everything through residential. See cheap proxies.

Choosing a strategy

Monitoring

Log per request: proxy (or session), domain, status, block detection, latency and bytes. Review success rates by proxy and by domain. Without data, rotation tuning is guesswork.

Combining strategies

Real systems combine several strategies:

for each request:
  pool = per-domain pool (health-weighted, subnet-aware)
  if task needs a session → reuse task's assigned proxy (sticky)
  else → pick from pool excluding proxies in cooldown
  on success → record ok
  on block → record fail, cooldown proxy, retry with next tier if retries exhausted

Metrics to drive rotation decisions

Rotation and fingerprints

If you rotate IPs but keep an identical, unusual fingerprint, sites can still group your requests. When rotating sessions, also rotate realistic browser identities (user agent family, viewport) and keep them consistent within each session. See browser fingerprinting and proxies.

Strategy by proxy product

See rotating residential proxies and datacenter proxies.

Rotation for different proxy pricing models

Rotation and target-specific memory

Some targets remember IPs for long periods; others forget within minutes. Learn each target's behaviour:

• Short memory: aggressive reuse of IPs after a short cooldown works.
• Long memory: treat blocked IPs as burned for that target and track them separately.

Keep a per-target blocklist of IPs (or subnets) that failed recently so your rotation avoids them.

Implementing per-target blocklists

import time
from collections import defaultdict

burned = defaultdict(dict)  # domain -> {proxy: expiry}

def is_burned(domain, proxy):
    exp = burned[domain].get(proxy)
    return exp is not None and exp > time.time()

def burn(domain, proxy, hours=24):
    burned[domain][proxy] = time.time() + hours * 3600

Combine with health-weighted selection: pick from proxies that aren't burned for the current domain.

For browser-specific patterns, see Puppeteer rotating proxies; for scale, Puppeteer cluster proxies.

Choosing a cooldown length from data

The cooldown strategy above needs a number: how long should a proxy rest after a block? Guessing leads to either wasted IPs (too long) or repeat blocks (too short). Measure it instead:

1. When a proxy gets blocked (403, 429 or a challenge) on a target, record the time and keep it out of the main rotation.
2. Probe it gently with a single light request at increasing intervals: 1 minute, 5, 15, 30, 60, 120.
3. Record the first interval at which it succeeds again.
4. Repeat for 30 to 50 blocked proxies on the same target.
5. Set the cooldown near the 75th percentile of recovery times. Most proxies will be ready when they return, and you don't sideline them for the slow few.

Recovery times differ by target and by block type: 429s often clear in minutes, while 403s from IP reputation may not clear at all. If a large share never recovers within a few hours, treat those as burned and replace them rather than cycling them back in.

Run the measurement again when results change; sites adjust their rules. Store cooldowns per target, since one site's five minutes can be another's five hours. For the related metrics, see the section on metrics above; for rate behaviour that avoids blocks in the first place, see rate limiting and proxies, and for reputation, IP reputation and proxies.

Key takeaways

• Round-robin and random rotation are simple but keep using bad proxies; health-weighted rotation with cooldowns performs better.
• Subnet-aware rotation avoids neighbouring IPs triggering subnet-level limits.
• Sticky assignment per task is essential for sessions and accounts.
• Per-domain health tracking and blocklists reflect that bans are site-specific.
• Multi-tier waterfalls keep most traffic on the cheapest working tier. See cheap proxies.

FAQ

Is random rotation better than round-robin?

Neither is best on its own. Health-weighted rotation with cooldowns usually beats both.

How long should a cooldown be?

Start with a few minutes for rate limits and longer for hard blocks, then tune by observing recovery.

Do I need rotation logic with a backconnect provider?

Less, but you still need session management and retries. See backconnect proxies.