Scaling Puppeteer With Proxies Using puppeteer-cluster

From one browser to many

A single Puppeteer script with one browser is fine for a few hundred pages. For tens of thousands, you need concurrency, error handling, retries and proxy management. puppeteer-cluster is a popular library that manages a pool of browsers, contexts or pages, distributes tasks to workers and handles failures. This guide shows how to combine it with proxies.

Concurrency models

puppeteer-cluster offers three concurrency models:

Simple setup: one rotating gateway for all workers

With a rotating residential gateway and session IDs, every task can get its own IP even though all browsers use the same proxy host:

import { Cluster } from 'puppeteer-cluster';

const GATE = 'http://gate.example.com:7777';
const session = () => `customer-USER-country-us-session-${Math.random().toString(36).slice(2, 10)}`;

const cluster = await Cluster.launch({
  concurrency: Cluster.CONCURRENCY_BROWSER,
  maxConcurrency: 8,
  retryLimit: 2,
  timeout: 60000,
  puppeteerOptions: { args: [`--proxy-server=${GATE}`] },
  monitor: false,
});

await cluster.task(async ({ page, data: url }) => {
  await page.authenticate({ username: session(), password: 'PASS' });
  await page.setRequestInterception(true);
  page.on('request', (r) => (['image', 'media', 'font'].includes(r.resourceType()) ? r.abort() : r.continue()));
  const res = await page.goto(url, { waitUntil: 'domcontentloaded' });
  const html = await page.content();
  if (!res || res.status() >= 400 || /captcha|access denied/i.test(html)) {
    throw new Error(`blocked ${res?.status()}`); // triggers retry
  }
  // parse and store html...
});

cluster.on('taskerror', (err, url) => console.error(`failed ${url}: ${err.message}`));

for (const url of urls) cluster.queue(url);
await cluster.idle();
await cluster.close();

Because CONCURRENCY_BROWSER uses separate browsers, each retry gets a fresh browser and connection, so the new session ID reliably gives a new IP.

Static proxy lists: a proxy per worker

With ISP or datacenter proxy lists, give each browser its own proxy. One approach is to run several clusters, one per proxy, or to manage your own pool of browsers keyed by proxy. Another is to use per-context proxies (in recent Puppeteer versions) inside the task:

await cluster.task(async ({ page, data }) => {
  // With CONCURRENCY_PAGE on a shared browser, create a context with its own proxy instead:
  const browser = page.browser();
  const proxy = nextProxy(); // { server, username, password }
  const ctx = await browser.createBrowserContext({ proxyServer: proxy.server });
  try {
    const p = await ctx.newPage();
    await p.authenticate({ username: proxy.username, password: proxy.password });
    await p.goto(data.url, { waitUntil: 'domcontentloaded' });
    // ...
  } finally {
    await ctx.close();
  }
});

See Puppeteer proxy per page and Puppeteer rotating proxies.

Tuning concurrency

• Start with a few workers per CPU core and measure memory.
• Keep concurrency per proxy IP modest to avoid rate limits.
• Cap concurrency per target domain regardless of proxy count. See rate limiting and proxies.

Cost control

Every browser page load through per-GB proxies costs money. Block images, fonts and media; skip third-party trackers; and consider ISP proxies with unlimited bandwidth for very heavy workloads. See headless browser proxies.

Monitoring

Log, per task: URL, proxy/session, status, block detection, duration and retry count. Watch success rate per proxy and per domain, and recycle browsers periodically to avoid memory growth.

A proxy pool helper for cluster tasks

class ProxyPool {
  constructor(proxies) {
    this.items = proxies.map((p) => ({ ...p, ok: 1, fail: 1, coolUntil: 0 }));
  }
  pick() {
    const now = Date.now();
    const live = this.items.filter((p) => p.coolUntil <= now);
    const pool = live.length ? live : this.items;
    // weight by success ratio
    const weights = pool.map((p) => p.ok / (p.ok + p.fail));
    let r = Math.random() * weights.reduce((a, b) => a + b, 0);
    for (let i = 0; i < pool.length; i++) { if ((r -= weights[i]) <= 0) return pool[i]; }
    return pool[pool.length - 1];
  }
  report(p, ok) {
    ok ? p.ok++ : p.fail++;
    if (!ok) p.coolUntil = Date.now() + 5 * 60 * 1000;
  }
}

Use pool.pick() inside your task to choose a proxy for a new context, and pool.report() after the page loads. Proxies that fail rest for five minutes. See proxy rotation strategies.

Memory and stability tips

• Recycle browsers every few hundred pages to avoid memory growth.
• Close contexts in finally blocks.
• Set timeout in cluster options so stuck tasks don't block workers.
• Use retryLimit and retryDelay for transient failures.
• Monitor with the cluster's built-in monitor during development (monitor: true).

Cost and throughput planning

Start small, measure pages per minute and cost per page, then scale. See Puppeteer proxies and cheap proxies.

Example: scraping 50,000 product pages

Plan:

• Rotating residential gateway, US targeting.
• CONCURRENCY_CONTEXT with maxConcurrency: 16 on an 8-core server.
• New session ID per task; images, fonts and media blocked.
• Retry limit 2; block detection on content.
• Per-domain cap of 8 concurrent pages for the target.

Rough estimates:

Measure the first 1,000 pages, then extrapolate cost and duration before running the rest. See proxy bandwidth pricing explained.

Choosing concurrency per proxy type

Graceful shutdown

On shutdown signals, stop queueing, wait for cluster.idle(), then cluster.close(). This avoids half-finished contexts holding proxy sessions and leaking resources.

For per-page proxy options, see Puppeteer proxy per page and the Puppeteer proxies pillar.

Different launch options per browser with perBrowserOptions

With Cluster.CONCURRENCY_BROWSER, each worker gets its own browser, which makes it the natural model for one proxy per worker. puppeteer-cluster supports this directly through perBrowserOptions, an array of launch options with one entry per browser:

import { Cluster } from 'puppeteer-cluster';

const proxies = ['http://10.0.0.1:8000', 'http://10.0.0.2:8000', 'http://10.0.0.3:8000'];

const cluster = await Cluster.launch({
  concurrency: Cluster.CONCURRENCY_BROWSER,
  maxConcurrency: proxies.length,
  perBrowserOptions: proxies.map((p) => ({ args: [`--proxy-server=${p}`] })),
});

When perBrowserOptions is set, puppeteerOptions is ignored, so put every launch setting you need into each entry. Authentication still happens per page with page.authenticate() inside your task, or through IP whitelisting.

Some practical notes:

• Match maxConcurrency to the number of entries, so every browser has a defined proxy.
• Log the exit IP at the start of each task in early runs, to confirm the mapping is what you expect.
• Check the library's activity. puppeteer-cluster is still released (0.25.0 arrived in November 2025), but it moves slower than Puppeteer itself; test after upgrading either. Its repository documents every option.

For rotating gateways, a single proxy for all browsers with different session IDs per task remains simpler. See Puppeteer rotating proxies and Puppeteer proxies.

FAQ

Which concurrency model is best with proxies?

CONCURRENCY_BROWSER is simplest for per-worker proxies; CONCURRENCY_CONTEXT is lighter if you use per-context proxies.

Can puppeteer-cluster rotate proxies automatically?

Not by itself. Combine it with a rotating gateway or your own proxy selection logic.

Is Playwright better for scaling?

Playwright's per-context proxy support is mature, and its test runner parallelises well. Both can scale. See Playwright proxies.