Rayobyte vs Geonode vs Webshare for Scrapy

Scrapy on all three

Set request.meta["proxy"] to a proxy URL from any of the three. A small middleware can route each domain to a different provider:

ROUTES = {"easy.example": "ws", "hard.example": "gn", "impossible.example": "ry_unblock"}

class Router:
    def process_request(self, request, spider):
        host = urlparse(request.url).hostname or ""
        request.meta.setdefault("proxy", PROXIES[ROUTES.get(host, "ws")])

See Scrapy proxies and Rayobyte with Scrapy.

Differences

Picking

Webshare lists for easy domains, Geonode residential for protected ones at modest volume, Rayobyte's Web Unblocker for the impossible few. Keep per-IP concurrency low on lists.

Settings that work with all three

CONCURRENT_REQUESTS = 64
CONCURRENT_REQUESTS_PER_DOMAIN = 8
DOWNLOAD_TIMEOUT = 30
RETRY_TIMES = 2
RETRY_HTTP_CODES = [403, 429, 500, 502, 503, 504]
AUTOTHROTTLE_ENABLED = True

Keep total concurrency under your plan's limits (Rayobyte's monthly plans allow one concurrent request per dollar), and keep per-IP concurrency low on Webshare lists, since many requests share each IP. See Rayobyte with Scrapy, Geonode with Scrapy and Webshare with Scrapy.

Handling JavaScript pages

Scrapy doesn't run JavaScript. For the few pages that need it, use scrapy-playwright with a provider's proxy in the launch options, or send those URLs to Geonode's Scraper API or Rayobyte's Scraping API and parse the returned HTML in your spider.

Strengths and watch-outs

Bottom line

A routing middleware with three routes covers almost any Scrapy project.

For the bigger picture beyond our three partners, read our best proxies buyer's guide, and see cheap proxies for ways to cut cost per successful request. To understand the IP type behind this, read our residential proxies guide.

For head-to-head versions of this comparison, see Rayobyte vs Geonode for Scrapy, Rayobyte vs Webshare for Scrapy, Webshare vs Geonode for Scrapy.

Key takeaways

• All three plug into request.meta["proxy"].
• A routing middleware mixes providers per domain.
• Rayobyte's unblocker fits Scrapy natively; Geonode adds crawl APIs.

FAQ

Do I need a rotation package?

Not with gateways; a small middleware handles Webshare lists.

Which is cheapest for big Scrapy crawls?

Webshare lists on lenient sites.

Can Scrapy call Geonode's API?

Yes, as normal requests to the API URL.

Can Scrapy use SOCKS5?

Not natively; use HTTP proxies with Scrapy on all three.

How do I keep one IP across paginated requests?

Pass a session ID through meta and reuse the same proxy URL with response.follow().

Can scrapy-playwright use these proxies?

Yes; pass the proxy in Playwright's launch or context options.

Sources and further reading

Independent references that back up the points above, from standards bodies, governments, universities and non-profits. They open in a new tab.

• Downloader Middleware (Scrapy documentation): Scrapy's documentation for the middleware where proxy rotation lives.
• RFC 6585: Additional HTTP Status Codes (429 Too Many Requests) (IETF / RFC Editor): Defines status 429, the code sites send when you make too many requests.
• urllib.request documentation (Python Software Foundation): Python's standard library documentation, including ProxyHandler.
• Retry-After header (MDN Web Docs (Mozilla)): The header that tells clients how long to wait before retrying.
• Internet Engineering Task Force (IETF): The body that publishes the internet standards proxies are built on.