Rayobyte vs Geonode for Scrapy: Proxy Middleware, Routing and Costs

Scrapy setup on both

Scrapy reads request.meta["proxy"]. Put a Rayobyte or Geonode proxy URL there, credentials included, and the built-in middleware does the rest. A small custom middleware can route each domain to a different product, or a different provider. See Rayobyte with Scrapy, Geonode with Scrapy and Scrapy proxies.

Routing across both providers

TIERS = {
    "ry_dc": os.environ["RAYOBYTE_DC_PROXY"],
    "gn_resi": os.environ["GEONODE_RESI_PROXY"],
    "ry_unblock": os.environ["RAYOBYTE_UNBLOCKER_PROXY"],
}
ROUTES = {"hard.example": "gn_resi", "impossible.example": "ry_unblock"}

class ProxyRouter:
    def process_request(self, request, spider):
        host = urlparse(request.url).hostname or ""
        request.meta.setdefault("proxy", TIERS[ROUTES.get(host, "ry_dc")])

Because Rayobyte's Web Unblocker is a proxy endpoint, it fits Scrapy's proxy model directly. Geonode's Scraper API is an HTTP API, so you'd call it as a URL rather than a proxy.

Pagination sessions

Pass a session ID through meta and reuse it with response.follow(). Rayobyte sessions last up to 120 minutes and Geonode's up to 24 hours; a typical paginated crawl needs minutes.

Concurrency

Set CONCURRENT_REQUESTS below your plan's limit and CONCURRENT_REQUESTS_PER_DOMAIN low enough to be polite. Rayobyte's Premium Residential lists unlimited concurrency; Scale Residential and monthly rotating ISP allow one request per dollar.

Costs for a typical crawl

A 500,000-page crawl at 120 KB per page is about 60 GB. On Rayobyte rotating datacenter that's roughly $17 if the sites allow it. On residential, Geonode is cheaper per GB at that size, while Rayobyte would cost $120 at the $2 tier. See pricing compared.

Which to choose

With Webshare too: Scrapy across all three.

A real-world example

A Scrapy project crawls 300 sites nightly. A downloader middleware routes 250 sites to Rayobyte rotating datacenter, 45 to Geonode residential and 5 to Rayobyte's Web Unblocker. Scrapy stats record responses by provider and status, so each morning's report shows block rates per route. Sites whose block rate passes 10% on datacenter are promoted to residential automatically for the next run.

Pros and cons for Scrapy

Rayobyte

• Every product, including the unblocker, fits Scrapy's proxy model.
• Cheapest datacenter per GB.

Geonode

• Cheaper residential at modest volume.
• Crawl API as an alternative for discovery.
• API-based managed scraping needs request changes, not just a proxy URL.

If you're still shortlisting, compare these options with the wider market in our best proxies guide, and use our cheap proxies guide to estimate a sensible budget. To understand the IP type behind this, read our residential proxies guide.

Key takeaways

• Both plug into Scrapy through request.meta["proxy"].
• A router middleware can mix products and providers per domain.
• Rayobyte's Web Unblocker fits Scrapy's proxy model directly.
• Geonode is cheaper for modest residential; Rayobyte for datacenter.

FAQ

Do I need a rotation package for Scrapy?

Not with rotating gateways on either provider.

Can Scrapy use Geonode's Scraper API?

Yes, by requesting the API URL with the target as a parameter instead of using it as a proxy.

Which is cheaper for big Scrapy crawls?

On lenient sites, Rayobyte datacenter. On protected sites at modest volume, Geonode residential.

Sources and further reading

Independent references that back up the points above, from standards bodies, governments, universities and non-profits. They open in a new tab.

• Downloader Middleware (Scrapy documentation): Scrapy's documentation for the middleware where proxy rotation lives.
• urllib.request documentation (Python Software Foundation): Python's standard library documentation, including ProxyHandler.
• Web scraping (Wikipedia): Encyclopedia overview of web scraping techniques and legal history.
• Proxy servers and tunneling (MDN Web Docs (Mozilla)): Mozilla's developer explainer on forward proxies, reverse proxies and tunneling.
• RFC 7617: The Basic HTTP Authentication Scheme (IETF / RFC Editor): The scheme most proxy providers use for username and password authentication.