Using Geonode Proxies for Web Scraping

Why scrapers need proxies

Websites limit how many requests a single IP can make, and many treat traffic from hosting providers with suspicion. A scraper running from one server hits rate limits quickly, and on protected sites it may be blocked outright. Proxies spread requests across many IPs and, with residential proxies, make those requests come from ordinary consumer connections.

Geonode suits scraping because it offers the three products scrapers need under one account: rotating residential proxies for protected targets, rotating datacenter proxies for easy ones, and a Scraper API for the targets that aren't worth the engineering. This guide shows how to use each, with settings that work. For the general theory, see web scraping proxies.

Pick the product per target, not per project

The most expensive scraping mistake is sending every request through the most trusted (and most expensive) proxy. Instead, sort your targets.

Test each target with a few hundred requests on the cheaper option before moving it up the table.

Setting up Geonode for a scraper

1. In the Geonode dashboard, open the residential proxy configuration.
2. Choose rotating sessions, your target country, and the host:port:username:password endpoint format.
3. Copy the values into environment variables so they never appear in your code.
4. Test with a single request before writing the full scraper.

import os
import requests

HOST = os.environ["GEONODE_HOST"]
PORT = os.environ["GEONODE_PORT"]
USER = os.environ["GEONODE_USER"]
PASS = os.environ["GEONODE_PASS"]

def proxy(country=None):
    user = f"{USER}-country-{country}" if country else USER
    url = f"http://{user}:{PASS}@{HOST}:{PORT}"
    return {"http": url, "https": url}

r = requests.get("https://api.ipify.org", proxies=proxy("us"), timeout=30)
print(r.text)

Geonode's documentation shows targeting added to the username with suffixes such as -country-jp. If your dashboard shows a different format, use that. More detail in Geonode with Python.

Rotation and sessions

• Rotate per request for independent pages: product pages, articles, listings. Each request gets a new IP from the pool.
• Use sticky sessions when the site sets cookies you need across pages, for pagination, search flows, or anything after a login. Generate a sticky endpoint in the dashboard and keep it for the flow. Geonode lists sticky sessions of up to 24 hours, but shorter is better for scraping: a few minutes per flow keeps identities fresh.
• Don't rotate mid-flow. A session that jumps between IPs looks suspicious and often breaks.

See sticky vs rotating sessions and proxy rotation strategies.

Handling blocks properly

A good scraper treats blocks as data. Three rules:

1. Check content, not status codes. Many blocks return a 200 with a CAPTCHA or "access denied" page. Look for an element that only appears on real pages.
2. Retry with a new IP and a delay. Cap retries at two or three; a target that blocks every attempt needs a different approach, not more attempts.
3. Slow down automatically. Track blocks per domain per minute and reduce concurrency when they rise.

import time

def fetch(url, marker, attempts=3):
    for n in range(attempts):
        try:
            r = requests.get(url, proxies=proxy("us"), timeout=30)
            if r.status_code == 200 and marker in r.text:
                return r.text
        except requests.RequestException:
            pass
        time.sleep(1.5 * (n + 1))
    return None

If a target still blocks residential requests, the problem is usually your client (headers, TLS fingerprint) or your rate, not the IP. See avoiding IP bans when scraping and browser fingerprinting and proxies.

Keeping bandwidth costs down

Residential traffic is billed per GB, so page weight is your biggest cost driver.

• Fetch HTML with an HTTP client rather than a browser wherever the data is in the HTML.
• Look for JSON endpoints in the browser's network tab; they're often a fraction of the page size.
• Block images, media and fonts when you must use a browser. See Geonode with Puppeteer.
• Don't re-fetch unchanged pages. Use conditional requests where supported, and schedule slow-changing pages less often.
• Move easy targets to datacenter traffic, which costs far less per GB.

Because Geonode's residential bandwidth doesn't expire, savings carry forward rather than vanishing at the end of the month. See cheap proxies for more techniques.

When to use the Scraper API instead

For targets where you'd otherwise need a headless browser, stealth tweaks and a lot of maintenance, Geonode's Scraper API can be simpler: send a URL, get back HTML or Markdown, optionally rendered with JavaScript. It's priced per request, with a free monthly allowance at the time of writing. A sensible pattern is to run your own scraper with proxies for most targets and route a handful of awkward ones through the API. See the Geonode Scraper API and scraping API vs proxies.

Scraping responsibly

• Respect robots.txt and rate limits; never degrade a site for its users.
• Avoid collecting personal data unless you have a lawful basis and a real need.
• Check the site's terms, particularly if you've created an account.
• Use proxies for legitimate data collection only.

See ethical web scraping and are proxies legal?.

FAQ

Which Geonode product is best for web scraping?

Rotating residential for protected sites, rotating datacenter for lenient ones, and the Scraper API for a few hard, low-volume targets. Most scrapers use a mix.

Should I rotate on every request?

For independent pages, yes. For anything with cookies or a multi-step flow, use a sticky session for the flow.

How much residential bandwidth will I need?

Measure the size of your target pages with your real fetch method, multiply by monthly page count, and add 20 to 30% for retries. HTML-only scraping typically needs a fraction of what a browser-based scraper uses.

Can I use Geonode with Scrapy?

Yes. Set the proxy in request meta through a downloader middleware. See Geonode with Scrapy.