Proxies in research
Researchers collect web data for studies of prices, online discourse, misinformation, platform behaviour, algorithmic personalisation, accessibility, labour markets and much more. Two questions come up again and again: do I need proxies, and how do I use them in a way that will satisfy an ethics board, a journal's reviewers and my own standards?
The answer to the first is often "less than you think". The answer to the second is about transparency and restraint. This guide covers both, with Geonode as an example provider. Geonode's pay-as-you-go residential billing with non-expiring bandwidth suits research budgets, which are typically small and project-based.
Do you need proxies at all?
| Situation | Proxies needed? |
|---|---|
| Collecting from sites that publish data openly, at low rates | Usually not |
| Using official APIs or data access programmes | No; use the API |
| A site rate-limits your institution's IP | Possibly datacenter proxies to spread load politely |
| Studying how content varies by country or city | Yes, residential with location targeting |
| A site blocks university or cloud IP ranges | Possibly residential, if collection is otherwise appropriate |
| Studying personalisation or algorithmic differences | Yes, with careful experimental design |
Start without proxies. Add them only when the research design requires location variation, or when polite collection is still being blocked and the collection is otherwise justified.
Research designs that need location
Some of the most interesting web research is comparative: how do search results, prices, news recommendations, ad delivery or content moderation differ across countries? For these, location is the independent variable, and residential exits targeted by country or city are the instrument. Design carefully:
- Control everything else: fresh browser profiles, the same device type, the same time window, the same language settings where appropriate.
- Replicate: multiple sessions per location to separate real differences from noise.
- Verify location: record the exit IP's location and what the site reports.
- Randomise order of locations to avoid time-of-day bias.
Ethics and approvals
- Ethics review: many institutions require review for web data collection, especially when it involves people's content or personal data.
- Personal data: collect the minimum, pseudonymise early, secure storage, defined retention. Data protection law applies to research too, though some laws provide research exemptions with conditions.
- Platform terms: some platforms offer researcher access programmes; use them where available.
- Harm to sites: polite rates, off-peak collection, caching, and never degrading a service.
- Harm to peers: residential traffic flows through real people's connections. Use it proportionately and choose providers with clear consent practices. Geonode states its residential IPs come from opt-in partners.
Proxy tip: Before scaling, run a short pilot on your real targets with Geonode and record success rate, latency and GB per result. Our our complete Geonode guide explains which Geonode product to start with.
Documenting proxy use in your methods
Reviewers increasingly ask how data was collected. Record and report:
- Provider and product type (residential, datacenter)
- Targeting level (country, city) and how location was verified
- Session settings (rotating or sticky, duration)
- Request rates and collection dates
- How blocks and failures were detected and handled
- Limitations, such as geolocation inaccuracy
This makes your study reproducible and defensible.
Budget-friendly setup
- Start with no proxy, then datacenter, then residential only for the parts that need it.
- Fetch HTML only; avoid images and media unless they're your data.
- Buy a small non-expiring residential balance for the project.
- Ask providers about academic or nonprofit discounts.
See cheap proxies.
A worked example: comparing search results across countries
A media researcher wants to study how search results for health-related queries differ across five countries.
- Query set: 40 queries, translated by native speakers for each country.
- Instrument: a headless browser with fresh profiles, routed through Geonode residential exits targeted to each country, with the search engine's language and region settings set to match.
- Protocol: each query run three times per country per day, at randomised times, for two weeks.
- Data: result URLs, positions, SERP features and the location the search engine reported.
- Ethics and limits: low request rates, no personal data collected, the study's method and proxy settings documented in the paper's appendix.
The design separates genuine country differences from day-to-day noise, and the documentation lets reviewers judge whether geolocation inaccuracy could affect conclusions.
Questions reviewers often ask
- How did you verify that exits were in the intended locations?
- How did you prevent personalisation from earlier sessions?
- How did you handle blocked or failed requests, and how many were there?
- Could the proxy provider's network characteristics have biased results?
- Was collection consistent with the platforms' terms and your ethics approval?
Answering these in your methods section, with numbers where possible, makes proxy-based research far more convincing. See ethical web scraping.
FAQ
Will reviewers object to proxy use?
Not if it's justified by the design and documented clearly. Undisclosed or unjustified evasion of blocks is what raises concerns.
Can I use proxies to get around a platform's research restrictions?
That's risky ethically and legally. Use official research access where it exists, and get advice from your institution.
How do I verify my exits are where I think?
Log exit IPs, look them up in more than one geolocation database, and check what the target site displays.
What's a typical research budget for proxies?
Often small: a few GB of residential traffic for a comparative study, plus cheap datacenter traffic for bulk collection.
Editorial note: Proxy providers change their pools, features and prices often, so treat any figures here as typical ranges at the time of writing and confirm current terms with the provider before you buy.