Why ethics matters in scraping
Web scraping powers price comparison, research, journalism, search engines, market intelligence and much more. It can also overload small sites, harvest personal data, copy creative work, and erode trust. Proxies are neutral tools, but they make it easier to scrape at scale and harder to attribute traffic, which raises the stakes.
These principles help you collect data in a way that's defensible legally, ethically and reputationally. They're not legal advice; see are proxies legal? for the legal context.
1. Prefer official access
Before scraping, check for an API, data feed, licence, partnership or public dataset. Official access is more reliable and avoids most legal questions.
2. Read robots.txt and the terms
robots.txt expresses the site owner's crawling preferences. Terms of service may restrict automated access. You may decide some restrictions don't apply to your use, but make that a conscious, documented decision, ideally with legal input, not an oversight.
3. Don't harm the site
- Limit concurrency and request rates per domain.
- Scrape during off-peak hours where possible.
- Honour
Retry-Afterand back off on errors. - Cache aggressively and don't re-fetch unchanged pages.
A scraper that slows a site for real users is causing harm. See rate limiting and proxies.
4. Minimise personal data
Avoid collecting personal data unless it's essential and you have a lawful basis. If you collect it, apply data protection principles: purpose limitation, minimisation, security, retention limits and rights handling under GDPR, UK GDPR, CCPA and similar laws.
5. Respect creative work
Facts (like prices) are generally treated differently from creative expression (like articles, photos and reviews). Republishing copyrighted content wholesale is a different matter from extracting factual data points.
6. Don't bypass authentication
Accessing pages behind logins you're not authorised to use, or circumventing access controls, moves from scraping into unauthorised access.
7. Be transparent where you can
For large crawlers, identify yourself with a descriptive user agent and a contact URL, and publish how site owners can opt out. Not every use case allows this, but transparency builds trust.
Proxy tip: Want to skip proxy management on some targets? Geonode offers both raw proxies and a Scraper API that returns HTML or Markdown with optional JavaScript rendering. We compare the two approaches in responsible research with Geonode.
8. Use ethically sourced proxies
Residential IPs belong to real people. Choose providers that obtain informed consent and compensate peers. See ethically sourced proxies and how proxy providers source IPs.
9. Secure what you collect
Protect scraped data like any business data. Breaches of scraped datasets cause real harm.
10. Know when not to scrape
- When the data is sensitive (health, finance, children).
- When the site has clearly and specifically prohibited it and you have no strong justification.
- When your volume would materially burden a small site.
- When the purpose is deceptive or harmful.
A simple ethics checklist
| Question | Yes/No |
|---|---|
| Is there an official API or licence we could use instead? | |
| Have we reviewed robots.txt and terms? | |
| Are request rates safe for the target? | |
| Are we avoiding or lawfully handling personal data? | |
| Are we using consent-based proxy networks? | |
| Do we have a documented purpose and retention policy? |
Building ethics into your pipeline
| Pipeline stage | Ethical control |
|---|---|
| Target selection | Check for APIs, licences, robots.txt and terms |
| Collection | Rate limits, off-peak scheduling, caching |
| Processing | Remove personal data you don't need |
| Storage | Access controls, encryption, retention limits |
| Use | Purpose limitation; no republishing of creative content |
| Review | Periodic audits of targets and data |
Handling requests from site owners
If a site owner contacts you:
- Respond promptly and respectfully.
- Pause collection while you review.
- Consider their concerns (load, data type, terms).
- Adjust or stop, and document the outcome.
Many disputes end quickly with a polite conversation and a slower crawl rate.
Writing a one-page scraping policy
Good intentions don't survive staff changes; a written policy does. One page is enough, and it gives engineers clear answers before they ask:
| Section | What it says |
|---|---|
| Purpose | What the data is for, and what it must not be used for |
| Sources | Which sites or types of sites are in scope, and how new ones are approved |
| Access rules | Prefer APIs; respect robots.txt; no logins or paywalls without permission |
| Rates | Default limits per site, and who can approve exceptions |
| Identification | The user agent and contact address you use, where appropriate |
| Personal data | What you avoid collecting, how you minimise it, and the lawful basis if you need it |
| Retention | How long raw pages and extracted data are kept, and how they're deleted |
| Proxies | Which providers you use and why they meet your sourcing standards |
| Contact and complaints | How site owners reach you and how quickly you respond |
| Review | Who owns the policy and when it's next reviewed |
Keep it next to the code, link it from your crawler's contact page if you publish one, and review it at least once a year or whenever you add a new kind of source.
For the proxy side of the policy, see ethically sourced proxies. For the legal background, see are proxies legal, and for AI-specific obligations, AI training data proxies.
Key takeaways
- Prefer official APIs, licences and datasets before scraping.
- Review robots.txt and terms deliberately, and document your decisions.
- Never degrade a site for real users: limit rates, cache and scrape off-peak.
- Minimise personal data and apply data protection principles to anything you keep.
- Respect creative work, don't bypass authentication, and be transparent where possible.
- Use consent-based proxy networks. See ethically sourced proxies.
FAQ
Is web scraping unethical?
Not inherently. It depends on what you collect, how, and why.
Does using proxies make scraping unethical?
No. Proxies are standard infrastructure. Using them to cause harm or evade legitimate opt-outs is the problem.
Should I always obey robots.txt?
It's best practice to honour it. Some contexts (such as archiving or research) have nuanced views; decide deliberately.
Editorial note: Proxy providers change their pools, features and prices often, so treat any figures here as typical ranges at the time of writing and confirm current terms with the provider before you buy.