The e-commerce scraping landscape
E-commerce data falls into a few types: catalogs (products, categories, attributes), offers (price, stock, sellers), reviews and ratings, and search rankings within the store. Big marketplaces protect all of them; smaller shops often protect none. A good setup treats them differently. Our best proxies for e-commerce scraping guide gives the general view.
Rayobyte lets you mix datacenter, ISP, residential and managed products under one account, which fits this landscape well.
Product choices
| Site type | Product |
|---|---|
| Shopify and smaller independent stores | Rotating datacenter |
| Mid-size retailers with basic protection | Rotating ISP |
| Large marketplaces and retailers | Premium Residential |
| Country-specific storefronts | Premium Residential with country targeting |
| Sites with heavy JavaScript and challenges | Web Unblocker, Scraping API or rayobrowse |
Catalog crawls
Full catalog crawls are bandwidth-heavy. Start from sitemaps or category pages, collect product URLs, then fetch product pages. Use an HTTP client, extract structured data (many shops embed JSON-LD product data in the page), and only fall back to a browser for pages that need it. For enormous untargeted crawls by a verified business, Scale Residential brings residential cost to around a cent per GB.
Reviews
Review pages paginate, and many sites check that pages 2, 3 and 4 are requested by the same visitor. Use a short sticky session per product's reviews and fetch pages in order. See Rayobyte sticky sessions.
Stock and sellers
Stock often depends on location, and seller offers can vary by country or ZIP. Target the location you care about and record it with the data. See Rayobyte geo-targeting.
Residential tip: Before scaling, run a short pilot with Rayobyte on your real targets and record success rate, latency and GB per result. Our Rayobyte for web scraping explains which Rayobyte product fits which job.
In-store search rankings
To see where products rank in a store's own search, rotate per query and match the storefront's country. Search pages are often more protected than product pages.
Browser tips
- Block images, fonts and media unless you need them.
- Wait for the right element, not a fixed delay.
- Reuse a browser across several pages in one sticky session to reduce overhead.
- Keep fingerprints consistent with the proxy's location.
See Rayobyte with Puppeteer and Rayobyte with Playwright. Our Puppeteer proxies guide covers the full setup.
Costs
A catalog of 200,000 products refreshed weekly is about 800,000 product fetches a month. At 150 KB each that's around 120 GB. On Premium Residential, buying 250 GB at $1.50 covers two months for $375; if half the sites accept datacenter traffic, you can cut that substantially. See Rayobyte pricing explained.
Staying within the rules
Collect public product data, avoid personal data in reviews you don't need, and respect sites' load. Logged-in seller dashboards are a different matter: login pages are blocked on Rayobyte residential traffic. See compliance review and restricted sites.
Proxy tip: If your residential usage goes up and down, compare Geonode: its per-GB residential bandwidth doesn't expire, so you only pay for what you use. See Geonode for price monitoring.
Related guides
For marketplace specifics, see Rayobyte for Amazon data. For ongoing price tracking, see Rayobyte for price monitoring.
How it compares
See Rayobyte vs Geonode for e-commerce, Rayobyte vs Webshare for e-commerce and all three for e-commerce.
Not sure this is the right proxy type for the job? The best proxies in 2026 guide compares residential, ISP, datacenter and mobile options, and our cheap proxies guide covers budget picks that still work.
Budget tip: If you want a cheap proxy to try this with, Webshare has a free plan with 10 datacenter proxies and 1 GB of bandwidth a month, no card required. Paid lists let you pick proxy count, bandwidth and location. See Webshare for e-commerce scraping.
Key takeaways
- Small shops usually work on rotating datacenter; big marketplaces need residential or managed products.
- Use sitemaps, JSON-LD and HTTP clients to keep catalog crawls cheap.
- Use sticky sessions for paginated reviews and location targeting for stock and sellers.
- Scale Residential suits very large untargeted catalog crawls.
FAQ
Can Rayobyte scrape Shopify stores?
Yes. Most Shopify stores work with rotating datacenter proxies at modest request rates.
Do I need a browser for e-commerce scraping?
Not usually. Many product pages include structured data in the HTML. Use a browser only when content is rendered by JavaScript.
How do I scrape country-specific storefronts?
Target the storefront's country with Premium Residential and set the matching language and currency.
Sources and further reading
Independent references that back up the points above, from standards bodies, governments, universities and non-profits. They open in a new tab.
- Bandwidth (computing) (Wikipedia): What bandwidth means, the unit most residential proxies are priced by.
- Regulation (EU) 2016/679 (GDPR) (EUR-Lex, European Union): The full text of the EU's General Data Protection Regulation.
- Quarterly Retail E-Commerce Sales (US Census Bureau): Official US e-commerce sales data, the market price monitoring watches.
- Fingerprinting (glossary) (MDN Web Docs (Mozilla)): Mozilla's definition of browser fingerprinting.
- Cover Your Tracks (Electronic Frontier Foundation): The EFF's test that shows how unique your browser fingerprint is.
Editorial note: Proxy providers change their pools, features and prices often, so treat any figures here as typical ranges at the time of writing and confirm current terms with the provider before you buy.