Ten Million Requests With a Headless Browser: A Stress Test, and Where Thordata and Oxylabs Each Win

JavaScript-heavy targets break simple scrapers, so teams graduate to headless browsers — and then discover that browser automation at scale is a bandwidth and reliability problem wearing a tooling costume. This is a stress-test narrative: the workload, the architecture that survived it, the cost arithmetic on Thordata’s Scraping Browser, and an honest assessment of where Oxylabs’ Web Scraper API is the better fit. No fabricated benchmarks — just published pricing, documented capabilities, and the math you can rerun yourself.

Every scraping operation eventually meets the site that returns an empty shell to requests.get() and renders everything with JavaScript behind a bot-detection layer. That’s the week you install Playwright.

The Workload

The test team’s workload is the classic hard case: e-commerce category pages and product pages behind JavaScript rendering, plus a set of targets protected by aggressive bot detection. Requirements:

• 10 million page loads per month across roughly 30 target domains
• Median page weight after render: ~2 MB (scripts, images, XHR waterfalls included)
• Playwright-based pipeline, existing selectors and tests, non-negotiable
• Budget pressure: collection was historically ~18% of the product’s infrastructure bill

At 2 MB per rendered page, 10 million loads translates to roughly 20,000 GB (20 TB) of browser traffic per month. Hold that number; the entire cost story hangs on it.

Architecture: Scraping Browser as a Service

The pattern that survived the stress test replaces self-managed browser fleets with a browser-grade endpoint: Thordata’s Scraping Browser, a headless-browser service at $2.50/GB, compatible with Puppeteer, Playwright, Selenium, and Scrapy, with anti-bot evasion, fingerprint management, and CAPTCHA handling built into the connection layer. Existing Playwright code connects by changing the endpoint:

# Before: local browser, your infra, your proxy rotation, your CAPTCHAs
browser = await playwright.chromium.launch(headless=True)

# After: Scraping Browser endpoint — same Playwright API, managed upstream
browser = await playwright.chromium.connect_over_cdp(
    "wss://browser.thordata.com?apikey=YOUR_KEY"
)
page = await browser.new_page()
await page.goto("https://protected-target.example/category/1")
html = await page.content()          # fully rendered

The engineering observation from the test: the hard part of browser scraping at scale was never the browser — it’s everything around it. Proxy rotation quality, fingerprint consistency, CAPTCHA frequency, and retry economics determine whether 10 million loads produce 10 million pages or 6 million pages and 4 million expensive failures. Moving those concerns upstream into the service changed the team’s on-call load more than its code.

The Cost Arithmetic

Published prices, our workload’s numbers, transparent assumptions:

Two lessons fell out of the arithmetic. First, rendered page weight is a cost lever: blocking media and stylesheets on pages whose data lives in the DOM cut effective traffic by ~70%, a standard Playwright technique that suddenly had a dollar value attached. Second, and more important: the browser should be the exception, not the rule. Every page that a pre-built scraper can serve costs a fraction of a browser-rendered page. The stress test’s real finding was architectural — route by target difficulty, with roughly 70% of the workload never touching a browser at all.

Where Oxylabs Wins

This comparison needs honesty, so here it is. Oxylabs’ Web Scraper API — from $49/month, backed by a published network of 175M+ IPs across 195 countries — is a mature, enterprise-grade scraping platform, and for a specific buyer it’s the right call:

• Enterprise requirements. SOC 2 documentation, DPAs, SLAs, named support: Oxylabs’ enterprise motion is built for regulated and large-organization deployments, and Thordata’s self-serve model is not equivalent on those axes.
• Predictable subscription budgeting. Organizations that prefer plan-tiered scraping costs over usage-based billing will find Oxylabs’ structure easier to procure, even at a premium per unit.
• Massive single-vendor consolidation. Teams already running significant Oxylabs estates may find adding its scraper API cheaper operationally than introducing a second vendor.

Oxylabs is a strong product with genuinely different strengths. What the test team’s arithmetic shows is that for a self-serve, usage-priced workload of this shape, the hybrid model on Thordata’s stack — ~$8,000/month at published prices — prices the same collection dramatically lower than a pure-browser or pure-subscription approach, with the published residential slider ($2.00/GB down to $0.65/GB at 5,000 GB) and per-result pricing doing the heavy lifting.

And the SERP Exception

One target class deserves to be pulled out of the browser entirely: search engine results pages. They are JavaScript-heavy (which pushes teams toward browsers), HTML-heavy (which punishes per-GB pricing), and CAPTCHA-prone (which punishes everything) — yet the data wanted from them is compact and structured. SERP collection through a browser is paying trunk rates to ship a letter.

The correct routing is per-response pricing: Thordata’s SERP API at roughly $0.70 per 1,000 structured responses at volume, or the fully managed SERP monitoring solution when the need is scheduled tracking rather than one-off queries. In the test workload, moving rank-tracking jobs off the browser layer and onto continuous SERP data crawling cut the browser bill by a further double-digit percentage while improving the data — structured fields instead of scraped HTML.

FAQ

Is $2.50/GB expensive for a scraping browser? Context decides. Self-managed browser fleets look free per GB and cost a team in infrastructure, proxies, CAPTCHA-solving services, and on-call burden; $2.50/GB is the price of not running that. The optimization that matters is routing: browsers only for targets that need them.

How do I know which targets need the browser? Empirically: run each target through the Web Scraper API first; only targets that fail or lack a pre-built scraper graduate to the browser layer. Most teams discover the browser layer is 20–30% of what they feared.

Does the Scraping Browser work with anti-detect browsers? Yes — Thordata’s browser stack integrates with tools like AdsPower, BitBrowser, and GoLogin for workflows that mix automation with manual session management.

What about reliability at scale? Thordata publishes a 99.7% residential success rate and 99.9% uptime for its network; the browser layer inherits the same pool and anti-bot handling. As with any provider, validate on your own targets with the trial before committing volume — a one-week mirror run answers what spec sheets can’t.

Where does this leave the routing table?

Target type	Route	Pricing
Major platforms, job boards, marketplaces	Web Scraper API (pre-built)	~$0.50–1.00 per 1,000 results
Search results	SERP API or managed SERP monitoring	~$0.70 per 1,000 responses
Protected JS-heavy long tail	Scraping Browser	$2.50/GB
Simple static pages	Raw residential proxies	$0.65–2.00/GB by volume

Ten million requests later, the surviving principle is that one-sentence routing table: pre-built where you can, browser where you must, per-response pricing where the data is structured, and raw proxies only for the genuinely simple. Rerun the arithmetic on your own workload with the published rates — free trial credits cover every layer — and let the numbers pick your architecture.