Web Scraper API Guide: Python, Node.js & cURL

There are two ways to scrape the web. The hard way involves managing thousands of proxies, rotating User-Agents, solving CAPTCHAs, and patching your Puppeteer script every time Chrome updates.

The smart way is to offload that entire stack to a Web Scraper API. You send a URL; you get back clean HTML or JSON. No blocks. No headaches.

In this guide, I will show you how to integrate Thordata’s API into your workflow using standard cURL requests, and more importantly, using our robust Python and Node.js SDKs.

1. The Concept: Your Proxy on Steroids

A standard proxy just changes your IP address. A Scraper API acts as a complete browser in the cloud.

• Request: You call api.thordata.com with your target URL.
• Processing: We spin up a headless browser, route traffic through a residential IP, solve any challenges (Cloudflare Turnstile, etc.), and render the JavaScript.
• Response: We return the final HTML source code to you.

2. Quick Start: cURL (The Universal Language)

The fastest way to test an API is via the terminal. This verifies your API token and the connectivity.

curl "https://api.thordata.com/v1/scraper?token=YOUR_API_TOKEN&url=https://www.amazon.com&render_js=true"

3. The Modern Way: Thordata Python SDK

Using raw `requests` is fine, but the official SDK handles error parsing, type hints, and retries for you. This is the preferred method for enterprise Python projects.

pip install thordata-sdk

from thordata import ThordataClient

# Initialize with your token
client = ThordataClient(scraper_token="YOUR_TOKEN")

print("🚀 Scraping dynamic content...")

# universal_scrape handles JS rendering automatically
html = client.universal_scrape(
    url="https://www.nike.com/launch",
    js_render=True,
    wait_for=".product-card",  # Wait for specific element
    country="us"
)

print(f"Success! Retrieved {len(html)} bytes.")

# You can now parse 'html' with BeautifulSoup

4. The Node.js SDK (TypeScript Ready)

For JavaScript/TypeScript developers, our NPM package provides full type support.

npm install thordata-sdk

import { ThordataClient } from "thordata-sdk";

const client = new ThordataClient({
  scraperToken: "YOUR_TOKEN"
});

async function run() {
  const html = await client.universalScrape({
    url: "https://www.walmart.com",
    jsRender: true,
    outputFormat: "html"
  });

  console.log("Data fetched:", html.substring(0, 200));
}

run();

5. The Economic Argument: Build vs. Buy

Why pay for an API when you can write a scraper for free? Because “free” scrapers are expensive to maintain.

Conclusion

The Web Scraper API allows you to focus on what matters: the data. By abstracting away the complexity of proxies and headless browsers, you turn web scraping into a simple function call. Start with the SDKs today to build resilient pipelines in minutes, not days.