ChatGPT Web Scraping: AI Code & Parsing Guide

When ChatGPT launched, every developer tried asking it: “Write me a Python script to scrape Amazon.”

The result? Usually a basic script using `requests` and `BeautifulSoup` that gets blocked instantly by Amazon’s anti-bot protection. While useful for beginners, this approach fails in production. The problem isn’t the AI; it’s the context you provide.

In 2026, we don’t just ask ChatGPT to write code; we integrate Large Language Models (LLMs) into the scraping pipeline. In this guide, I will show you the two distinct ways to leverage AI: generating robust, proxy-aware code, and using the OpenAI API to parse unstructured data that traditional selectors cannot handle.

Part 1: The Code Generator (Writing Better Prompts)

If you ask generic questions, you get generic code. To get a script that actually works in the real world, you need to prompt-engineer for “Resilience.”

The “Bad” Prompt

Result: A script with no headers, no error handling, and no proxies. It will fail after 5 requests.

The “Pro” Prompt

Here is the kind of high-quality code ChatGPT generates with that prompt:

from playwright.sync_api import sync_playwright
import time
import random

# Thordata Residential Proxy (Host:Port)
PROXY_SERVER = "http://gate.thordata.com:12345"
PROXY_AUTH = {"username": "user", "password": "pass"}

def run():
    with sync_playwright() as p:
        # Launch with Proxy Configuration
        browser = p.chromium.launch(
            proxy={"server": PROXY_SERVER, "username": "user", "password": "pass"}
        )
        page = browser.new_page()
        
        try:
            page.goto("https://example.com", timeout=60000)
            # Random delay to mimic human behavior
            time.sleep(random.uniform(2, 5))
            print(page.title())
        except Exception as e:
            print(f"Error: {e}")
        finally:
            browser.close()

if __name__ == "__main__":
    run()

Part 2: The Data Parser (AI as a Tool)

This is the exciting part. Sometimes, data is unstructured. Maybe it’s a messy address block like “123 Main St., near the old oak tree, Apt 4B, NY”. Regex will fail here. XPath will fail here.

But LLMs excel at this. We can download the HTML, strip the tags, and ask the OpenAI API to format it into JSON.

import openai
from bs4 import BeautifulSoup

client = openai.OpenAI(api_key="YOUR_OPENAI_KEY")

# Imagine we scraped this messy string from a website
raw_html_content = "<div>Contact: John Doe (Manager) - Phone: 555-0199 or email johnd@corp.com</div>"

# 1. Clean it first (Reduce tokens)
soup = BeautifulSoup(raw_html_content, "html.parser")
clean_text = soup.get_text()

# 2. Ask LLM to format it
response = client.chat.completions.create(
    model="gpt-4-turbo",
    messages=[
        {"role": "system", "content": "Extract name, role, phone, and email into JSON."},
        {"role": "user", "content": clean_text}
    ],
    response_format={"type": "json_object"}
)

print(response.choices[0].message.content)
# Output: {"name": "John Doe", "role": "Manager", "phone": "555-0199", "email": "johnd@corp.com"}

3. The Future: Agentic Scraping

We are moving towards “Agentic” workflows. Tools like LangChain allow you to create an AI agent that can browse the web, click buttons, and decide what to scrape next.

However, agents are currently slow and expensive per page. For high-volume data extraction (e.g., scraping millions of e-commerce products), the traditional approach of “Optimized Script + Rotating Proxies” remains 100x cheaper and faster.

AI is best used to write the script or clean the specific data points that break your regex rules.

Conclusion

ChatGPT is a force multiplier for data engineers. Use it to generate boilerplate code that includes Thordata Residential Proxies, and use its API to parse the unparsable. But remember: code generated by AI is only as good as the prompt you give it.