Markdown, Parquet, and the Death of the Crawler: What “RAG-Ready” Data Actually Ships As

The phrase on every data buyer’s list in 2026 is “LLM-ready” — and almost nobody can define it, which is why procurement conversations stall on vibes. This piece is the missing spec sheet: the delivery formats, the freshness guarantees, and the cleaning levels that separate genuinely RAG-ready web data from a CSV wearing a marketing hat. Read it before your next vendor call.

The Question Behind the Question

When a retrieval team asks for “clean web data for embeddings” or “LLM ready web datasets,” the underlying requirements are concrete:

1. Chunking survives delivery. The data should arrive segmented in ways a retriever can consume — or at least structured so your chunker doesn’t fight it.
2. Noise is already gone. Boilerplate, nav bars, cookie banners, ad blocks — stripped before delivery, not by your pipeline after.
3. Freshness is a number, not a vibe. “Updated regularly” means nothing; “refreshed daily/hourly, timestamped per record” means a refresh contract.
4. Formats match the toolchain. Markdown for the LLM-context folks, JSON/NDJSON for the pipeline folks, Parquet for the warehouse folks — ideally all three from the same source of truth.

Vendors that answer all four in writing are selling RAG-ready data. Vendors that answer with the word “AI-powered” are selling a CSV.

The Format Field Guide

Markdown — the context-window format. The quiet winner of the LLM era: web pages converted to clean Markdown feed retrieval contexts without HTML stripping logic, and model token budgets stop paying for <div> soup. When a data catalog offers Markdown output as a first-class delivery format — Thordata’s AI-ready web data line does exactly this — it means the vendor did the extraction work your pipeline was doing after delivery.

JSON / NDJSON — the pipeline format. Structured records with stable schemas, one record per line for streaming ingestion. This is the shape your embedding jobs, dedup passes, and incremental loaders want. Watch for schema stability guarantees — a field that silently disappears in a refresh is a pipeline outage you’ll diagnose for days.

Parquet — the warehouse format. Columnar, compressed, query-able. When the same dataset ships in Parquet alongside JSON, the vendor is telling you they expect you to keep it in a lakehouse — which is correct, because RAG corpora get analyzed more than anyone predicts.

S3 delivery — the enterprise format. “Dataset delivery to S3” on a vendor’s spec page means the data lands in your bucket, on your schedule, inside your security perimeter. It’s the least glamorous line in the spec sheet and the one your platform team cares about most.

The Freshness Contract

Retrieval quality decays with corpus age, and the decay is measurable: product answers go stale in days, pricing answers in hours, factual answers in months. So the freshness question is really three questions:

• What’s the refresh cadence — per dataset, per record class?
• Is each record timestamped, so your retriever can prefer recency where it matters?
• Can refresh be scheduled — a standing job, not a re-purchase negotiation?

Thordata’s data-feed side answers this with subscription-style delivery on the collection infrastructure it already operates (Scraper API at ~$0.50–$1.00 per 1,000 results, raw residential from $0.65–$2.00/GB at volume), and the catalog side with dated records from roughly $0.25 per 1,000 records. The pattern to demand from any vendor: cadence, timestamps, scheduling — all three, in writing.

The Cleaning Ledger

“Clean” is doing a lot of work in “clean web data for embeddings,” so make the vendor itemize it:

The last row is the one that surfaces late and costs most: retrieval corpora that answer customer questions are a commercial use, and the license question procurement asks about training data applies to RAG corpora just as firmly.

A Reception Pipeline Worth Stealing

# Illustrative: receiving a scheduled delivery into a retrieval store
for batch in s3_feed.batches(since=last_ingest):
    for record in batch:                      # NDJSON, one record per line
        doc = {
            "id": record["record_id"],        # stable across refreshes
            "text_md": record["markdown"],    # context-ready body
            "url": record["url"],
            "published": record["timestamp"], # recency-aware retrieval
            "domain": record["domain"],
        }
        if passes_dedup(doc):                 # your boundary check
            index.upsert(embed(doc), metadata=doc)

Note what’s absent: no HTML parsing, no boilerplate stripping, no layout heuristics. That absence is the product — the vendor’s pipeline did it upstream, and your engineering went to the parts only you can do (retrieval quality, ranking, evaluation).

FAQ

Is RAG-ready data different from training-ready data?
Overlapping but not identical: training wants volume and diversity; retrieval wants freshness, provenance, and chunk-friendly structure. The good news is the same vendor machinery serves both — records are records — so the conversation is about which guarantees you need, not which product line.

How much of a retrieval corpus should be bought versus collected?
Buy the broad base (catalog datasets, subscription feeds), collect the narrow crown jewels (your domain’s sources, via scraper API or residential), and mirror the live web where freshness is survival. Most shipped RAG systems are all three layers, one schema.

Where do live agent workflows fit?
When the requirement is “the agent checks the live web at run time,” that’s collection, not corpus — the API layer, priced per request. The corpus is what the agent already knows; the live layer is what it verifies. Design both, from one account.

What about search results as a corpus?
Underrated and now cheap: SERP data as structured records — positions, features, ads — feeds competitive retrieval (“what outranks us for this query?”) and market-awareness agents alike. The SERP monitoring solution delivers it at roughly $0.70 per 1,000 responses, and continuous SERP data crawling keeps the corpus current on a schedule. For teams measuring retrieval quality against the live web, that feed doubles as an evaluation source — fresh questions with fresh answers attached.

What’s the fastest honest evaluation of a “RAG-ready” claim?
Ask for one sample file in each format they claim, load them into your stack, and run your current eval set against the resulting corpus. One day, zero procurement, and the format spec sheet above becomes your scoring rubric. Then ask the one question that ends most vendor meetings productively: “which of these four guarantees — cadence, timestamps, scheduling, license — will you put in writing?” The SERP monitoring answer to that question is a per-response price list; demand the equivalent transparency from every layer of your data stack.