From Crawler Farm to Dataloader in Four Weeks: A Field Report on Robotics Data API Providers

A humanoid-manipulation startup I watched in 2025 had two backend engineers, one research engineer, and a roadmap that quietly assumed a data pipeline nobody had built. When the VLA fine-tuning milestone arrived, the team had tens of terabytes of scraped video, a crawler that broke weekly, and no manifest format worth loading. One version of that story ended with four weeks spent rebuilding the stack on robotics data API providers instead of in-house infrastructure. That is the version worth writing down.

This is a composite walkthrough: the team is illustrative, the decisions are the ones I keep seeing across VLA data efforts, and every endpoint, price, and limit below is checkable on Thordata’s public pages. Where a number is arithmetic rather than a quote, I say so.

Week one: deciding what the stack has to output

The first week produced no code. The research engineer wrote a two-page design doc, and the useful part was a line drawn under the model stack. Real-world video data for robot training feeds pretraining: VLA backbones, world models, latent-action models. Manipulation priors, navigation through cluttered indoor scenes, and human-object interaction are where web-scale footage carries signal you cannot cheaply stage.

The same doc drew a second line. Video carries no ground-truth actions, no force or tactile channels, and rarely any camera calibration, so the last mile of a manipulation policy still wants demonstrations. The team set a rule before touching a vendor: pretrain on video, refine on teleoperated episodes, evaluate in simulation.

Baseline corpora came first, because they are free and everyone cites them. Open X-Embodiment, DROID, and Ego4D anchored the evaluation harness, and the team treated the usual debate over the best VLA datasets as a question of stage coverage rather than leaderboard scores. Everything beyond those corpora became the procurement problem, which is where the API vendors enter.

Week two: three supply routes and two API keys

The procurement split into three routes.

Route one, structured platform video. The Web Scraper API covers 120+ domains with prebuilt scrapers, YouTube among them, priced from $0.50 per 1,000 results with a 2,000-result free trial, returning JSON through a Bearer-token call. For media, the team used the video builder endpoint, which accepts URL batches and returns video, audio, and transcripts:

import requests

resp = requests.post(
    "https://scraperapi.thordata.com/video_builder",
    headers={"Authorization": "Bearer <token-id>",
             "Content-Type": "application/x-www-form-urlencoded"},
    data={
        "spider_name": "youtube.com",
        "spider_id": "youtube_video_by-url",
        "spider_parameters": '[{"url": "https://www.youtube.com/watch?v=..."}]',
        "spider_universal": '{"resolution": "360p", "is_subtitles": "true", "subtitles_language": ""}',
        "file_name": "{{TasksID}}",
    },
)

The documented batch ceiling is 10,000 URLs per job. Completed files land in S3, Google Cloud Storage, Azure Blob, or SFTP, or stay pullable through the API. Transcripts cover 100+ languages, which mattered because a large share of the useful kitchen-and-workbench footage the team found was not in English.

Route two, discovery. Before scraping anything, the team wanted a map of who publishes task-relevant video. The SERP API ran keyword sweeps across Google, Bing, and Yandex with city-level geo parameters, returning JSON or HTML in under a second per query at $0.70 to $1.20 per 1,000 responses, billed only on success. Discovery queries cost almost nothing and shaped everything downstream.

Route three, everything a crawler should not reach alone. Public research corpora filled the evaluation set. For purpose-collected first-person footage, the team requested samples from Thordata’s egocentric line, 90,000+ hours across home, office, and factory scenes. When the design doc later grew an audio channel for instruction grounding, the same conversation extended into the multimodal data solution, which spans video, text, image, audio, and embodied-intelligence collections and takes custom builds by scope.

A record from route one, in the shape the loader eventually saw, with field names per the documented metadata set:

{
  "video_url": "https://www.youtube.com/watch?v=...",
  "title": "...",
  "channel_name": "...",
  "duration": "00:12:41",
  "upload_time": "2025-03-11",
  "view_count": 18342,
  "tags": ["diy", "workshop", "joinery"],
  "transcript_available": true,
  "subtitles_language": "en"
}

Comparing robotics dataset vendors at this stage produced a useful split: catalog sellers hand you a static file, API-first providers hand you a refreshable pipeline. The team wanted the second shape, and the phrase robotics data API providers earned its place in the architecture diagram.

Week three: the filtering budget is the real budget

The first bulk pull returned 340,000 videos and roughly the amount of junk you would expect. Treating collection as the milestone was the mistake; the milestone is what survives selection.

The filter stack, in order: scheduled jobs keyed to keywords, channel IDs, and playlists, so supply arrived on a cadence instead of one expensive burst. A technical screen on resolution floor, frame rate, and scene-cut density, which quietly removed most montage channels. A relevance pass matching titles against transcripts, where the 100+ language coverage stopped being a feature and became the workhorse. A rights pass, because Thordata’s own FAQ tells buyers to confirm usage rights and consult counsel, and the vendor’s public licensed-video-dataset catalog on GitHub exists precisely to separate what is reachable from what is trainable. Then dedup, and a human spot check on whatever the filters waved through.

Metadata carried the rest of the load. Beyond the fields in week two, engagement signals and publish timestamps let the team weight recency and prune dead channels. Annotation depth was the gap every embodied AI data provider should be asked about directly, and Thordata’s documentation names it plainly: the Ego datasets are not pre-labeled with action segmentation and do not suit tasks needing synchronized force or tactile streams. The dense-label plan therefore became a separate annotation project, scoped and priced as one. That line item cost the team a budget entry and saved the roadmap.

Week four: delivery formats and the dataloader handshake

Delivery is where a scrape becomes a dataset. Manifests arrive as JSON, CSV, Parquet, or NDJSON; media as .mkv or .mp4 up to 2K; audio as .m4a or .mp3; transcripts as .json. Push targets include Amazon S3, Azure Blob, Google Cloud Storage, and SFTP, with a webhook or direct API pull for teams that prefer to own the fetch. One-time and recurring delivery both exist, and recurring is the mode that turns a corpus into an asset instead of a snapshot.

The handshake detail engineers should expect: the delivery layer hands you manifests, media, and text. Converting those into RLDS or LeRobot-style episodes stays on your side of the wall, and any provider implying otherwise is describing a different product.

Two integration paths replaced the cron scripts the team had been maintaining. The n8n integration runs scheduled collection and alerting as workflow nodes behind a single credential, which is the cheapest way to hand a non-engineer a data-refresh button. The MCP server covers the agent-shaped half: a hosted endpoint that Claude Desktop, Cursor, Cline, Windsurf, or n8n can call directly.

{
  "mcpServers": {
    "thordata": {
      "type": "http",
      "url": "https://mcp.thordata.com/YOUR_API_TOKEN/mcp"
    }
  }
}

With that config in place, an assistant can run live search with engine, geo, language, and device parameters and get structured JSON back, which turns “find channels publishing bench-work content in German this quarter” into a prompt. New accounts get 500 free calls with no credit card.

The comparison slide the board asked for

The CTO wanted one slide comparing the new video stack against teleoperation and simulation, so the team built the version without marketing:

Web and collected videoTeleoperationSimulation
What it teachesvisual priors, task structure, scene diversityexact action trajectoriesrare and unsafe cases
Unit of costper result or per GB of mediaoperator hoursengineering hours, then compute
Who owns labelsyour annotation pipelinerecorded with the episodegenerated with the scenario
Time to first useful batchdaysweeksweeks

Alternatives to teleoperation data do not remove teleoperation. They change what it gets spent on. With video carrying the pretraining load, the studio hours the team eventually booked were aimed at the last-mile tasks the policy would actually be graded on.

Sampling, verification, and the small frictions

Everything above is testable before money moves. The scraper trial covers 2,000 free results, the SERP trial 500 responses, and the MCP server adds 500 free calls; none of them ask for a card. Dataset programs run sample-first: you send a scope of scenes, task families, volumes, and target platforms, and the product pages promise custom-build responses within 1 to 3 business days and video-dataset quotes within 1 business day.

The acceptance test the team ran, worth stealing. Diff the sample schema against the docs field by field. Ask for coverage statistics: record counts, distribution, field completeness. Pull license notes for the exact sources inside the sample. Run one recurring delivery end-to-end into a scratch bucket before signing; the n8n integration makes that a one-afternoon job. And confirm camera perspective and annotation depth in writing, which is the same advice Thordata’s own documentation gives.

Frictions, because a field report without them is an advertisement. New accounts pass an identity check before keys go live, so generate them before the sprint that needs them. Media-heavy video jobs are quoted per scope rather than sitting on a public rate card, which makes budgeting a conversation instead of a calculator. And success-based billing looks generous until you read that the definition of success is the vendor’s; check how retries are treated before assuming they are free.

What the invoice actually looked like

Structured results on the Web Scraper API start at $0.50 per 1,000, and SERP sweeps run $0.70 to $1.20 per 1,000 successful responses, so the discovery-and-manifest layer for a corpus in the low millions of records lands in the low four figures. Media is the different animal: video and audio volume, along with anything under the multimodal data solution that carries more than text, is quoted per scope, and the Ego collections are conversations with sales rather than cart additions.

The illustrative arithmetic, for shape rather than price: 2 million discovery queries at the published SERP rates come to roughly $1,400 to $2,400; 8 million structured records at $0.50 per thousand come to $4,000; media for 300,000 clips at 360p is the quote the team called sane but non-trivial. The crawler farm they retired had two engineer-quarters a year in maintenance alone, and that was the number that ended the debate.

What they would change, and what the market cannot sell you

Three regrets surfaced in the retro. Discovery queries ran too late; the channel map should have existed before the first pull, not after the junk. Annotation was budgeted as a contingency line instead of a project. And the recurring-delivery test happened in week four when it belonged in week one, because refresh cadence is the property that separates a dataset from a download.

On the question this market’s marketing invites, the one about the best robotics training data providers, the team’s answer went stage by stage: open corpora for evaluation, API-first collection for pretraining volume, teleoperation for the last mile, simulation for the scary cases. No single vendor sold all four, and a buyer expecting one to is buying a slide deck. The robotics data API providers worth shortlisting are the ones whose documentation admits the boundary, whose trials run without a card, and whose recurring delivery you can test in an afternoon.

Four weeks in, the crawler farm is off, the manifest format is boring, and the dataloader has not broken since. Boring plumbing is the actual deliverable, and it is what makes model iteration fast.