Robotics Training Data Sample: Field Documentation for VLA and Embodied AI Teams
What this sample covers
This field documentation explains the structure of a real-world robotics training data sample for teams building vision-language-action (VLA) models, robot foundation models, imitation-learning systems, and embodied AI evaluation workflows.
The sample is designed around a practical question: can your team quickly understand what is in each episode, assess whether it fits the target task, and integrate it into a training pipeline without rebuilding the data schema from scratch?
Thordata provides multimodal data collection and AI training datasets, including robotics and embodied AI data such as video, sensor data, 3D data, environment data, and hand-object interaction data.
Who should use this data
This data structure is most useful for teams working on:
| Team or model type | Typical use case |
|---|---|
| VLA model teams | Train models to connect visual observations, language instructions, and robot actions |
| Robot foundation model teams | Build generalizable manipulation, navigation, and interaction capabilities |
| Imitation learning teams | Learn policies from human demonstrations or teleoperation trajectories |
| World model teams | Learn physical dynamics, object states, scene context, and action consequences |
| Evaluation teams | Build task-specific validation sets with consistent metadata and quality labels |
| Data platform teams | Ingest, filter, version, and query robotics datasets through an API or data warehouse |
What models it fits
The sample schema is intended for models that require aligned multimodal observations and structured context, including:
- Vision-language-action models (VLAs) that need visual observations, task instructions, and action labels.
- Imitation learning models that require demonstration episodes with observations, actions, and success or failure outcomes.
- Multimodal video models that learn from first-person video, task demonstrations, action sequences, and scene context.
- World models that benefit from temporal sequences, object states, scene descriptions, and interaction events.
- Retrieval and evaluation systems that need to search episodes by task, scene, embodiment, modality, or quality score.
Thordata’s robotics and embodied AI data offering includes real-world interaction and sensor data, while its video datasets can include first-person video, task demonstrations, action sequences, and scene context.thordata+1
What tasks it supports
A well-structured robotics dataset should support more than one downstream task. This field structure can support:
- Manipulation: pick, place, grasp, insert, sort, stack, open, close, and tool-use tasks.
- Navigation: indoor or outdoor movement, obstacle avoidance, route following, and goal-directed navigation.
- Human-robot interaction: hand-over, instruction following, collaborative tasks, and social interaction.
- Object interaction: contact-rich manipulation, hand-object interaction, tool use, and multi-step task execution.
- Scene understanding: object recognition, spatial relationships, affordance detection, and environment context.
- Policy evaluation: measuring task completion, robustness, recovery behavior, and generalization across scenes or objects.
Data sources
The sample can be built from one or more of the following sources:
- Real-world video: first-person, third-person, wrist-mounted, or multi-camera recordings of tasks and environments.
- Human demonstrations: people performing tasks naturally, with optional task instructions and annotations.
- Teleoperation sessions: operators controlling a robot while observation, action, and state streams are captured.
- Robot logs: joint states, end-effector poses, gripper states, forces, and other robot-side signals.
- Sensor data: camera, depth, IMU, tactile, force-torque, audio, and other modality-specific inputs.
- Environment data: scene descriptions, object inventories, spatial layout, lighting conditions, and task setup details.
- Public web video: consent-approved or licensed video used for pretraining, retrieval, scene understanding, or evaluation support.
Thordata’s video datasets are described as containing video, audio, metadata, subtitles, and comments, with delivery formats including MP4, JSON, and other formats.
Illustrative episode structure
The following JSON is an illustrative example, not an actual Thordata data sample. Available fields, annotations, usage rights, and delivery options must be confirmed for each dataset.
{
"episode_id": "example_episode_001",
"source_type": "human_demonstration",
"task": {
"name": "pick_and_place",
"instruction": "Pick up the cup and place it on the tray."
},
"environment": {
"scene_type": "kitchen",
"objects": ["cup", "tray"]
},
"video": {
"file_name": "example_episode_001.mp4",
"camera_view": "first_person",
"fps": 30,
"duration_seconds": 24.6
},
"annotations": {
"task_outcome": "success"
},
"robot_action_data_included": false
}
Field definitions
Episode-level fields
| Field | Type | Description | Why it matters |
|---|---|---|---|
episode_id | String | A unique identifier for one task attempt or demonstration. | Prevents duplicate records and enables stable references across systems. |
dataset_version | String | The version of the dataset schema and contents. | Supports reproducibility, audits, and controlled updates. |
collection_date | Date | The date the data was collected. | Helps teams control data recency and detect distribution shift. |
task | Object | Task identity, instruction, category, and success criteria. | Makes the data searchable and aligned with training objectives. |
embodiment | Object | Whether the source is a human, teleoperator, or robot, plus hardware details. | Helps teams match data to the target robot or transfer-learning setup. |
environment | Object | Scene, location, lighting, objects, and context. | Supports filtering by environment and evaluating generalization. |
modalities | Object | References to video, sensor, audio, and text streams. | Enables multimodal training and modality-specific preprocessing. |
annotations | Object | Action segments, object tracks, success labels, and quality scores. | Supports supervised learning, filtering, and evaluation. |
license | Object | Usage rights, consent, redaction, and license ID. | Reduces legal and compliance uncertainty before data enters training. |
delivery | Object | Manifest, storage format, and API endpoint. | Makes ingestion and pipeline integration easier. |
Task fields
| Field | Example | Description |
|---|---|---|
task_id | pick_and_place_cup | A stable identifier for the task type. |
task_name | Pick up a cup and place it on a tray | A human-readable task description. |
task_category | manipulation | A high-level grouping such as manipulation, navigation, or interaction. |
instruction | Pick up the red cup and place it on the tray. | The language instruction associated with the episode. |
success_criteria | The cup is stably placed on the tray without spilling. | The condition used to label success or failure. |
Clear task metadata matters because VLA and imitation-learning teams need to connect language instructions, visual observations, and action outcomes. It also makes it possible to build balanced training and evaluation sets across tasks, scenes, objects, and embodiments.
Embodiment fields
| Field | Example | Description |
|---|---|---|
embodiment_type | human_demonstration | Indicates whether the episode comes from a human, teleoperation, or real robot. |
robot_platform | null or custom_arm_v2 | The robot hardware platform, when applicable. |
end_effector | parallel_gripper or five_finger_hand | The gripper, hand, or tool used for interaction. |
camera_rig | ["ego_camera", "third_person_camera"] | The camera views available in the episode. |
This distinction is important: human demonstrations, teleoperation sessions, and real-robot logs are not interchangeable. Human demonstrations can provide scalable task coverage, while teleoperation and real-robot data can provide embodiment-aligned actions, joint states, end-effector poses, and physical interaction signals.
Environment fields
| Field | Example | Description |
|---|---|---|
scene_id | kitchen_counter_004 | A stable identifier for the recording location or scene template. |
location_type | indoor | The broad environment category. |
lighting | indoor_artificial | Lighting conditions that may affect visual robustness. |
objects | ["red_cup", "tray", "table"] | Key objects present in the episode. |
background_description | Free text | A concise description of the surrounding scene. |
Environment metadata allows teams to filter for specific deployment conditions, such as cluttered kitchens, warehouse shelves, laboratory benches, or outdoor sidewalks. It also helps identify whether a model is overfitting to one scene or object arrangement.
Video fields
| Field | Example | Description |
|---|---|---|
stream_id | ego_rgb | A stable identifier for the video stream. |
file_path | episodes/.../ego_rgb.mp4 | The location of the media file. |
resolution | 1920x1080 | Video resolution. |
fps | 30 | Frames per second. |
duration_seconds | 24.6 | Episode duration. |
camera_view | first_person | Egocentric, third-person, wrist, or other viewpoint. |
First-person video, task demonstrations, action sequences, and scene context are particularly relevant for robotics and embodied AI use cases.thordata
Annotation fields
| Field | Example | Description |
|---|---|---|
action_segments | JSON file reference | Time-stamped action phases, such as approach, grasp, transport, and release. |
object_tracks | JSON file reference | Object identities, bounding boxes, masks, poses, or tracked states over time. |
success_label | true | Whether the episode met the task success criteria. |
failure_reason | null or object_slipped | A structured reason for failure, when applicable. |
quality_score | 0.92 | A normalized quality score based on defined acceptance criteria. |
Quality scores should be based on documented criteria, such as camera visibility, synchronization, task completion, annotation accuracy, motion blur, occlusion, and personally identifiable information. A score without a defined rubric is difficult to trust.
How data is filtered
Before delivery, episodes can be filtered using fields such as:
- Task category and task ID.
- Embodiment type, robot platform, and end effector.
- Scene, location, lighting, and object list.
- Video resolution, frame rate, duration, and camera view.
- Sensor modality and availability.
- Success label, failure reason, and quality score.
- License status, consent status, and redaction status.
- Collection date and dataset version.
For example, a VLA team may request only successful manipulation episodes with first-person RGB video, English instructions, a quality score above 0.85, and consent-approved usage rights.
How data is delivered
A production-ready robotics dataset should be delivered with:
- Media files: MP4, MKV, or other agreed video formats.
- Structured metadata: JSON, CSV, or Parquet files.
- Annotation files: JSON, COCO-style files, or task-specific formats.
- Manifest files: One record per episode, linking every media and annotation file.
- Cloud delivery: Amazon S3, Azure Blob Storage, Google Cloud Storage, or SFTP.
- API access: For querying, filtering, previewing, and pulling selected episodes.
- Version control: Dataset version, schema version, and change logs.
Thordata describes delivery options that include Amazon S3, Azure Blob, Google Cloud Storage, SFTP, webhook, and direct API access, with JSON, CSV, or Parquet formats for structured data.
How to connect it to training
A typical integration workflow is:
- Ingest the manifest. Load episode-level metadata into your data catalog or training database.
- Validate synchronization. Check that video, sensor, action, and annotation timestamps align.
- Filter the corpus. Select episodes by task, embodiment, scene, quality, license, and modality.
- Convert to training format. Transform episodes into the format required by your training framework.
- Split train and evaluation sets. Keep scenes, objects, or tasks held out to measure generalization.
- Track data lineage. Record dataset version, episode IDs, filters, and preprocessing steps for every training run.
- Request targeted top-ups. Identify failure cases and request additional data for underrepresented tasks, scenes, or conditions.
How it differs from simulation and teleoperation data
| Data type | Strengths | Limitations | Best use |
|---|---|---|---|
| Real-world video data | Captures natural visual variation, real objects, lighting, clutter, and human behavior. | May lack perfectly aligned robot actions or precise physical states. | Pretraining, scene understanding, task diversity, evaluation, and VLA data enrichment. |
| Simulation data | Scalable, controllable, and often includes precise states and actions. | Can differ from real-world physics, visuals, and object behavior. | Pretraining, synthetic augmentation, safety testing, and controlled evaluation. |
| Teleoperation data | Provides aligned robot observations, actions, and often robot-state signals. | Can be costly and slower to collect at scale. | Imitation learning, policy fine-tuning, and embodiment-specific training. |
| Human demonstration data | Provides natural task strategies and broad task coverage. | May require adaptation to a different robot embodiment. | VLA pretraining, demonstration learning, and task understanding. |
The strongest robotics data strategy usually combines these sources rather than treating one as a replacement for the others. Real-world video can broaden visual and task coverage, while teleoperation and robot logs provide the action and state alignment needed for policy learning.
Sample request checklist
When requesting a robotics data sample, ask the provider to include:
- A sample of 10 to 50 episodes across your priority tasks.
- The complete field dictionary and data schema.
- Example video, sensor, annotation, and manifest files.
- A description of the collection method and data source.
- Quality-control criteria and quality-score definitions.
- Consent, licensing, redaction, and permitted-use terms.
- Delivery format, storage destination, API access, and refresh cadence.
- A pilot plan for custom collection, if existing coverage does not match your task.
Request a sample
If your team is evaluating robotics training data providers, VLA datasets, real-world video data for robot training, or alternatives to teleoperation data, request a sample matched to your target task.
A useful first request is:
“Please provide a 20-episode sample for [task type], including first-person video, task instructions, action or interaction annotations, environment metadata, quality scores, license terms, and a manifest in JSON or Parquet.”
Thordata supports multimodal data collection and AI training datasets across video, text, image, audio, and embodied intelligence, including robotics, autonomous systems, and real-world interaction data.