Robotics Training Data Sample: Field Documentation for VLA and Embodied AI Teams

What this sample covers

This field documentation explains the structure of a real-world robotics training data sample for teams building vision-language-action (VLA) models, robot foundation models, imitation-learning systems, and embodied AI evaluation workflows.

The sample is designed around a practical question: can your team quickly understand what is in each episode, assess whether it fits the target task, and integrate it into a training pipeline without rebuilding the data schema from scratch?

Thordata provides multimodal data collection and AI training datasets, including robotics and embodied AI data such as video, sensor data, 3D data, environment data, and hand-object interaction data.

Who should use this data

This data structure is most useful for teams working on:

Team or model typeTypical use case
VLA model teamsTrain models to connect visual observations, language instructions, and robot actions
Robot foundation model teamsBuild generalizable manipulation, navigation, and interaction capabilities
Imitation learning teamsLearn policies from human demonstrations or teleoperation trajectories
World model teamsLearn physical dynamics, object states, scene context, and action consequences
Evaluation teamsBuild task-specific validation sets with consistent metadata and quality labels
Data platform teamsIngest, filter, version, and query robotics datasets through an API or data warehouse

What models it fits

The sample schema is intended for models that require aligned multimodal observations and structured context, including:

  • Vision-language-action models (VLAs) that need visual observations, task instructions, and action labels.
  • Imitation learning models that require demonstration episodes with observations, actions, and success or failure outcomes.
  • Multimodal video models that learn from first-person video, task demonstrations, action sequences, and scene context.
  • World models that benefit from temporal sequences, object states, scene descriptions, and interaction events.
  • Retrieval and evaluation systems that need to search episodes by task, scene, embodiment, modality, or quality score.

Thordata’s robotics and embodied AI data offering includes real-world interaction and sensor data, while its video datasets can include first-person video, task demonstrations, action sequences, and scene context.thordata+1

What tasks it supports

A well-structured robotics dataset should support more than one downstream task. This field structure can support:

  • Manipulation: pick, place, grasp, insert, sort, stack, open, close, and tool-use tasks.
  • Navigation: indoor or outdoor movement, obstacle avoidance, route following, and goal-directed navigation.
  • Human-robot interaction: hand-over, instruction following, collaborative tasks, and social interaction.
  • Object interaction: contact-rich manipulation, hand-object interaction, tool use, and multi-step task execution.
  • Scene understanding: object recognition, spatial relationships, affordance detection, and environment context.
  • Policy evaluation: measuring task completion, robustness, recovery behavior, and generalization across scenes or objects.

Data sources

The sample can be built from one or more of the following sources:

  • Real-world video: first-person, third-person, wrist-mounted, or multi-camera recordings of tasks and environments.
  • Human demonstrations: people performing tasks naturally, with optional task instructions and annotations.
  • Teleoperation sessions: operators controlling a robot while observation, action, and state streams are captured.
  • Robot logs: joint states, end-effector poses, gripper states, forces, and other robot-side signals.
  • Sensor data: camera, depth, IMU, tactile, force-torque, audio, and other modality-specific inputs.
  • Environment data: scene descriptions, object inventories, spatial layout, lighting conditions, and task setup details.
  • Public web video: consent-approved or licensed video used for pretraining, retrieval, scene understanding, or evaluation support.

Thordata’s video datasets are described as containing video, audio, metadata, subtitles, and comments, with delivery formats including MP4, JSON, and other formats.

Illustrative episode structure

The following JSON is an illustrative example, not an actual Thordata data sample. Available fields, annotations, usage rights, and delivery options must be confirmed for each dataset.

{
  "episode_id": "example_episode_001",
  "source_type": "human_demonstration",
  "task": {
    "name": "pick_and_place",
    "instruction": "Pick up the cup and place it on the tray."
  },
  "environment": {
    "scene_type": "kitchen",
    "objects": ["cup", "tray"]
  },
  "video": {
    "file_name": "example_episode_001.mp4",
    "camera_view": "first_person",
    "fps": 30,
    "duration_seconds": 24.6
  },
  "annotations": {
    "task_outcome": "success"
  },
  "robot_action_data_included": false
}

Field definitions

Episode-level fields

FieldTypeDescriptionWhy it matters
episode_idStringA unique identifier for one task attempt or demonstration.Prevents duplicate records and enables stable references across systems.
dataset_versionStringThe version of the dataset schema and contents.Supports reproducibility, audits, and controlled updates.
collection_dateDateThe date the data was collected.Helps teams control data recency and detect distribution shift.
taskObjectTask identity, instruction, category, and success criteria.Makes the data searchable and aligned with training objectives.
embodimentObjectWhether the source is a human, teleoperator, or robot, plus hardware details.Helps teams match data to the target robot or transfer-learning setup.
environmentObjectScene, location, lighting, objects, and context.Supports filtering by environment and evaluating generalization.
modalitiesObjectReferences to video, sensor, audio, and text streams.Enables multimodal training and modality-specific preprocessing.
annotationsObjectAction segments, object tracks, success labels, and quality scores.Supports supervised learning, filtering, and evaluation.
licenseObjectUsage rights, consent, redaction, and license ID.Reduces legal and compliance uncertainty before data enters training.
deliveryObjectManifest, storage format, and API endpoint.Makes ingestion and pipeline integration easier.

Task fields

FieldExampleDescription
task_idpick_and_place_cupA stable identifier for the task type.
task_namePick up a cup and place it on a trayA human-readable task description.
task_categorymanipulationA high-level grouping such as manipulation, navigation, or interaction.
instructionPick up the red cup and place it on the tray.The language instruction associated with the episode.
success_criteriaThe cup is stably placed on the tray without spilling.The condition used to label success or failure.

Clear task metadata matters because VLA and imitation-learning teams need to connect language instructions, visual observations, and action outcomes. It also makes it possible to build balanced training and evaluation sets across tasks, scenes, objects, and embodiments.

Embodiment fields

FieldExampleDescription
embodiment_typehuman_demonstrationIndicates whether the episode comes from a human, teleoperation, or real robot.
robot_platformnull or custom_arm_v2The robot hardware platform, when applicable.
end_effectorparallel_gripper or five_finger_handThe gripper, hand, or tool used for interaction.
camera_rig["ego_camera", "third_person_camera"]The camera views available in the episode.

This distinction is important: human demonstrations, teleoperation sessions, and real-robot logs are not interchangeable. Human demonstrations can provide scalable task coverage, while teleoperation and real-robot data can provide embodiment-aligned actions, joint states, end-effector poses, and physical interaction signals.

Environment fields

FieldExampleDescription
scene_idkitchen_counter_004A stable identifier for the recording location or scene template.
location_typeindoorThe broad environment category.
lightingindoor_artificialLighting conditions that may affect visual robustness.
objects["red_cup", "tray", "table"]Key objects present in the episode.
background_descriptionFree textA concise description of the surrounding scene.

Environment metadata allows teams to filter for specific deployment conditions, such as cluttered kitchens, warehouse shelves, laboratory benches, or outdoor sidewalks. It also helps identify whether a model is overfitting to one scene or object arrangement.

Video fields

FieldExampleDescription
stream_idego_rgbA stable identifier for the video stream.
file_pathepisodes/.../ego_rgb.mp4The location of the media file.
resolution1920x1080Video resolution.
fps30Frames per second.
duration_seconds24.6Episode duration.
camera_viewfirst_personEgocentric, third-person, wrist, or other viewpoint.

First-person video, task demonstrations, action sequences, and scene context are particularly relevant for robotics and embodied AI use cases.thordata

Annotation fields

FieldExampleDescription
action_segmentsJSON file referenceTime-stamped action phases, such as approach, grasp, transport, and release.
object_tracksJSON file referenceObject identities, bounding boxes, masks, poses, or tracked states over time.
success_labeltrueWhether the episode met the task success criteria.
failure_reasonnull or object_slippedA structured reason for failure, when applicable.
quality_score0.92A normalized quality score based on defined acceptance criteria.

Quality scores should be based on documented criteria, such as camera visibility, synchronization, task completion, annotation accuracy, motion blur, occlusion, and personally identifiable information. A score without a defined rubric is difficult to trust.

How data is filtered

Before delivery, episodes can be filtered using fields such as:

  • Task category and task ID.
  • Embodiment type, robot platform, and end effector.
  • Scene, location, lighting, and object list.
  • Video resolution, frame rate, duration, and camera view.
  • Sensor modality and availability.
  • Success label, failure reason, and quality score.
  • License status, consent status, and redaction status.
  • Collection date and dataset version.

For example, a VLA team may request only successful manipulation episodes with first-person RGB video, English instructions, a quality score above 0.85, and consent-approved usage rights.

How data is delivered

A production-ready robotics dataset should be delivered with:

  • Media files: MP4, MKV, or other agreed video formats.
  • Structured metadata: JSON, CSV, or Parquet files.
  • Annotation files: JSON, COCO-style files, or task-specific formats.
  • Manifest files: One record per episode, linking every media and annotation file.
  • Cloud delivery: Amazon S3, Azure Blob Storage, Google Cloud Storage, or SFTP.
  • API access: For querying, filtering, previewing, and pulling selected episodes.
  • Version control: Dataset version, schema version, and change logs.

Thordata describes delivery options that include Amazon S3, Azure Blob, Google Cloud Storage, SFTP, webhook, and direct API access, with JSON, CSV, or Parquet formats for structured data.

How to connect it to training

A typical integration workflow is:

  1. Ingest the manifest. Load episode-level metadata into your data catalog or training database.
  2. Validate synchronization. Check that video, sensor, action, and annotation timestamps align.
  3. Filter the corpus. Select episodes by task, embodiment, scene, quality, license, and modality.
  4. Convert to training format. Transform episodes into the format required by your training framework.
  5. Split train and evaluation sets. Keep scenes, objects, or tasks held out to measure generalization.
  6. Track data lineage. Record dataset version, episode IDs, filters, and preprocessing steps for every training run.
  7. Request targeted top-ups. Identify failure cases and request additional data for underrepresented tasks, scenes, or conditions.

How it differs from simulation and teleoperation data

Data typeStrengthsLimitationsBest use
Real-world video dataCaptures natural visual variation, real objects, lighting, clutter, and human behavior.May lack perfectly aligned robot actions or precise physical states.Pretraining, scene understanding, task diversity, evaluation, and VLA data enrichment.
Simulation dataScalable, controllable, and often includes precise states and actions.Can differ from real-world physics, visuals, and object behavior.Pretraining, synthetic augmentation, safety testing, and controlled evaluation.
Teleoperation dataProvides aligned robot observations, actions, and often robot-state signals.Can be costly and slower to collect at scale.Imitation learning, policy fine-tuning, and embodiment-specific training.
Human demonstration dataProvides natural task strategies and broad task coverage.May require adaptation to a different robot embodiment.VLA pretraining, demonstration learning, and task understanding.

The strongest robotics data strategy usually combines these sources rather than treating one as a replacement for the others. Real-world video can broaden visual and task coverage, while teleoperation and robot logs provide the action and state alignment needed for policy learning.

Sample request checklist

When requesting a robotics data sample, ask the provider to include:

  • A sample of 10 to 50 episodes across your priority tasks.
  • The complete field dictionary and data schema.
  • Example video, sensor, annotation, and manifest files.
  • A description of the collection method and data source.
  • Quality-control criteria and quality-score definitions.
  • Consent, licensing, redaction, and permitted-use terms.
  • Delivery format, storage destination, API access, and refresh cadence.
  • A pilot plan for custom collection, if existing coverage does not match your task.

Request a sample

If your team is evaluating robotics training data providers, VLA datasets, real-world video data for robot training, or alternatives to teleoperation data, request a sample matched to your target task.

A useful first request is:

“Please provide a 20-episode sample for [task type], including first-person video, task instructions, action or interaction annotations, environment metadata, quality scores, license terms, and a manifest in JSON or Parquet.”

Thordata supports multimodal data collection and AI training datasets across video, text, image, audio, and embodied intelligence, including robotics, autonomous systems, and real-world interaction data.