Raw in, training-ready outRobot and human data, before and after training

Physical AI does not run short of recordings. It runs short of recordings a trainer can use. A raw file says what a camera saw and what a robot logged. A training set needs to know which hand acted, on which object, in which step, where the camera was, what the robot was told to do, and whether any of it can be trusted.

That gap exists on both sides of a training run. Before training, human video, teleoperation, simulation and image sets have to become labeled, synchronized episodes. After training, deployed robots produce logs that record what a policy actually did, and that record has to become evaluation and the next round of data. Euler does both with one pipeline and one review surface.

Every input, one pipeline

Each recording is inspected and routed to the right reader, so a team rarely has to name a format. The data kind is then derived from what the episode actually contains, and it decides which layers and which checks exist at all. A first-person clip has no gripper to check, a flight log has no hands to pose, and a loose image has no timeline.

That one rule is what lets the same platform serve data before training and data after it.

Every stream on one clock

A policy learns from observations paired with actions. If the camera and the arm disagree on when something happened, the model learns to act on what it has not seen yet. So the first thing Euler does with any recording is put every stream on the video's clock: commanded actions, joint state, gripper, inertial telemetry, pose and velocity.

The review surface plots those streams under the picture with one playhead, next to the signals the layers derive, such as hand speed or head motion. A click on a graph seeks the video. A stream the recording does not carry is labeled as not recorded rather than drawn as a flat line, because a zero is a measurement and an absence is not.

Head cameras often carry more than pixels: an accelerometer, a gyroscope, a gravity vector and a per-frame orientation. Aligned to the video, that is the inertial half of a camera path, already on the memory card. Flight logs are the same idea with different streams. Position, attitude and velocity arrive as tables, and Euler decodes them into named, grouped graphs on the same timeline.

Language at the grain of an action

Language comes in three layers: a caption per clip, named steps per segment, and dense actions at the level of single moments. The grain matters. A policy cannot learn picking up a sponge from a span that also contains rinsing it and putting it back.

On delivered first-person data, dense actions have a median length of 2.5 s and cover 100% of each episode's time.¹ Full coverage is the point: a gap in the action timeline is a stretch of video a trainer can only use by guessing. Every action sits inside its step and names the hand that performed it.

Objects, outlines and masks

Objects are found against the team's own vocabulary and stored as outlines from instance masks, not as boxes alone. Those outlines are what perception models and pretraining runs consume, and what a reviewer checks a label against.

A head camera never stops moving, which is hard on tracking. On head-mounted footage, the overlap between one frame's mask and the next has a median of only 0.41 to 0.55, so a mask carried forward slides off its object within seconds. Euler's tracks are built for that motion and carry an outline on every tracked frame. Figure 5 shows a fast head turn: most identities hold, and the caption says where they do not.

The same layer runs on loose images. A vision dataset gets outlines against its vocabulary, low-confidence detections routed to review, and image checks for exposure, sharpness and contrast measured against the rest of the set.

Hands, in two dimensions and in three

Human video teaches manipulation through the hands. Euler stores 21 keypoints per hand, left and right, with handedness decided once per track so a hand never swaps sides mid-clip. On first-person data a filter keeps only the camera wearer's hands, so a person standing beside the wearer never becomes training signal.

On a recording with hand motion capture, a refinement pass raised the share of frames with both hands present from 67.8% to 79.8%, and the pixel error on matched frames did not move.² Presence is not accuracy, which is why we report both.

3D hands

3D hands are now a shipped option, chosen per project on the 3D layer. They sit in the same metric, gravity-aligned frame as the camera path and the point cloud, so a hand trajectory reads in meters against the scene it acts on. That is the form retargeting and 3D policy learning need.

Their accuracy is tied to depth. Given exact depth, the lift recovers a synthetic hand to under 5 mm;⁵ on real frames, the depth scale along the viewing ray sets the error. The review surface marks hands far from a depth keyframe as lower trust, and an export can leave the layer out.

Geometry for world models

World models learn how a scene changes as the camera moves through it. That needs the motion of the camera and the shape of the scene, in units. Euler produces metric depth per keyframe with the camera's intrinsics, a camera path over the whole episode in a gravity-aligned frame, and point clouds built from both.

The depth view carries its own instruments: a metric color bar with a range lock, the depth under the pointer in meters, a two-point measure and a wipe between picture and depth. When a record is relative rather than metric, it says so.

We measure geometry against ground truth, not by eye. Depth has the right shape: fitting a single scale per frame takes its error from 0.209 to 0.076. The camera path holds metric scale to within 1.4% with no scale fit. On wide indoor lenses the raw depth reads about a fifth long, so scale, not structure, is the quantity we watch.

Robot data, before and after training

Robot data arrives in containers: MCAP and ROS bags, HDF5 recordings, parquet tables beside video, existing LeRobot datasets. Euler decodes them into episodes with every topic's timing summarized, so a dropped camera or a stalled joint stream is visible before anyone trains on it.

After training, the same reader takes in what deployed robots record. A rollout log carries the policy's commands and the robot's state beside its cameras. Scored for readiness and reviewed on the same surface, those logs become evaluation sets and the next round of fine-tuning data, with each finding placed at the moment it happened. The perception layers run on robot video too, so a rollout can be found by what was on the table and what the arm did with it.

Before training, some of the most plentiful robot data comes from people holding a gripper and from simulation. A handheld gripper records its camera and its own trajectory; a simulator records exact state. Both run through the same layers and checks as data from a real robot, so a mix of sources lands in one consistent format.

Readiness you can check

Readiness is Euler's judgment of whether a recording is fit to train on. It is a blend of deterministic checks, each with a plain name, a reason it matters for training, and the data kinds it can never apply to. Sixteen general checks cover every kind:

• Timeline and clock stability
• Control smoothness and dead time
• Sensor alignment
• Gripper signal
• Instruction match
• Camera coverage and image quality
• Label readiness and annotation coverage

First-person data adds checks on how it was captured and on what Euler produced from it, each measured in physical units such as a rate in hertz, a gap in seconds or a coverage share:

• Hands in shot
• Camera on the head
• Clear lens
• Exposure stability
• Head motion
• Frame integrity
• Uninvolved objects
• Near-duplicate clips
• Telemetry sync
• Step and action coverage
• Hand and track coverage
• Depth and camera-path coverage

We list what is checked, not how it is scored. Every check that can place its finding in time does, so the review surface can show the moment and the frames.

The more important choice was what gets certified. A certificate that reads only the raw input will say first-person video is not ready for a vision-language-action model, because language, a task timeline and aligned telemetry are missing. They are missing because producing them is the pipeline's job. Euler certifies the delivered output, and re-measures it from stored files without running any model again.

We audit our own flags too. On one batch, every flag was checked against the frames it pointed at, and each traced to a defect in the measurement, not in the data: a clock offset, a countertop counted as an untouched object, dark clothing read as clipped exposure.⁶ Those were fixed in the checks. Two rules hold throughout: a flag routes an episode to human review and never rejects it on its own, and a re-measure never overwrites a human decision.

A frame is a frame

A reviewer judges a label by looking at it on the picture. If a skeleton trails its hand by one frame, a correct label looks wrong. So we measured the player. The test clip carries its frame number as a burned-in barcode, the overlay draws the number of the frame it was drawn for, and the composited screen is read back frame by frame.⁷

At first, picture and pose came from the same frame on only 17% of screen frames. The clock was exact; the lag was presentation, because an overlay drawn by the page reaches the screen a compositor frame or two after the video. The obvious fix, drawing on a canvas in the video frame callback, measured 26%. Painting the picture and the pose in the same task reached 99.5%, with the rest one frame apart.

Every layer, at a glance

Thirteen layers and the recorded streams, each with what it produces, what it trains and how it holds up. A layer runs only where the data can carry it, and the settings screen says why when it cannot.

What this adds up to

One recording can serve several kinds of model, if its layers are there and can be trusted. Human video becomes policy data through its hands, steps and actions. Ego-motion and depth become world-model data. Outlines and masks become perception and pretraining data. Readiness becomes the evidence for evaluation and for deciding what goes into the next run.

Every export carries that evidence with it: a README written from the export's own files that states each file's schema, units, coordinate frame and time base, a readiness receipt per episode, and the project's certificate. Location and face streams never leave in an export.

An earlier post described how a readiness certificate is built. This one covers the layers that certificate is about, across every input we read, and the measurements behind them.

Method notes

1. 1Counted on delivered first-person data after the language layers were re-run. Coverage is the share of each episode's duration that falls inside a dense-action span. ↩
2. 2A recording with hand motion capture. Presence is counted per frame; pixel error is measured only on frames where both the prediction and the capture exist. ↩
3. 3Frames sampled across a recording with dense metric depth ground truth and a confidence map, scored after 2x2 area averaging. One scale per frame is the median ratio of truth to prediction. "Within 25%" is the usual δ1 measure. ↩
4. 4Absolute trajectory error against the recording device's own head pose, with rigid alignment only and no scale fit. ↩
5. 5A synthetic hand with exact depth, lifted with the same code that runs on real frames. On real frames the error is set by the depth scale along the viewing ray. ↩
6. 6Every flag on one audited batch was checked against the frames it pointed at, one quality component at a time. ↩
7. 7Production build in headless Chromium. The test clip carries its frame number as a burned-in barcode, the overlay draws the number of the frame it was drawn for, and the composited screen frames are read back and the two numbers compared. ↩

← Back to the Euler blog