Watch a person sit down. They turn, shift their weight, and trust that the seat will meet them where they expect. For a humanoid, a small error in that last expectation can end in a fall. A video of a successful sit leaves out the hard questions: where did the reference motion come from, what taught the robot to balance, and how much of the room did the system already know?
Apple's HALO gives unusually concrete answers. The research is titled Perceptive Humanoid Motion from Simulation; it uses a Unitree G1. We are interested in the division of work that makes the result possible, especially the boundary between producing convincing observations and learning physically executable behavior. [1]
A modular data factory, before there is a policy
| Module | Input and output | What to audit |
|---|---|---|
| Motion generation | Task recipe and constraints → reference joint trajectory | Candidate acceptance and segment continuity |
| Scene generation | Procedural room → geometry, textures, and placement metadata | Geometry coverage and valid free space |
| Observation rendering | Placed trajectory and camera → aligned RGB and motion labels | Camera calibration and pixel–action agreement |
| Physical tracking | Reference plan and onboard observations → joint targets | Contact, stability, and recovery under perturbation |
HALO's public data pipeline has separate stages for scene generation, motion generation, and staging/rendering. Infinigen supplies indoor rooms. Kimodo supplies constrained whole-body motion. The rendering stage combines a trajectory with a room and exports aligned camera observations and motion labels. These intermediate artifacts are useful interfaces, not merely folders in a repository. [3]
For sitting, an authored recipe breaks the behavior into walking, turning, and sitting. Waypoints and keyframes constrain the motion generator; candidate checks catch problems such as foot sliding or floor penetration. A placement stage then fits the trajectory and chair into free space in a room. Generating more examples becomes a matter of varying these inputs and rerunning the relevant stage. [3]
This is where the design earns its keep. Keep the motion fixed and change the room to study visual generalization. Keep both fixed and change the renderer to isolate appearance. Replace the chair and audit placement. A monolithic pipeline makes these questions harder to separate, and a failed experiment becomes harder to diagnose.
The software deserves an audit as well as the diagram. The released sit pipeline records acceptance flags, can retain a best candidate after retries, and documents a placement fallback. Teams reproducing the work should inspect these flags and fallbacks before treating every exported episode as a verified demonstration. A filter is a procedure with measurable errors. [3]
What 'synthetic data' still owes to people
The reported corpus contains 162,698 episodes and 296.3 hours across six skills, staged in 1,000 training rooms. The ten dataset instructions are much narrower than free-form conversation. Several reference keyframes come from edited poses, phone video, motion capture, or G1 logs. The robot policy is simulation-trained; the pipeline still uses human motion knowledge and task design. [1]
The scaling question is therefore: how much work does the next skill require? A new room may be cheap once the scene generator works. A new interaction may need a new recipe, constraints, quality tests, and a reward. These costs scale differently. Counting only synthetic hours hides the expensive part if every new task needs substantial authoring.
Rendering: bake the light, replay the motion
The scene-generation code bakes lighting into unlit textured meshes. The renderer can reuse those assets in Isaac Sim or MuJoCo. The released render stage replays joint trajectories and produces a calibrated head-camera view, with seeded visual randomization. This route separates expensive scene preparation from repeated episode rendering. [4] [5]
An intuitive example is a patch of light on the floor. Full rendering would recompute how light reaches the surface; a baked room carries much of that appearance in its textures. Moving the camera is relatively cheap. Changing the illumination or a reflective surface is less faithful to that shortcut. This is a deliberate approximation with a specific scope.
The paper reports a median bake time of 38.5 minutes per room on one GPU and a held-out-view error of 3.8 grey levels. Those measurements assess image approximation. They do not measure contact accuracy or prove that a policy will transfer. For replication, the code also matters: the MuJoCo path warps a pinhole render to the calibrated fisheye, while the Isaac path uses native fisheye rendering. [1] [5]
Training the planner and training the body are different jobs
The motion generator reads an RGB image and an instruction and predicts a five-second plan. The released configuration freezes Qwen3.5-2B and trains the projections and diffusion-transformer decoder. It uses flow matching, 150 motion frames with 36 values each, and previous-chunk inpainting. Thus, HALO's generator produces a kinematic plan; it does not jointly generate future video as a world-action model would. [6] [2]
The frozen encoder keeps language and visual representations available without updating the full backbone. The smaller decoder learns the robot motion representation. Committing an initial part of a plan also gives the next prediction a continuity constraint: a new plan has to fit what the body is already doing. Otherwise, repeated replanning can ask the controller to follow incompatible futures.
The tracker has its own training curriculum. First, proximal policy optimization (PPO) trains skill experts with privileged simulator state. Next, dataset aggregation (DAgger) distills teachers into a student with onboard observations, including a small depth image. Finally, multi-critic PPO fine-tunes the shared student with wider randomization around a system-identified robot model. [7]
Different skill rewards can have different scales. A critic for each skill family can estimate its own returns while the actor learns shared behavior. The student's depth input addresses another problem: the robot needs local geometry while executing the plan, especially when its body is close to furniture. Semantic recognition alone cannot tell a knee how to clear a table edge.
The public tracker trains in Isaac Lab and supports MuJoCo and hardware deployment. The runtime hierarchy runs the orchestrator around 0.5 Hz, the motion generator around 3 Hz, and the tracker at 50 Hz. These clocks express different jobs: decide the active subtask, replan motion, and react while tracking. Neither the renderer nor the language model supplies the fast physical feedback alone. [7] [2]
What the real-room tests establish
Apple reports 91% single-skill success over 360 real trials in six unseen, unscanned rooms. Three chained tasks average 70% over 30 trials in one room. This is zero-shot transfer to new deployment spaces, within the evaluated skill set. It is not an evaluation of arbitrary household skills. The chained-task gap is a useful warning about what happens at skill boundaries. [1]
A plausible plan can still leave the next skill in an awkward state: the robot finishes one action facing away from the next target, for example. Our deployment concern is that a skill library needs contracts for its exit states and recovery, as well as a success label. More visual diversity alone would not guarantee that the robot searches for a target it has lost.
Can generative rendering replace the simulator?
For HALO's offline RGB dataset, replacing the observation renderer is a reasonable experiment. A generated image could be conditioned on the same geometry, robot pose, camera calibration, depth, and segmentation. The motion labels would stay fixed. The proposed benefit is broader visual coverage without building every material and lighting setup by hand. HALO has not demonstrated that substitution; this is our proposed extension.
LucidSim provides an earlier example on quadruped visual parkour: generative appearance is conditioned by simulator geometry while physics remains in the loop. RoboRender similarly converts simulated trajectories into RGB videos conditioned on depth and robot masks. These studies support testing a visual replacement. Their tasks and bodies differ from HALO's, so their success rates cannot be carried over to a G1 sitting near furniture. [9] [10]
The difficult part is keeping labels true. If generation makes a chair look a few centimetres wider, moves its visible seat, or invents a clear gap under a table, the unchanged motion may become the wrong answer for the new image. Temporal consistency matters when the camera turns; RGB and depth must also describe the same obstacle. A prettier training frame can quietly be a mislabeled example.
Replacing dynamics asks a harder question. The tracker learns from actions that change physical state: feet make contact, the body accelerates, a chair supports weight. A video generator must then preserve action-dependent outcomes and expose state and rewards accurately enough for learning. Visual plausibility is an inadequate test. Reward exploitation and long-rollout drift would need separate, adversarial evaluation.
The practical order is therefore to trial generative RGB offline while keeping geometric labels and physical rollouts fixed. Only after that should a team consider learned dynamics for narrow, validated transitions. For contact-rich whole-body reinforcement learning, a calibrated physics engine remains the more inspectable source of training feedback.
An experiment that could settle the rendering question
Use exactly the same accepted trajectories, scene split, camera poses, and training budget. Compare the current baked renderer, stronger conventional randomization, and geometry-conditioned generative rendering. Keep a fourth mixed-data arm to see whether generated diversity helps alongside the original images. This is a proposed protocol, not an experiment reported by Apple.
Before training, check projected seat and obstacle boundaries, robot silhouettes, camera motion, and agreement with depth. After training, evaluate complete real tasks on held-out furniture and rooms, with trial counts and uncertainty. Track falls, contacts with furniture, interventions, recovery, and completion time. Image quality scores can help debug the renderer; the real policy outcome decides whether to adopt it.
Count rejected generations and storage as well as inference time. A renderer that needs many attempts to preserve the chair boundary may cost more per useful episode than rasterization. For a small team, the relevant number is cost per successful real task after adaptation, including authoring and validation. GPU throughput on its own cannot answer that.
Where we would invest next
The paper reports 250 H100 GPUs for roughly 48 hours of corpus synthesis, 64 B200s for generator training, and four H100s for tracker training. GPU counts without all durations do not give a full training bill. Released code also does not include the trained generator, tracker weights, or dataset, so reproduction requires regeneration and training. [1] [7] [8]
We classify HALO as a controlled experimental demonstration with meaningful real-world transfer evidence. Long-duration intervention rates, safety incident rates, and a complete cost accounting remain unestablished for deployment. The comparisons with earlier systems also use different protocols; they cannot isolate a single architecture's contribution under a matched budget.
Our first investment would be the module interfaces: traceable motion specifications, reusable scene assets, and tests that catch a mismatch between pixels and actions. Then we would test generated RGB on a bounded skill. There is something appealingly practical about that order. A team can improve one expensive part of the pipeline and still know what the rest of the robot learned.
Generated appearance cannot replace contact evidence
ViewCrafter clarifies what generative rendering can contribute: new views guided by imperfect geometry. Anchor1 clarifies a different boundary: accurate whole-body target tracking still leaves grasp stability and force unresolved. Together these analyses argue for a modular HALO-style experiment in which the renderer changes while dynamics, controller, and evaluation remain fixed. Any claim of simulator replacement must then demonstrate equivalent physical task outcomes, not merely better-looking video.
Primary sources
- HALO paper: methods, experiments, and appendix
- HALO project: onboard stack and training overview
- Data pipeline and acceptance/fallback details
- Scene generation and baking
- Renderer implementation and camera calibration
- Generator configuration
- Tracker training and deployment
- Generator training and release scope
- LucidSim: visual parkour from generated images
- RoboRender: robot-oriented generative rendering