Apple's research team has disclosed progress on its robot "brain" research. On November 4 local time, the Apple research team submitted a paper titled "Perceptive Humanoid Motion from Simulation" on the preprint platform arXiv, unveiling the HALO autonomous control system for humanoid robots.

The research team deployed this system on Unitree Technology's G1 humanoid robot for validation, enabling the robot to carry movements learned in simulation environments into unfamiliar real-world rooms for execution.

The research team also publicly released data generation, model training, and deployment code, covering motion generation, motion tracking, task orchestration, as well as simulation and real robot operation.

The Unitree G1 robot equipped with the HALO system demonstrated tasks such as crawling under tables, sitting on sofas, and pushing boxes in different real environments.

The project page showcased the robot following instructions to push boxes, follow people, sit on chairs, sidestep through half-open doors, and bend down or crawl under tables. These tasks require the robot to understand its surroundings and adjust its body posture based on spatial conditions.

When executing tasks, HALO is coordinated through three hierarchical levels of control. The top layer is a task orchestration module based on a vision-language model (VLM), which by default uses Qwen3.6-35B-A3B, running locally on a Mac mini carried by the robot and sharing the same model serving engine with the motion generator, requiring no internet connection to operate. The system can also optionally call cloud-based models through API to obtain stronger reasoning and planning capabilities.

This layer needs to handle ambiguous information in instructions and arrange actions based on the environment. For example, "sit down" does not explicitly specify a target, requiring the robot to judge where to sit from the current scene; "sit on the chair" can be executed directly when the path is clear, but if a table is blocking the way, it needs to be broken down into consecutive subtasks such as bending down to pass through, approaching the chair, and sitting down.

The task orchestration module continuously observes the environment through the robot's perspective video, judges whether the current action is successful, and advances the plan or retries accordingly, issuing one clear subtask at a time to the motion generator.

The middle motion generation layer uses another Qwen model, Qwen3.5-2B. The research team used it as a parameter-frozen encoder, working in conjunction with a DiT decoder responsible for generating motions, to plan full-body movements for approximately the next 5 seconds based on what the robot sees and the current subtask. This layer replans about 3 times per second, enabling the motion plan to be continuously updated as the environment changes.

The bottom layer is a motion tracker trained through reinforcement learning, policy distillation, and other steps, running at 50 times per second, combining depth images and the robot's own state to convert motion plans into joint control commands. The three layers respectively handle task arrangement, motion planning, and body control, collaboratively completing the process from understanding instructions to executing actions.

According to the paper, the research team conducted 360 single-task tests in 6 real environments that were not pre-scanned or mapped, achieving an average success rate of 91%. In another real environment, 10 tests were conducted for each of 3 combined tasks, achieving an average success rate of 70%. Among them, the combination of "bending down to pass through obstacles—walking toward the target—pushing the box" had a success rate of 50%.

How the robot obtains the data needed to complete these tasks is a question this research focuses on. The research team synthesized approximately 300 hours of primary training data in 1,000 procedurally generated simulated home environments, pairing what the robot "sees," what instructions it "receives," and "how it moves." Specifically, the system first generates motions, then places the motions into virtual rooms, and finally generates visuals from the robot's perspective.

The project page used "sitting on an armchair" as an example, breaking it down into segments such as walking forward, turning around, and sitting down. The motion generation model generates motion sequences based on text prompts, waypoints, and key poses, then filters motions through geometric checks to reduce issues such as foot sliding and body penetrating the ground. Subsequently, the system arranges the motions into procedurally generated rooms, checks the positional relationships between the robot and furniture, finds collision-free placement solutions, and then uses simulation tools to render the visuals the robot sees while executing motions. During training, conditions such as lighting, color tone, and image blur levels are also varied to make visual input more diverse. In this way, the same pipeline can generate paired data containing instructions, visual frames, and full-body motions.

According to the paper, the research team used 250 NVIDIA H100 GPUs and took approximately 48 hours to synthesize the primary training data. The motion generator was trained through imitation learning using 64 B200s; the motion tracker was trained through reinforcement learning, policy distillation, and other steps using 4 H100s.

It should be noted that simulation-generated data has not completely eliminated human involvement. The team stated that newly added physical tasks still require manual prompt writing and motion keyframe design, and some motions also used human videos or motion capture footage.

The research team also pointed out that the robot currently relies mainly on visual perception and has difficulty directly sensing contact conditions; when objects leave the camera's field of view or are occluded, performance on tasks involving more contact degrades. Adding tactile perception is one of the directions for future improvement.