Companies are investigating the use of brain wave data to train physical AI, seeking to overcome a significant scarcity of real-world training information for robotics.
Solving the Physical AI Data Bottleneck
The development of advanced physical AI, particularly in robotics, faces a substantial challenge: a lack of comprehensive real-world training data. Encord, a company specializing in data tooling for AI models, is experimenting with novel methods to generate this crucial information.
One such method involves a pilot, Andrew Ceja, wearing a headset equipped with sensors that measure brain waves. This data is collected while Ceja performs tasks, such as carefully disassembling a Jenga tower, in San Leandro, California. The aim is to deduce mental states like error, intent, and surprise to create more effective datasets.
New Data Modalities for Robotics
Encord's initiative is a collaboration with Zander Labs, a German neuroscience startup. The trial aims to create an initial dataset tagged with brain wave activity. This data will then be evaluated for its impact on customer robotics models before potential scaling.
"The amount of brain activity used at any point during a given task offers clues for model builders trying to figure out when they need to deploy their highest-effort models," stated Lucas Gehrke, a Zander neuroscientist supervising the work.
Vineeth Velmurugan, Encord's head of robot learning, described this effort as being on the "bleeding edge" of resolving the robotics data bottleneck. Velmurugan, who previously worked at OpenAI's robot lab and Berkshire Grey, joined Encord to establish its data-creation team.
Manufacturing Data for Advanced AI
Encord was initially founded to assist companies with machine-vision applications in annotating data and evaluating models. However, as clients increasingly applied end-to-end learning to robotic manipulation, Encord recognized the necessity of producing training data rather than just managing it. "The data simply does not exist," Velmurugan commented.
The challenge mirrors that of self-driving car companies, which also collect their own physical-world data, a process that is difficult to scale. While training from video is an option, it lacks the fidelity of real-world data. Velmurugan estimates that a dataset five times the size of YouTube's video corpus might be required to achieve a significant breakthrough, highlighting why data generation has become a distinct business sector.
Companies building robot intelligence are turning to two primary sources: "egocentric" video, captured by workers wearing cameras and often supplemented with additional viewpoints and metrics, and data from remotely operated robots. Encord is engaged in both, collecting egocentric data from factories globally and using its San Leandro facility for experimentation with new methods like brain wave measurement and skill-specific dataset creation.
Future Implications and Industry Trends
Velmurugan noted that progress is ongoing, with Encord observing industry-wide developments in effective data techniques. This intermediary position allows Encord to identify emerging trends in physical AI model development before individual customers can.
The company employs approximately a dozen pilots, including Sofia Infante, who is developing techniques for robots to perform precise tasks like plugging and unplugging ethernet cables. These pilots represent a growing workforce focused on building blocks for neural networks.
Encord also utilizes a set of forearm sensors to detect electrical signals in muscles, aiming to create a more robust 3D understanding of hand movements for AI models, even when video data does not capture the entire hand.
The intensive annotation of physical descriptions for video data, such as "right hand tightens bolt," is estimated by Velmurugan to be significantly more valuable for training specific tasks than less detailed "ego data," despite a higher production cost.
