Are brain waves the next unlock for physical AI?

The Frontier of Physical AI Data

The frontier of physical AI is currently represented by a Jenga game inside a warehouse located in San Leandro, California. That warehouse is occupied by Encord, a company that builds data tooling used to train AI models. Andrew Ceja works as a pilot, which is the company’s term for its robotic trainers. He carefully pulls wooden blocks from a tottering tower while wearing a headset equipped with a camera that tracks what he sees. While wearing cameras is fairly common for collecting robot training data, this specific headset includes sensors that measure his brain waves as he disassembles the block tower.

Encord is one of a growing number of startups betting that the next real constraint on humanoid and warehouse robotics will be the scarcity of real-world physical training data. The company is building a business not just to manage that data, but to manufacture it. According to Vineeth Velmurugan, Encord’s head of robot learning, this is the bleeding edge of the effort to solve the robotics data bottleneck. Velmurugan is a veteran of OpenAI’s robot lab and Berkshire Grey, the warehouse automation firm, who joined Encord to build the company’s internal data-creation team.

Measuring Brain Activity and Mental States

The brain wave headset worn by Ceja was built by Zander Labs, a German neuroscience startup. Zander Labs is betting that measuring brain activity to deduce mental states like error, intent, and surprise can create a more useful dataset to train models. Encord’s work with Zander is currently a trial run. Encord states that the goal is to build an initial brain wave-tagged dataset, run it through customer robotics models, and evaluate if it actually improves performance before deciding whether to scale it up.

Lukas Gehrke, a Zander neuroscientist supervising the work, explains that the amount of brain activity used at any point during a given task offers crucial clues for model builders. These clues help them figure out when they need to deploy their highest-effort models. This experimental modality is part of a broader push to find new ways of capturing the fidelity needed for physical AI, a topic detailed further in the original TechCrunch report.

The Economics and Challenges of Manufacturing Data

Encord was originally founded to help companies building machine-vision applications annotate data and evaluate models. As their customers began to apply end-to-end learning to robotic manipulation tasks, executives realized they would have to produce training data themselves rather than simply manage it. Velmurugan notes that the required data simply does not exist. The bet that generative AI can do for robots what it has done for chatbots keeps running into this same wall. While self-driving car companies collect physical-world data themselves, it remains hard to scale. Training from video can work, but it lacks the fidelity of real-world data.

Velmurugan estimates that training will take a dataset something like five times the size of YouTube’s video corpus to break through. This massive scale helps explain why data-generation itself has become a business rather than just a research problem. Scraping text off the internet cost frontier labs next to nothing, but generating physical training data does not. This reality changes the economics of building these models, because physical training data has to be manufactured rather than merely collected.

Diverse Modalities and Dense Annotations

Companies building robot brains now turn to two main sources: egocentric video collected by workers wearing cameras, often augmented with additional camera angles and other metrics, and data from robots operated remotely. Encord does both, drawing egocentric data from several factories around the globe, and using its San Leandro facility to experiment with new modalities like brain waves, or to collect datasets around specific skills for fine-tuning.

During visits to the facility, pilots use leader-follower rigs, which are paired robotic arms where one is controlled directly by a human operator and the other mimics its movements. They create data about tasks like pouring coffee from a pot into mugs and stacking poker chips. Storage racks hold cartons of fake flowers in vases, books, plastic vegetables, kitty litter trays and scoops, and bags and bundles of wires. At another station, a pilot maneuvers robotic arms to plug and unplug ethernet cables from the back of a server, mimicking work data center operators want automated.

Expanding the Workforce for Neural Networks

Another new data modality that Encord is developing uses a set of sensors strapped to the forearm to detect electrical signals in muscles. Video taken of human hands manipulating objects typically does not capture the entire hand, but Velmurugan hopes to build a 3D depiction of hand position at any time based on the arm sensors, creating a more robust understanding for models.

Encord’s datasets are annotated with physical descriptions of what each video contains, such as a right hand tightening a bolt, to aid LLM-based models in understanding what is happening. Velmurugan estimates this kind of dense annotation is worth 100 times as much as junky ego data for training specific tasks, and it only costs 20 times more to produce.

This continuous experimentation keeps the dozen or so pilots at Encord’s facility busy. Pilots Sofia Infante and Andrew Ceja are part of a burgeoning workforce developing the building blocks for neural networks, having previously worked at Scale before joining Encord. Ceja, who previously worked at a waste management company keeping a robotic trash sorter running, enjoys the challenge of solving training tasks for robots as the Jenga tower topples.