In 2023, a collaboration of 21 institutions pooled robot data from 22 different robots into one training set covering 527 skills, published as Open X-Embodiment. A year later the DROID dataset added 76,000 demonstration trajectories, about 350 hours of interaction, gathered by 50 collectors across 564 scenes. Data like this teaches a robot how to grasp, wipe, fold and place. None of it describes the warehouse, ward or plant room where a particular humanoid will start work next month.
Humanoid robot training data comes in two kinds. Skills data, from teleoperation, motion capture, human video and simulation, teaches a robot what to do and transfers between places. Site data describes one real place: its layout, objects, rules and how it changes. A humanoid needs both to work reliably on a specific site.
What skills data teaches a humanoid
Skills data is any recording that links what a robot sees to what it should do with its body. A typical sample pairs camera frames, joint positions and gripper states with the action that followed, often with a short language instruction such as "put the cup in the sink". Train a policy on enough of these samples and it learns a mapping from observations to motor commands. The result is a skill: open a drawer, pick a box from a shelf, hand an object to a person, walk up a step.
The field has moved quickly from single-task policies to generalist models. NVIDIA describes GR00T N1 as a vision-language-action model for humanoids with two coupled parts: a vision-language module that interprets the scene and the instruction, and a diffusion transformer that produces motor actions in real time. Models of this kind are trained once on a large, mixed dataset and then fine-tuned for a specific robot or task. The more varied the training data, the better the model copes with objects and situations it has not seen.
The important property of skills data is that it is meant to generalise. A good grasping skill should work on a mug in Rotterdam and a mug in Lyon. Collectors deliberately vary lighting, tables, objects and backgrounds so the model does not memorise one room. That is exactly what makes a skill useful across customers. It is also why skills data, by design, says very little about any single place.
Where skills data comes from
There are four main sources, and most humanoid programmes use a mix of them.
The first is teleoperation. A person controls the robot directly, through a leader arm, a VR headset or an exoskeleton, and every movement is recorded with the robot's own sensors. The ALOHA project showed that a low-cost bimanual set-up and about 10 minutes of demonstrations per task could teach fine skills such as slotting a battery, with 80 to 90 percent success on the tasks reported. Teleoperation produces high-quality data in the robot's own body, but it is slow and needs trained operators.
The second is motion capture of people. Suits or marker systems record how a human walks, lifts and reaches, and that motion is retargeted onto the robot's joints. It is a fast way to get natural whole-body movement, especially for locomotion and balance, but the result still has to be adapted to the robot's mass, joint limits and hands.
The third is human video. Large egocentric datasets such as Ego4D, with 3,670 hours of daily activity filmed by 931 people across 74 locations, show how people handle objects in real homes and workplaces. Video carries no robot actions, so models use it to learn what tasks look like rather than how to move a specific body.
The fourth is simulation. Physics engines generate millions of attempts at a skill under randomised conditions. This is cheap and safe, and it is a common way to train walking and balance controllers. The weakness is the gap between simulated and real physics, which we return to below.
Several companies now supply skills data as a service: teleoperation fleets, data collection studios, motion libraries and annotation pipelines. For a humanoid maker this is a sensible way to scale. The skills come in faster than any one team could record them, and the maker can focus on hardware, control and deployment.
Why general skills meet specific sites
A humanoid that has learned to pick a box is not yet ready to pick boxes at a customer's distribution centre. The skill is there. What is missing is knowledge of the place.
Think about what the robot needs to know on day one. Which of the 40 identical racks holds the totes it should pick from. Which door leads to the dispatch area, and whether it opens inwards. Where the floor changes from concrete to a steel plate that is slippery when wet. Which corridor is busy at 12:00 when shifts change. Which areas are off limits during a forklift delivery. None of this can be learned from a dataset collected somewhere else, because none of it is true anywhere else.
The International Federation of Robotics makes the case for humanoids in its position paper on humanoid robots: our environments are built around the human body, so a robot with a human form could work in them without rebuilding the site. That argument is sound. It also means every site a humanoid enters was designed for people who learned it by walking around, asking colleagues and reading signs. The robot does not have that background. It has to be given it.
In practice, integrators fill this gap by hand today. They map the site with the robot's own sensors, mark zones, name locations and write site-specific logic. That works for one pilot. It does not scale to a fleet of humanoids across dozens of sites, and it tends to break the first time the site changes.
What site data contains
Site data is a structured description of one real place, built so that machines and people can query it. It has several layers, and each answers a different question the robot will ask.
Geometry is the base layer: walls, floors, ceilings, racks, stairs, doors and fixed equipment, with real dimensions and positions. It comes from CAD and BIM models, lidar scans, photogrammetry and drone surveys. Geometry tells the robot where it can physically go and what it might collide with.
Semantics adds meaning. A shape becomes "rack B2, shelf 3" or "fire door D-14, opens inwards, must stay closed". Objects get identifiers and classes, rooms get names, and assets link to records in the customer's own systems. This is what lets an instruction like "take tote 14 to bay 4" resolve to a real place. Our article on semantic maps covers this layer in more detail.
Rules turn the site into something a robot can work in safely. Keep-out zones, speed limits near people, areas that are only open at certain times, routes reserved for forklifts. These change during the day and need start and end times, not just shapes.
State and history record what changed and when: a rack that moved, a pallet left in an aisle, a door that is now locked. Patterns in that history, such as when corridors are busy, help a robot plan around people rather than through them.
Every new worker gets a site induction
A useful way to see the difference between the two kinds of data is to think about how people start a new job.
A staffing agency places an experienced warehouse operative at a new client. The operative already has the skills: lifting safely, driving a pallet truck, scanning stock, reading a pick list. Nobody teaches those again. What happens on the first morning is a site induction. Someone walks the new starter round the building, shows the fire exits, explains which aisles are one-way, points out where the chilled goods are and where visitors must not go. Within a day the skilled worker becomes a useful worker on that site.
Skills data does the job of training and experience. Site data does the job of the induction. A humanoid with excellent skills and no induction is like a highly qualified temp dropped at the gate with no map and no briefing. It can do the work in principle. In practice it will waste time, make mistakes and need constant supervision.
There is one important difference from the human case. A person keeps learning the site every day, from colleagues and from what they see. A robot only knows what it has been given and what its own sensors can see right now. If the site model is out of date, the robot is working from last month's induction.
Why the site keeps changing
Most buildings look stable to the people who work in them. To a robot they change all the time.
In a warehouse, racking is reconfigured for seasonal stock, pallets land in walkways, temporary barriers appear and new equipment arrives. In a hospital, beds and trolleys move constantly, wards are reorganised and some doors are locked at night. In a plant, maintenance closes areas for hours or days, scaffolding goes up and comes down, and hot-work zones open and close. Outdoors, vegetation grows, ground conditions change with the weather and temporary structures come and go.
A robot's own sensors catch some of this. If a pallet is in front of it, the camera sees the pallet. What the sensors cannot do is tell the robot about changes it has not reached yet, or explain what a change means. A new barrier might be temporary or permanent. A locked door might be locked for the night or for good. That context lives in the site model and in the people who run the site.
This is why site data has to be maintained, not just captured once. The useful version is a living model: updated from repeat scans, from the robots' own observations and from site systems, with each change confirmed and stored as a new version. Robots and people then work from the same, current picture.
Testing in a copy of the site
The second job of site data is to let teams test before they deploy. A humanoid that will work in a specific building can rehearse in a simulation of that building, with the right layout, the right objects and the conditions the site actually has.
This matters because general simulation and real sites differ. Research on sim-to-real transfer describes how the gap between simulated and real worlds degrades the performance of policies once they move onto physical robots. Part of that gap is physics. A large part is the environment itself: a generic warehouse scene is not the warehouse the robot will work in. Building the scene from the real site's data closes the environment part of the gap.
Modern robotics simulators make this practical. OpenUSD is an open format for describing and exchanging complex 3D scenes made from many assets, and NVIDIA Isaac Sim is a USD-native simulator for robots and scenes with sensors. A site scene in OpenUSD can carry geometry, labels and materials, and can be loaded into the simulator alongside the robot model. Teams can then check whether routes fit, whether the robot can reach the shelves it needs, how it handles a blocked aisle and what happens when a door is closed.
The same scene can be changed to test conditions that are rare or risky on site: a busy corridor at shift change, a closed fire door, stock in unusual places. Problems show up on a screen rather than in front of a customer.
How skills and site data fit together
The two kinds of data are not competing. They do different jobs at different points in a humanoid's life, and they come from different suppliers. The table below summarises the split.
| Skills data | Site data | |
|---|---|---|
| Question it answers | What to do and how to move | Where to do it, and under which rules |
| Examples | Grasping, walking, folding, handing over | Layout, rooms, racks, doors, zones, assets |
| Main sources | Teleoperation, motion capture, human video, simulation | CAD and BIM, lidar, drone surveys, site systems, robot observations |
| Scope | General, transfers between sites | Specific to one site |
| How often it changes | When the robot learns a new skill | Daily or weekly, as the site changes |
| Who usually supplies it | Robot makers and skills data companies | Site model and digital twin providers |
In a working deployment the two meet inside the robot's software. The skill policy decides how to pick the tote. The site model tells it which tote, on which shelf, in which aisle, and which route to take to get there without crossing a closed zone. The robot's own navigation stack plans the path and controls the motion, using the site model as its reference.
It is tempting to treat site knowledge as a one-off integration task at the end of a project. Site knowledge is better treated as a second data stream, supplied and maintained alongside skills, with its own owner, formats and update cycle.
Where Kaiinos fits
Kaiinos builds the site side. Our work starts with the data that already describes a place, such as CAD and BIM models, lidar point clouds, drone surveys and satellite imagery, and turns it into an operational digital twin: a current, queryable model of the site with assets, zones, change history and APIs. More than a decade of geospatial work sits behind this, from city-scale mapping to digital twins of bridges and large conservation programmes.
For humanoid makers and integrators, that twin becomes the site induction. We label rooms, doors, racks, shelves and assets, store zones with validity times, and expose all of it through APIs the robot software can call. You can see how this works for humanoids on our humanoids page.
We also build simulation scenes from the same data. Scenes are delivered in OpenUSD for NVIDIA Isaac Sim and can be exported to Gazebo. For sites that change in predictable ways, we model those states too. In greenhouse work, for example, a crop growth model converts biomass into plant height, canopy width and leaf area for each growth stage, so a robot can be tested against the crop as it will be, not only as it is today.
The next layer we are building is a perception engine and an edge module, Kaiinos Node, so that what robots see on site can update the shared model in near real time. We are building towards that. It is not part of today's offering, and Kaiinos does not navigate or control robots. Your robot's own stack stays in charge of motion.
What this means for humanoid makers
If you build humanoids, the practical question is how site data gets into each deployment without your team rebuilding it by hand every time. A few points are worth settling early.
Decide who owns the site model. If integrators build it from scratch for each pilot, every site becomes a custom project. A shared model, maintained by a specialist and consumed through APIs, lets your software treat sites the same way it treats robots: as configurations, not one-offs.
Separate skills from site logic in your software. Skills should not contain hard-coded coordinates, room names or zone shapes. Those belong in the site model, so a skill trained once can run in many places.
Use the site model for testing, not just for operation. A scene built from the real site lets you check reach, clearance and routes before shipping a robot, and lets customers see what will happen before they commit floor space.
Plan for change from the start. Ask how the site model will be updated, how changes are confirmed and how the robot fleet learns that a new version exists.
Think about safety alongside data. Standards for this class of robot are still being written. ISO is developing ISO 25785-1, safety requirements for dynamically stable industrial mobile robots, whose scope includes legged robots. Up-to-date zones and site rules will be part of how deployments show they are safe.
Frequently asked questions
What is the difference between skills data and site data?
Skills data teaches a robot how to do things: grasp, walk, fold, open a door. It is collected through teleoperation, motion capture, human video and simulation, and is meant to work in many places. Site data describes one real place: its layout, objects, rules and changes. A humanoid needs skills to act and site data to act in the right place.
Can a foundation model replace site data?
No. A foundation model can recognise a door or a shelf it has never seen, which is very useful. It cannot know which door is the fire exit in your building, which shelf holds tote 14 or which corridor is closed this afternoon. That knowledge is specific to the site and has to come from a site model.
How is site data collected?
Usually from data the site already has, such as CAD and BIM models, plus new capture where needed: lidar scans, photogrammetry or drone surveys. Assets and zones are then labelled and linked to site systems. Once in use, the model is updated from repeat scans, robot observations and confirmed changes reported by the site team.
Do you need a digital twin to deploy a humanoid?
Not for a single pilot. A team can map one site by hand. For several sites, or a site that changes often, a maintained digital twin saves time and reduces errors, because every robot and person works from the same, current picture and changes are tracked rather than rediscovered.
Can site data be used for simulation?
Yes. The same model can be exported as a simulation scene, for example in OpenUSD for NVIDIA Isaac Sim. Teams then test routes, reach and edge cases against the real layout before the robot arrives, and repeat the tests when the site changes.
Give your humanoids a site induction
Skills will keep improving, and much of that work will come from robot makers and skills data companies. The place each robot works in still has to be described, tested and kept current. If you are planning humanoid deployments and want to talk about site data, book a technical call.