An occupancy grid tells a robot which cells are free and which are blocked. It cannot tell the robot that one blocked cell is a wall, another is a pallet that will be gone in an hour, and a third is a fire door. Geometry is enough until a task names a thing: fetch tote 14, open the door to room 2.07, stay out of the cleanroom. From then on the robot needs a map that knows what things are. That map is a semantic map.
A semantic map is a spatial model of an environment in which geometry carries meaning: each region, surface or object has a class label, an identity and often attributes and relations, such as "door D-14, opens inwards, connects corridor C2 to room 2.07". Robots use them to plan tasks that refer to objects and places, not coordinates.
Geometry alone is not enough
Classical robot mapping produces metric maps: occupancy grids, point clouds, voxel maps or meshes. They answer one question well, which is where the free space is. Simultaneous localisation and mapping (SLAM) builds such a map while estimating the robot's pose in it, and for point-to-point navigation that is often all a mobile base needs. A metric map has no notion of objects, rooms or functions. A chair and a concrete column are both occupied cells, and a corridor and a loading bay are both free space.
This gap shows up as soon as an instruction refers to the world in human terms. "Go to the kitchen" requires the robot to know which region is the kitchen. "Pick the red tote from rack B3, shelf 2" requires it to know where rack B3 is, which level is shelf 2 and what a tote looks like. A metric map cannot answer either question, so the answers end up hard-coded as coordinates in mission files, which break the first time the layout changes.
The survey on semantics for robotic mapping, perception and interaction by Garg and colleagues frames this as the move from robots that navigate to robots that understand their surroundings well enough to interact with them. The authors organise the field around how semantics are extracted from sensor data and how they are used in mapping, navigation and manipulation. That split is useful in practice: building a semantic map and using one are separate engineering problems with different failure modes.
Semantics also change what the robot can ignore. A person standing in an aisle and a pallet left in the same aisle look similar in a lidar scan. A semantic layer that classifies the first as a person and the second as a static object lets the planner treat them differently: wait for the person, re-route around the pallet, and report the pallet as a change to the shared map.
What a semantic map contains
At minimum, a semantic map adds a class label to parts of the geometry: wall, floor, door, shelf, table, person. Most useful maps go further and add four kinds of information. Instance identity distinguishes door D-14 from door D-15, not only "door" from "wall". Attributes describe properties that matter to a task: the clear width of a doorway, the opening direction of a door, the height of a shelf, whether a floor area is a slope or a step.
Relations connect objects and places. A tote is on shelf 2, shelf 2 belongs to rack B3, rack B3 is in aisle 4 and aisle 4 is in hall 2. A door connects two rooms. These relations are what turn a labelled point cloud into something a task planner can query. Without them, the robot knows that it sees a shelf, but not which shelf, or which room to cross to reach it.
The fourth kind is time. Some labels are permanent, such as a structural wall. Others are valid for a period: a hot-work zone from 08:00 to 16:00, a corridor that is busy between 12:00 and 13:00, a pallet that appeared this morning. A semantic map that records when each label was observed, and how long it is expected to hold, can tell the robot which parts of its knowledge to trust and which to check again.
Every one of these elements needs a coordinate reference. Labels are useful only if they sit in the same frame as the robot's localisation map, at an accuracy that matches the task. A door label that is 30 centimetres off is harmless for room-level routing and useless for a manipulator reaching for the handle. Accuracy requirements therefore follow from the tasks, the same way fidelity follows from use in any digital twin.
Common semantic map representations
The simplest representation is a labelled 2D grid. Each cell in an occupancy grid carries a class as well as an occupancy value, or a separate raster layer holds zone types. Many mobile robot stacks already work this way without calling it a semantic map. In ROS 2, the Nav2 costmap configuration composes static, obstacle and inflation layers, and adds filters such as keep-out and speed filters that encode meaning on top of geometry.
The next step is a labelled 3D representation: a point cloud, voxel grid or mesh in which every element carries a class. Kimera, an open-source library from MIT, builds a 3D metric-semantic mesh in real time on a CPU by fusing semantically labelled images into a dense reconstruction. The result is a mesh where walls, floors, furniture and objects are coloured by class, which is useful for visualisation and for collision checking against specific object types.
Dense labelled geometry still lacks structure. A 3D scene graph adds it by storing the environment as a layered graph of nodes and edges. 3D Scene Graph, by Armeni and colleagues at Stanford, models a building with layers for objects, rooms and cameras, each with attributes such as class, material and room category. Edges record relations between layers, for example which room contains which object.
The choice between these representations depends on the task. A cleaning robot covering floor area is well served by a labelled 2D grid with zone types. A manipulator picking from shelves needs object instances with accurate 3D poses. A humanoid that moves between rooms and handles objects needs both, which is why hierarchical representations that link geometry, objects, places and rooms have become the default in research.
| Representation | What it stores | Typical use |
|---|---|---|
| Labelled 2D grid or zone raster | Class or zone type per cell | Zone-aware navigation, keep-out areas, speed limits |
| Labelled 3D point cloud or mesh | Class per point, voxel or face | Collision checking by object type, inspection, visualisation |
| Object map | Object instances with pose, size and attributes | Picking, fetching, inventory, inspection rounds |
| 3D scene graph | Layered nodes and relations: objects, places, rooms, buildings | Task planning, language instructions, multi-room missions |
How semantic labels are produced
Most semantic maps start from images. A semantic segmentation network assigns a class to every pixel, and an instance segmentation network separates individual objects of the same class. With a depth sensor or a stereo pair and a known camera pose, those 2D labels are projected into 3D and fused over many views, so that a label seen from ten angles carries more weight than one seen once. This is the pipeline Kimera and similar systems implement.
Lidar point clouds can be labelled directly with 3D networks, which avoids projection errors but needs annotated 3D training data. Public datasets have made this practical indoors. ScanNet provides about 2.5 million RGB-D views across 1,513 indoor scenes, annotated with camera poses, surface reconstructions and semantic segmentations, and it has been widely used to train and benchmark indoor semantic models.
Fusion matters as much as the network. A single frame misclassifies objects at the edges of the field of view, under poor lighting or when partly occluded. Mapping systems therefore keep a probability distribution over classes for each map element and update it as new observations arrive. The map label is the class with the highest accumulated evidence, and the confidence can be stored alongside it, which lets the robot treat low-confidence labels with caution.
Trained networks only know the classes in their training set. A model trained on office furniture will not recognise a greenhouse gutter or a pump skid. Industrial sites usually need fine-tuning on site-specific classes, or a source of labels that does not depend on perception at all, which is where existing engineering data comes in.
Semantic maps from CAD and BIM
Buildings and plants are usually designed before they are built, and their design files already hold much of what a semantic map needs. A BIM model in the IFC format, standardised as ISO 16739-1:2024, describes elements by type: walls, slabs, doors, windows, stairs and spaces, with properties and relationships between them. A door in IFC is a door object with a width, a host wall and connected spaces, not a gap in a mesh.
Converting this into a robot-usable map is not automatic. Design files describe the building as designed, not as built, and they rarely include furniture, racking, stock or temporary equipment. Coordinates are in a project reference frame that has to be registered to the robot's map, usually by aligning the model to a lidar scan of the same site. Levels of detail vary: some models carry door swing directions and clear widths, others carry only rough volumes.
The practical approach is to treat CAD and BIM as the structural backbone and perception as the source of everything that moves or was never modelled. Walls, doors, stairs, lifts and rooms come from the model, with stable identifiers. Racking, stock, equipment and temporary objects come from scans and from the robots' own observations. Each label then has a known source, which matters when the two disagree.
Registration is the step that most often goes wrong. A model that is offset by a few centimetres, or rotated by a fraction of a degree, will place door labels next to the actual doorways rather than in them. The fix is to register against the scan with an explicit error budget, check residuals at known control points such as columns and door frames, and record the transformation so it can be reapplied when either the model or the scan is updated.
3D scene graphs and open vocabularies
Scene graphs have moved from offline research tools to systems that run on robots. Hydra, presented at Robotics: Science and Systems in 2022, builds a 3D scene graph incrementally while the robot explores, with layers that range from low-level geometry to objects, places, rooms and buildings. The authors report accuracy comparable to offline batch methods while running online, which makes the representation usable during a mission, not only after it.
The hierarchy is what makes scene graphs efficient for planning. A planner asked to fetch an object from another floor first searches the building and room layers, which hold a few dozen nodes, then the place layer inside the target room, and only then the dense geometry near the object. Planning over the full mesh of a building would be far slower and would gain nothing from the extra detail.
Recent work removes the fixed list of classes. ConceptGraphs builds open-vocabulary 3D scene graphs by applying 2D foundation models to images and fusing their outputs into 3D through multi-view association. The map can then represent object types that were not in any training set and answer queries phrased in natural language, without collecting a large labelled 3D dataset for each new environment.
Open vocabularies trade predictability for coverage. A fixed class list can be validated: you know which classes the map contains and can measure accuracy for each. An open-vocabulary map can describe almost anything, but its labels are harder to audit. For industrial and safety-relevant use, a common pattern is a fixed, validated set of classes for anything that affects motion or safety, with open-vocabulary labels added for search and task description.
How robots use semantic maps
Navigation is the first beneficiary. Semantic zones turn into costmap rules: keep out of the cleanroom, slow down in the canteen, prefer the marked aisle. Nav2's keepout filter shows how this works in practice. A filter mask, stored as a map image, marks cells the robot must never enter, cells it should avoid where possible and lanes it should prefer, and the planner routes around them.
Task planning is the second. With object instances and relations, a mission can be written as "deliver tote 14 to station 3" instead of a list of coordinates. The planner resolves names to poses at run time, so a mission keeps working when a station is moved, as long as the map is updated. This is what makes fleets maintainable: missions refer to things, and the map tracks where things are.
Manipulation is the third. A humanoid or mobile manipulator needs object-level detail: which way a door opens, whether a handle is a lever or a knob, which drawer slides out, where a tote's grasp points are. Some of this comes from perception at the moment of grasping, but a prior from the map narrows the search and gives the robot something to check against. Our humanoids solution is built around exactly this kind of object and place model.
Inspection and maintenance robots use semantics to link observations to assets. A quadruped reading a pressure gauge has to store the reading against the right gauge ID, not against a coordinate. A semantic map that holds the asset register gives every reading a home, so values build into a history per asset. Our robotics solution describes how inspection missions are built from assets rather than waypoints for this reason.
Semantic maps in simulation
Simulation needs the same semantics, for two reasons. Robot software that queries the map for doors, shelves or totes has to find them in the simulated scene as well, or it cannot be tested before deployment. And synthetic training data for perception models is only useful if every rendered object carries its class, so that segmentation masks and bounding boxes are produced automatically.
In NVIDIA Isaac Sim, semantics are attached to prims in the OpenUSD scene. The Isaac Sim Replicator documentation states that semantically labelling prims is necessary for annotators such as semantic segmentation and bounding boxes, and provides a Semantics Schema Editor to add, edit and remove labels. A scene exported from a semantic map, with labels already attached, can produce annotated synthetic data without a separate labelling pass.
This creates a useful loop. The same labelled model drives the robot's map, the simulation scene used to test its tasks and the synthetic data used to train its perception. When the site changes, updating the semantic map updates all three. When the three are maintained separately, they drift apart, and tests pass in a simulated site that no longer matches the real one.
Keeping a semantic map current
A semantic map is only as good as its last update. Labels go stale faster than geometry: walls rarely move, but stock, racking, equipment and zone rules change daily or weekly. A map with confident but outdated labels is worse than a map with no labels, because the robot acts on information that is wrong and has no reason to doubt it.
Keeping it current needs three things. A change signal, from repeat scans, robot observations or fixed sensors, that detects when the observed scene no longer matches the map. A validation step, so that a single misclassified frame does not rewrite the map. And version history, so that every label records when it was observed, from which source, and what it replaced. Without history, nobody can tell whether a wrong label came from a bad observation or a real change.
Fleets add a further requirement. When one robot observes that a pallet has appeared in aisle 4, every other robot planning through aisle 4 should know. That needs a shared map that robots read from and report to, rather than separate maps on each machine. It also needs conflict rules for when two robots disagree, and a policy for how long an unconfirmed observation stays in the map before it expires.
This is the same problem as keeping any digital twin current, discussed in more detail in what is a digital twin: a defined update loop, a stated fidelity and a record of change. A semantic map is a digital twin built for machines to read, and it fails in the same way when the update loop is missing.
How Kaiinos builds semantic maps
Kaiinos builds operational digital twins of sites from CAD and BIM, lidar, drone, satellite and sensor data. For robot use, the twin is the semantic map: geometry registered to a common reference frame, elements labelled by type with stable identifiers, attributes such as door widths and shelf positions, and change history for every element. Our Smartout platform serves this model through APIs and exports it as OpenUSD scenes for simulation.
Our starting point is engineering data where it exists. Doors, walls, stairs, lifts and rooms come from the design model with their identifiers kept, and lidar or drone surveys provide the as-built geometry and the objects the model never contained. The two are registered, differences are flagged rather than silently overwritten, and each label carries its source. The same approach underpins our digital twins solution for sites without robots.
What we do not do today is run perception on the robot. The robot's own software, or its integrator's, handles localisation, planning and motion. The next layer we are building towards is a robot perception engine that takes camera and lidar feeds from robots, detects local changes and updates the shared twin in near real time. Until then, the map is updated from repeat surveys, scans and telemetry on a defined schedule.
Frequently asked questions
What is the difference between a semantic map and an occupancy grid?
An occupancy grid records whether each cell is free, occupied or unknown. A semantic map adds meaning: what each region or object is, its identity, attributes and relations to other objects and places. Many semantic maps include an occupancy grid as one layer, with labels, zones and object instances stored alongside it.
What is a 3D scene graph?
A 3D scene graph is a semantic map stored as a layered graph. Nodes represent things at different levels, such as objects, places, rooms and buildings, and edges represent relations, such as "contained in" or "connected to". The hierarchy lets planners reason about large environments quickly, from building level down to individual objects.
Can a semantic map be built from BIM?
Yes, for the structural elements. IFC models describe walls, doors, stairs and spaces with types, properties and relations. They need registration to a scan of the real site and usually lack furniture, racking, stock and temporary equipment, so perception or surveys fill in what the design model does not contain.
How accurate does a semantic map need to be?
It depends on the task. Room-level routing tolerates labels that are tens of centimetres off. Manipulation, such as opening a door or picking from a shelf, needs object poses at centimetre level in the robot's localisation frame. Set accuracy requirements per task and per class, and record the measured accuracy with each label.
Do semantic maps replace robot perception?
No. A semantic map is a prior: it tells the robot what it should expect to find and where. The robot still perceives the scene, detects people and unexpected objects, and confirms or corrects the map. The two work together, and observations from robots are one of the main ways the map is kept current.
Give your robots a map with meaning
If your robots need to find doors, shelves and assets by name rather than by coordinate, the map is where to start. We can build a semantic model of your site from the data you already have and keep it current as the site changes. Book a technical call to discuss your site and robots.