2D and 3D Machine Vision in Robotics: When Do You Need a Spatial Model? ART-010

First define what part pose the robot actually needs to know

The robot does not care what the camera manufacturer calls its product. To move correctly, it needs coordinates that are sufficient for the specific operation.

If parts lie on a known plane, remain at approximately the same height, and the robot approaches them from above in one direction, it is often enough to determine X and Y position plus rotation in that plane. This is a typical class of 2D machine-vision tasks: the camera finds a contour or characteristic features, the system converts image coordinates into robot coordinates, and the gripper corrects its position.

When a part may be higher or lower, tilted, standing on an edge, lying under another part, or arbitrarily oriented in space, two coordinates and one angle are no longer enough. The robot needs to know depth Z and, depending on the task, rotations about the other axes. That creates a need for spatial measurement or another way to recover the 3D pose.

So the choice between 2D and 3D starts with one question: how many degrees of freedom actually change from cycle to cycle?

When a plane is enough

A 2D solution works well when the production environment itself removes uncertainty in height and tilt. For example, parts may travel on a conveyor in one plane, lie in a flat tray, or be located against a table fixture. The camera only needs to determine where the part is in the plane and how it is rotated.

In such a system, mechanical fixturing performs part of the “intelligence”: it fixes Z and spatial tilt before measurement. Vision solves only the uncertainty that remains.

This can often be more practical than a more complex 3D solution. Less spatial data needs to be acquired and processed, and the overall system may be simpler to calibrate and diagnose. At the same time, 2D lighting requirements depend on the scene and on the features that must be detected reliably. If the task consistently reduces to X/Y plus in-plane rotation, an additional depth map may add no useful information for the robot.

At the same time, “2D” does not mean “no calibration.” The system must correctly transform image coordinates into robot coordinates. Cognex's official documentation for 2D vision-guided robotics explicitly describes hand-eye calibration and operation in X, Y, and rotation about Z at a constant working distance.

3D becomes necessary when depth changes the robot's decision

The additional Z coordinate is not valuable by itself. It matters when the robot cannot safely choose a grasp point, approach height, tool orientation, or trajectory without it.

Typical situations include:

• parts lie randomly in a bin at different heights;
• the uppermost part may be tilted or partially occluded;
• the robot depalletizes layers whose height changes;
• the system must inspect not only a contour but also surface shape or height;
• an assembly point is defined by spatial geometry rather than a flat mark;
• the gripper must approach the part from different angles.

In these tasks, a 3D system typically produces an X/Y/Z map or a point cloud. The algorithm then has to do more than simply “see an object”: it must connect the measured geometry to the work pose required by the robot.

“3D” can mean three different things

These concepts are often mixed together in robotics projects, making a system appear more capable than it actually is.

3D sensing is the sensor's ability to measure depth and produce spatial data: a height map, profile, or point cloud. The mere presence of X/Y/Z data does not mean the system already knows where the robot should pick the part.

3D model / CAD guidance is the use of known geometry to match a model to the measured scene. For example, a system may build or import a part model, align it with the point cloud, and associate allowed grasp points with that model.

Spatial pose estimation (pose estimation) is the result the robotic cell needs: the part's position and orientation in coordinates from which a usable gripper pose can be formed. A full spatial pose has six components: X, Y, Z, and three orientation angles.

These layers should not be treated as the same thing. A camera may output a high-quality point cloud, but the integrator still has to recognize the object, estimate its pose, check reachability, and pass usable coordinates to the robot.

A spatial pose does not always require a literal “3D camera”

There is an important engineering nuance: 3D sensing and 3D pose estimation are not the same thing. If object geometry is known and the system can reliably find corresponding points in the image, then in some tasks a spatial pose can be estimated mathematically even from 2D images. Cognex documentation, for example, describes a 3D pose estimator that determines pose by matching a set of known 3D model points with corresponding 2D image points.

This possibility does not remove the physical constraints of the task. If objects are piled randomly, occlude one another, have weak visual features, or their unknown depth directly affects the grasp, direct measurement of the spatial scene may provide a more reliable basis.

So a better technical specification is not “we need a 3D camera,” but “we need this pose, with this level of uncertainty, under these scene conditions.”

Randomly piled parts are a scene problem, not just a part problem

In a bin, the robot does not see one free part. It sees objects at different heights, occlusions, shadows, contact zones, and possible collisions between the gripper and the bin walls or neighboring parts.

Spatial vision is useful here for two reasons. First, it provides geometry for the visible part of the scene. Second, the system can evaluate not only “where is the part?” but also whether a particular grasp point is accessible and whether the tool can approach it.

That is why stable random bin picking combines 3D recognition with motion planning and collision checking. It is no longer just a camera, but a loop of “measure the scene → find a candidate → estimate the pose → choose a grasp → check the trajectory → execute the motion.”

If a supplier shows only an attractive point cloud but does not demonstrate a stable cycle on the actual bin, that is not yet evidence that the solution is production-ready.

Test shiny metal on your actual parts, not on the word “3D”

Dark, mirror-like, polished, or highly reflective surfaces can be difficult for optical measurement. The result depends on the sensing principle, wavelength, illumination pattern, viewing angles, exposure, and depth-reconstruction algorithms.

Manufacturers of industrial 3D systems develop dedicated modes for reflective parts; that alone shows that surface properties are a meaningful factor. But a marketing claim such as “works with shiny metal” should not be generalized to every part. Brushed stainless steel, black scale, oil, a mirror surface, and a geometrically narrow cylinder create different optical conditions.

The test should therefore reproduce the real surface, orientation, contamination, and range of distances. In metalworking, this can matter more than the sensor's nominal resolution.

Robot accuracy starts with the calibration chain

Even a highly accurate camera measures in its own coordinate system. For the robot to use that result, the spatial relationship between the camera, robot, worktable or bin, and, when relevant, the tool must be known.

Hand-eye calibration establishes the transformation between the vision system and robot coordinates. Error in that relationship adds to errors from the sensor, recognition, robot mechanics, gripper, and part fixturing.

That is why asking only “what is the camera accuracy?” is the wrong question. A production task needs final gripper-pose accuracy in the working area. It should be verified not in one convenient location but at representative positions throughout the required volume.

Calibration is also an operational issue. After an impact, camera movement, mounting change, or service work, the team needs a way to verify that the geometric relationship of the system has not changed.

Do not demand accuracy from machine vision that the rest of the process cannot support

In a technical specification, it is useful to break the allowable error down across the chain: scene measurement, localization algorithm, calibration, robot absolute/repeatable accuracy, gripper rigidity, part position in the gripper, and tolerances of the downstream operation.

If the part is mechanically located in a fixture after picking before the precise operation, machine vision may only need to provide coarser guidance. If the robot must insert the part directly into a small clearance, the requirements on the entire chain become much tighter.

This helps avoid two extremes: paying for sensor accuracy that the process does not use, or expecting micron-level results from a cell whose mechanical uncertainty is much larger.

Deeper information has a cost in cycle time and integration

A single 2D image is often enough, with relatively simple feature finding. 3D systems may use stereo vision, structured light, a series of projections, laser profiling, or another principle; the amount of data and the acquisition process can differ substantially.

So there is no universal rule that “3D is slow.” There is a specific cycle: exposure or scanning, spatial-data generation, recognition, pose estimation, transfer to the robot, and trajectory planning. Some computation may overlap with robot motion, while in other cells the scene must be scanned again after every pick.

Compare the complete cycle on the real task, not the time required to acquire a single frame. A system that sees the scene faster but frequently fails and triggers retries can have worse actual throughput.

Sometimes the best machine-vision solution is to simplify the feeding mechanics

If a part can be oriented cheaply and reliably with a guide, nest, vibratory feeder, or simple stop, that may be a better solution than reconstructing a complex 3D scene on every cycle.

Machine vision is especially valuable where mechanical fixturing is too expensive, slow, or inflexible because of a broad product mix. But vision flexibility also has a cost: algorithms, calibration, edge-case testing, and support after product changes.

That is why 2D/3D should be evaluated together with the feeding method. Sometimes it is worth investing more in versatile vision and eliminating dozens of dedicated fixtures. In other cases, one simple locating feature removes enough uncertainty that 2D becomes more stable and less expensive than 3D.

Seven questions that quickly identify the right class of system

1. Does part height change? If Z is constrained by the process, that is a strong argument for 2D. 2. Can the part tilt? If it can, the robot may need additional orientation angles. 3. Are there occlusions and randomly piled parts? That changes the task from locating one part to analyzing a spatial scene. 4. Is the exact object geometry known? CAD or a trained model can significantly help pose estimation, but it does not replace reliable measurement where the scene is ambiguous. 5. What final accuracy is required at the actual operation point? Not at the sensor, but after the complete calibration and mechanical chain. 6. What are the surface and lighting conditions? Real materials, oil, glare, shadows, and changes in surface orientation must be tested. 7. How much time is available for one cycle? Measure detection-to-pick, including rescans and failed grasps.

Once these questions are answered, technology selection becomes much more concrete and a supplier demonstration becomes testable. The effect of the selected vision system on output should then be evaluated in the context of the throughput of the entire robotic cell.

How to prepare the application for a robotic-vision test

For an initial assessment with L-SEL Group, it is useful to provide more than a photo of the part. Include the minimum and maximum scene height, allowed orientations, feeding method, surface type, required accuracy, expected cycle time, and a description of what the robot does after recognition.