Structure from Motion (SfM): How Overlapping Photos Become 3D
Structure from motion (SfM) is the first half of photogrammetry. Given a set of overlapping photos taken from different positions, with no record of where the camera was, it recovers the position of every camera and a sparse set of 3D points on the scene. Everything that follows in a mapping pipeline — the dense point cloud, the surface, the orthomosaic — is built on that result, so the quality of the SfM step sets the ceiling for the rest.
STREAM Capture is in development and not released yet. At launch it will come in the same download as its sister program STREAM Terrain; today's download contains STREAM Terrain.
What structure from motion means
The name says what happens. The structure is the 3D shape of the scene; the motion is the camera moving between photos. Move your head sideways and nearby objects shift against the background more than distant ones; that shift, called parallax, carries depth. SfM measures how thousands of points shift between photos and works backwards to the 3D positions that explain it.
Older photogrammetry needed cameras with a known calibration and surveyed positions before it could start. SfM removes that requirement: it solves the camera positions, the lens and the 3D points together, from the photos alone. That is why it works with an ordinary drone camera and an unordered folder of images, and why its raw result has an arbitrary scale and orientation until camera positions or ground control tie it to a coordinate system.
The steps of an SfM pipeline
Software finds distinctive points in each photo, recognises the same points in neighbouring photos, discards matches that cannot belong to one rigid scene, and then builds the camera network outward from a well-connected starting pair. Each point seen in several photos is triangulated, and a final adjustment refines everything at once.
| Stage | What happens | What usually goes wrong |
|---|---|---|
| Feature detection | Distinctive points are found in each photo and described so they can be recognised in other photos | Too little texture: still water, fresh snow, blurred photos |
| Matching | The descriptions are compared between overlapping photos to find the same point twice | Repetitive texture such as roof tiles or crop rows gives false matches |
| Geometric verification | Matches that do not fit one rigid scene are discarded | Moving objects, or strong parallax on steep terrain |
| Initial reconstruction | A well-connected pair of photos gives the first cameras and points; further photos are added to it | A weak photo network: groups of photos that never connect to the rest |
| Triangulation | Each point seen in several photos is placed in 3D where its viewing rays meet | Photos taken from almost the same spot give imprecise depth |
| Bundle adjustment | Cameras, lens parameters and points are adjusted together to agree with every measurement | A lens model that does not fit, or a wrong starting focal length |
Bundle adjustment: where the numbers come from
Bundle adjustment is the last and most important step. It moves the cameras, the lens parameters and the 3D points together to make the sum of squared reprojection errors as small as possible. The reprojection error it reports, in pixels, measures how well the photos and the solved model agree; our article on reprojection error explains how to read it.
Because the adjustment works on image measurements alone, an SfM result has no real-world scale, position or north until something supplies them: camera positions from an RTK or PPK drone, ground control points, or both. How well the finished map then sits on the ground is a separate question, answered at independent check points.
SfM, photogrammetry and SLAM
SfM is one stage of photogrammetry, not a synonym for it. Photogrammetry continues from the SfM result with dense depth estimation, surface building, orthorectification and an accuracy check against independent points. SfM software is therefore the alignment part of photogrammetry software.
SfM is also not SLAM. SLAM runs in real time on an ordered stream of images or sensor data, usually on a moving robot, and trades some accuracy for speed. SfM works offline on an unordered set of photos and can optimise everything globally, which suits mapping, where accuracy matters more than speed.
What SfM needs, and where it fails
SfM needs overlap, sharp photos and visible texture: each point on the ground should appear in several photos from different positions, and the software has to be able to tell one spot from another. At least three overlapping photos are needed to start and a real site needs far more. It struggles on still water and fresh snow, on repetitive texture, on things that move between photos, and on cameras with a rolling shutter, which image the scene row by row while the drone moves.
In STREAM Capture this is the Align stage. It extracts and matches SIFT features on the GPU, solves the cameras and the lens, and writes camera poses, a sparse cloud and the reprojection error at full resolution. Positions read from the photos act as a prior, a weak photo network is named in the report instead of being hidden, and the lens self-calibration is reported with its uncertainty.
Frequently asked questions
What is structure from motion (SfM)?
A method that recovers the position of each camera and a sparse set of 3D points from overlapping photos alone, using how points shift between photos as the camera moves. It is the alignment stage of photogrammetry.
How does structure from motion work?
It detects distinctive points in each photo, matches them between photos, removes wrong matches, builds the camera network from a well-connected pair, triangulates the points and refines cameras, lens and points together with a bundle adjustment.
What is the difference between SfM and photogrammetry?
SfM is the first stage: it gives camera positions and a sparse point set. Photogrammetry covers the whole chain, adding dense depth, surfaces, orthophotos and accuracy checks against independent points.
What is the difference between SfM and SLAM?
SLAM works in real time on an ordered stream of images or sensor data, usually on a moving robot. SfM works offline on an unordered set of photos and optimises globally, which suits mapping.
Does SfM need ground control points?
Not to build the model: SfM works from the photos alone, but the result has an arbitrary scale and position. Camera positions from an RTK or PPK drone, ground control points or both put it on the ground, and independent check points measure how well.
Why does my SfM alignment fail or leave cameras out?
Usually too little overlap, blurred photos, repetitive or featureless surfaces such as still water, or things that move between photos. Check the overlap map and the weak-network warning after processing, then add photos where coverage is thin.
See the Align stage's output: camera poses, sparse cloud and reprojection error
STREAM Capture is in development and not released yet. At launch it will come in the same download as its sister program STREAM Terrain; today's download contains STREAM Terrain.