Seeing around a corner without moving is a trick of physics and computation. For an autonomous vehicle, detecting a person or cyclist hidden behind a parked truck could mean the difference between a safe stop and a collision. Several research groups have demonstrated imaging systems that use lidar or ordinary cameras to reconstruct hidden scenes. The technology exists in the lab. It has not yet reached production cars.
The core principle is indirect illumination. A laser pulse or sunlight bounces off a visible wall, then strikes a concealed object, and some of that light reflects back to a detector. By measuring the time the light takes to travel, or by analyzing shifting shadow patterns, a computer can infer the hidden object's position and sometimes its shape. The challenge is that the indirect signal is extremely weak and must be extracted from noise.
As of December 2023, no automaker or autonomous vehicle operator has announced a production system that uses non-line-of-sight imaging for safety. The technology remains in the research phase, with Stanford University and the MIT Media Lab publishing the most influential results.
The Physics of Seeing Around Corners
Non-line-of-sight imaging relies on light that takes an indirect path. In a typical lidar system, a laser emits a pulse that travels directly to a target and back. In a corner-seeing system, the laser pulse hits a visible surface such as a wall or a parked car, scatters off it, and then illuminates the hidden area. A small fraction of that scattered light reaches the concealed object, reflects again, and eventually returns to the detector.
The round-trip travel time of the light carries information about the distance to the hidden object. But the signal is orders of magnitude weaker than a direct reflection. Detecting it requires sensors sensitive to individual photons. Single-photon avalanche diode detectors, known as SPADs, are the enabling hardware. These devices can register a single photon of light and timestamp its arrival with picosecond precision.
In 2018, the Stanford University Computational Imaging Lab published a paper in Nature demonstrating a lidar system that used a SPAD detector to reconstruct hidden objects. The system fired a pulsed laser at a wall and collected the faint returns. From those measurements, the researchers reconstructed a mannequin hidden around a corner. The reconstruction was not real time and required minutes of computation, but it proved the concept.
Two Approaches: Lidar and Shadows
There are two broad approaches to non-line-of-sight imaging. The first uses active illumination: a laser and a sensitive photon detector. This is the Stanford approach. The second uses passive ambient light and ordinary cameras, analyzing shadows or subtle changes in brightness to infer hidden objects.
In 2017, the MIT Media Lab's Camera Culture group demonstrated a system that used shadows to detect hidden objects around corners. The setup required no laser. An ordinary video camera watched a wall while a hidden person moved behind a barrier. The camera captured subtle changes in the shadow pattern on the wall. An algorithm reconstructed the hidden person's position and motion from those changes. The system worked at a distance of about one meter and could detect a person moving at walking speed.
The passive approach is cheaper and simpler than a lidar-based system. It does not require specialized laser hardware or photon detectors. However, it is limited to scenes with good ambient light and works only over short distances. For a vehicle traveling at highway speed, the active lidar approach is more promising because it can operate at night and over longer ranges.
The Computational Burden
Why Reconstruction Is Hard
Reconstructing a hidden scene from indirect light is a hard inverse problem. The measurements are noisy and sparse. The algorithms must estimate where the light traveled and what surfaces it struck. Early demonstrations from around 2010 to 2012 at the MIT Media Lab used femtosecond lasers and streak cameras to capture light in flight. Those experiments required massive computation and could not run in real time.
The Stanford Confocal Method
The Stanford 2018 system used a technique called confocal non-line-of-sight imaging. The laser and detector are aligned so that they focus on the same spot on the visible wall. By scanning that spot across the wall, the system builds up a set of time-of-flight measurements. A reconstruction algorithm then solves for the hidden geometry. The process took minutes for a single scene.
The Real-Time Barrier
For an autonomous vehicle, reconstruction must happen in milliseconds. The vehicle needs to know whether a pedestrian is about to step into the road, not after the pedestrian has crossed. Researchers have made progress on faster algorithms, but real-time reconstruction at highway speeds has not been demonstrated in a published, peer-reviewed system as of December 2023. The computational requirements remain a barrier to deployment.
Range and Speed Limits of Current Prototypes
How Far the Signal Reaches
The effective range of non-line-of-sight lidar prototypes is short. The Stanford 2018 system reconstructed objects at a distance of about one meter from the visible wall. The MIT shadow-based system worked at similar ranges. For a vehicle to detect a hidden pedestrian 20 meters ahead, the light must travel from the vehicle to a visible wall, bounce to the pedestrian, and return. The signal loss over that path is extreme.
Eye Safety and Signal Strength
Higher laser power could increase range, but automotive lidar systems are subject to eye-safety limits. A laser powerful enough to see around a corner at 50 meters could damage human eyes. Researchers are exploring alternative approaches, such as using multiple laser pulses and averaging the results, but the signal-to-noise ratio remains a fundamental challenge.
Speed Versus Reaction Time
Speed is another limitation. A pedestrian moves at about 1.4 meters per second. A car moving at 30 miles per hour covers 13.4 meters per second. The system must detect, reconstruct, and react in a fraction of a second. Current prototypes require seconds or minutes of computation. No published system has demonstrated real-time detection of a moving hidden object at the speed and range required for highway safety.
Who Is Working on This Technology
The Academic Leaders
The two leading academic groups are the Stanford University Computational Imaging Lab and the MIT Media Lab's Camera Culture group. Their published papers in Nature and other journals set the state of the art. Several companies have commercial interest in the underlying hardware.
Lidar Manufacturers
Velodyne Lidar, which went public via a SPAC merger in July 2021 at a valuation of about $1.8 billion, is a major supplier of automotive lidar. Its sensors are used in autonomous vehicle fleets. However, Velodyne has not announced a non-line-of-sight product. Luminar Technologies, which went public via a SPAC merger in December 2020, also supplies lidar to automakers. Luminar's lidar uses a 1550 nanometer wavelength laser, which is eye-safe at higher power than the lasers used by many competitors. That wavelength advantage could help with range, but Luminar has not demonstrated corner-seeing in a production system.
The Mobileye Factor
Intel's Mobileye subsidiary announced in 2021 that it would develop its own in-house lidar by 2025. Mobileye is a major supplier of driver-assistance systems. Its lidar plans could eventually include non-line-of-sight capabilities, but no details have been published.
How It Fits Into a Vehicle's Sensor Suite
The Existing Sensor Trio
Conventional autonomous vehicle sensor suites include radar, cameras, and standard lidar. Radar detects objects at long range and works in rain and fog, but has low angular resolution. Cameras provide rich visual information but struggle in low light and glare. Standard lidar provides accurate 3D point clouds but only for objects in direct line of sight.
A Fourth Capability
Non-line-of-sight imaging would add a fourth capability: detecting objects that are physically occluded. It would not replace radar or cameras. It would supplement them, providing an early warning that a hidden pedestrian or cyclist is about to become visible. The system would need to be tightly integrated with the vehicle's perception stack, feeding data to the path planning module.
Detection Versus Reconstruction
The difference between detecting a hidden object's presence and fully reconstructing its shape is important. A simple detection might be enough to slow the vehicle. A full reconstruction of shape and trajectory would be needed for evasive maneuvers. Current research systems aim for full reconstruction, but for a production vehicle, a simple presence-or-absence signal might be the first practical application. That simpler goal is closer to commercialization, but as of December 2023, no automaker has announced plans to ship it.
Key Facts
- First demonstrations: MIT Media Lab, around 2010-2012, using femtosecond lasers and streak cameras
- Stanford paper: Published in Nature in 2018, using lidar with single-photon avalanche diode (SPAD) detector
- MIT shadow system: Demonstrated in 2017, using ordinary video cameras and shadow analysis
- Velodyne Lidar: Went public via SPAC merger in July 2021, valuation about $1.8 billion
- Luminar Technologies: Went public via SPAC merger in December 2020
- Intel Mobileye: Announced in-house lidar development by 2025 in 2021
- Key hardware: Single-photon avalanche diode (SPAD) detectors
