Skip to main content

📝 Sensor Fusion

Description​

< What is it? >​

Sensor fusion combines measurements from multiple sensors into one estimate that is more reliable than any single measurement. In an autonomous vehicle, it is used for vehicle localization, object perception, and tracking.

  • Example: an IMU provides fast short-term motion, GNSS provides an absolute but sometimes noisy position, and LiDAR map matching provides geometric corrections. A fusion system combines all three to estimate the vehicle's pose through a tunnel entrance or an urban canyon.

Key points​

< State estimation >​

For localization, the system maintains a state estimate x^t\hat{x}_t, often including position, velocity, orientation, and sensor biases. It predicts the next state from the previous estimate and motion input, then corrects it using sensor measurements:

x^t∣t−1=f(x^t−1∣t−1,ut)\hat{x}_{t\mid t-1}=f(\hat{x}_{t-1\mid t-1},u_t) x^t∣t=Update⁡(x^t∣t−1,zt)\hat{x}_{t\mid t}=\operatorname{Update}(\hat{x}_{t\mid t-1},z_t)

Here, utu_t can come from an IMU or wheel odometry, and ztz_t can be a GNSS fix, LiDAR map match, camera observation, or other measurement. A Kalman filter, extended Kalman filter (EKF), unscented Kalman filter (UKF), or factor-graph optimizer commonly performs this prediction-and-correction process.

< What information is fused? >​

Fusion levelExamplePurpose
State fusionGNSS, IMU, wheel odometry, and LiDAR pose estimatesEstimate the ego vehicle's pose and velocity
Feature fusionCamera features aligned with LiDAR pointsImprove object or lane perception
Track fusionRadar, camera, and LiDAR object tracksMaintain robust estimates of nearby objects

Fusion is not simply concatenating data. It must account for each source's coordinate frame, timing, uncertainty, and failure modes.

< Uncertainty and measurement gating >​

A good fusion system gives less weight to a measurement when it is less trustworthy. It may reject an observation that is inconsistent with the predicted state—a process called measurement gating or outlier rejection.

  • Example: if a GNSS position suddenly jumps 30 meters while LiDAR matching and IMU motion remain consistent, the localization system can flag or down-weight that GNSS observation instead of immediately moving the vehicle pose.

< Common engineering challenges >​

  • Time synchronization: a fast-moving vehicle sees different locations if camera, LiDAR, and IMU timestamps are misaligned.
  • Extrinsic calibration: the fixed transform between sensors must be accurate.
  • Correlated errors: two sensors can fail for related reasons, so treating them as independent can create unjustified confidence.
  • Graceful degradation: the system needs a defined behavior when one or more sensors become unreliable.

Comparison​

< Sensor fusion vs SLAM >​

Sensor fusionSLAM
Main goalCombine measurements into a robust state or perception estimateEstimate pose while building or updating a map
Uses multiple sensors?Often, but not alwaysCan use one sensor or several
Produces a map?Not necessarilyYes, when mapping is active
RelationshipOften supplies motion and pose estimates to localizationOften uses fusion internally for robust localization

Reference​