Key takeaways
- No single sensor type covers the full range of conditions an ADAS system encounters in production — camera, lidar, and radar each have failure modes that the others compensate for.
- Time synchronisation across heterogeneous sensor streams is not a detail — desynchronisation of even a few milliseconds at highway speed produces environment models that the control system cannot safely act on.
- Early fusion — combining raw or near-raw sensor data before object detection runs — consistently outperforms late fusion in complex, occluded, and adverse-weather scenarios.
- The output of the fusion pipeline must be a single, confidence-weighted environment model with explicit uncertainty bounds, not a bag of sensor-specific object lists the control system must reconcile itself.
The challenge: one environment, three different pictures of it
A production ADAS system typically runs three sensor modalities simultaneously: cameras for visual classification and lane geometry, lidar for precise spatial depth, and radar for velocity measurement and performance in degraded visibility. Each modality is good at something the others are not. A camera in bright daylight produces rich classification data but degrades sharply in glare, fog, and darkness. A lidar unit produces precise point-cloud geometry but is expensive, affected by heavy rain, and cannot distinguish the material or colour of what it sees. A radar unit measures velocity with high reliability in almost all weather conditions but produces sparse spatial data and struggles to classify stationary objects.
Run these three sensors in parallel and you have three different, partially overlapping, partially contradictory pictures of the same environment arriving at slightly different times, in different coordinate frames, at different update rates. The core problem of ADAS sensor fusion is turning those three pictures into one — a unified environment model that is more accurate, more complete, and more reliable than any of the three inputs alone, delivered within the latency budget the control system requires to act safely.
"Three sensors. Three coordinate frames. Three update rates. One environment model the vehicle has to stake decisions on at 100 kilometres per hour."
The deployment context
This case study covers the sensor fusion work undertaken for a highway-capable ADAS platform operating at speeds up to 130 km/h, with a requirement to maintain reliable object detection and tracking across a defined range of adverse conditions including heavy rain, glare, twilight, and partial sensor obscuration from road debris and spray. The sensor suite comprised four forward and side-facing cameras, two long-range radar units, one short-range radar unit, and a solid-state lidar covering the forward arc.
The end-to-end latency requirement from sensor capture to a validated environment model available to the control system was 40 milliseconds. The safety requirement was that no object above a defined size threshold within a defined proximity zone could be absent from the environment model for more than two consecutive detection cycles without a degraded-mode flag being raised.
The synchronisation problem
Before fusion can begin, the data arriving from each sensor must be brought into temporal alignment. Cameras, radar units, and lidar typically operate at different frequencies — a camera might produce frames at 30 Hz, a lidar at 10 Hz, a radar at 20 Hz — and the precise timestamp of each measurement reflects the actual moment the sensor captured the data, not the moment it arrived at the compute platform. At 100 km/h, the vehicle travels approximately 28 metres per second. A desynchronisation of 50 milliseconds across sensor streams means the environment model is attempting to merge observations that were taken nearly 1.4 metres apart in vehicle position — producing ghost objects, missed detections, and velocity estimates that are meaningless.
The synchronisation layer in the Botron ADAS stack uses hardware timestamping at the sensor level, with a precision time protocol reference distributed across the sensor network. Every data packet carries a hardware-stamped capture time. The fusion pipeline interpolates each sensor's data to a common temporal reference frame before any fusion computation begins, using the vehicle's own motion model to compensate for ego-motion between capture timestamps. This step adds computational overhead but is not optional: fusion without temporal alignment is not fusion, it is noise addition.
Hardware timestamping vs. software timestamping
Software timestamps — applied when data arrives at the host compute platform — accumulate jitter from OS scheduling, bus transfer latency, and driver buffering. In a multi-sensor fusion system, that jitter is indistinguishable from real temporal offset between sensor captures, and the fusion pipeline cannot correct for it. Hardware timestamps, applied at the sensor itself at the moment of capture, eliminate this ambiguity. The difference in practice is the difference between a fusion pipeline that works on a development bench and one that holds up in a vehicle under realistic compute load.
The fusion architecture: why we chose early fusion
Sensor fusion architectures exist on a spectrum from late fusion — where each sensor runs its own full detection pipeline and the outputs are merged at the object level — to early fusion, where raw or near-raw sensor data is combined before detection runs at all. Late fusion is simpler to implement and easier to debug: each sensor's pipeline is independent, and the merge step is just a matter of reconciling object lists. It is also consistently less accurate in the scenarios that matter most.
Late fusion fails in occluded and adverse-weather scenarios because each sensor's individual pipeline has already discarded the uncertainty information that would have allowed the fusion step to correctly weight conflicting or absent detections. When a camera misses an object in rain and the radar detects it weakly, a late-fusion system sees a missing camera detection and an uncertain radar detection and may discard the radar result as noise. An early-fusion system, working with the raw data from both sensors simultaneously, can identify the correlation between the camera's degraded response in that region and the radar's weak return, and produce a detection with appropriate confidence rather than no detection at all.
| Fusion approach | How it works | Where it performs well | Where it falls short |
|---|---|---|---|
| Late fusion | Each sensor runs a full independent detection pipeline. Object lists are merged at the output stage. | Clear conditions, well-lit environments, low occlusion. Simpler to implement and validate per sensor. | Discards inter-sensor uncertainty before the merge step. Performs poorly when one or more sensors are degraded. |
| Mid fusion | Feature representations from each sensor are fused before final classification and localisation. | Better occlusion handling than late fusion. More tractable than full early fusion for heterogeneous sensor types. | Still discards raw measurement uncertainty. Requires carefully aligned feature spaces across modalities. |
| Early fusion | Raw or near-raw sensor data is combined in a shared spatial representation before any detection runs. | Adverse weather, low light, heavy occlusion, partial sensor degradation. Preserves full uncertainty information. | Higher compute and memory requirements. Requires precise calibration and temporal alignment across all sensor modalities. |
The Botron ADAS fusion stack uses an early-fusion architecture anchored on a shared voxel representation of the forward environment. Camera features, lidar point data, and radar returns are projected into a common coordinate frame and combined into a unified spatial occupancy grid before the detection and tracking networks run. The occupancy grid carries per-voxel confidence values derived from each sensor's contribution, so the downstream detection network can weight its predictions by the reliability of the underlying sensor data in each region of the scene.
Tracking across the fused environment model
Object detection on a single frame produces a list of detections with positions and classes. A control system making decisions at speed needs more than that — it needs to know which detections correspond to the same physical object across frames, what each object's velocity and heading are, and what the confidence on each of those estimates is. This is the tracking layer, which runs on top of the fused detection output.
The tracking implementation uses an extended Kalman filter bank, with one filter per tracked object, predicting each object's next state based on a motion model and updating it with each new detection that associates to that object. The association step — matching new detections to existing tracks — uses a combination of spatial proximity, class consistency, and velocity model agreement, with explicit handling of occlusion intervals so that tracks are maintained through gaps in detection rather than being dropped and re-initialised each time an object is temporarily missed.
Why track maintenance through occlusion matters
A vehicle that passes behind a lorry and re-emerges on the other side is the same vehicle. A control system that drops the track during the occlusion interval and re-initialises it on re-emergence has lost the velocity and heading history that allows it to predict where the vehicle will be in the next half-second. In a lane-change or emergency-braking scenario, that lost history is the difference between a system that correctly anticipates the re-emerging vehicle's trajectory and one that reacts to it as if it appeared from nowhere.
Running the full pipeline on automotive-grade edge hardware
A fusion architecture that works on a development workstation with a high-end GPU is not a production ADAS system. The full pipeline — synchronisation, spatial projection, early fusion, detection, tracking, and confidence estimation — must run within the 40-millisecond latency budget on the automotive-grade system-on-chip in the vehicle, which has a fraction of the compute available on the development platform, strict power and thermal constraints, and determinism requirements that GPU workloads on a development machine do not face.
The Botron stack is designed for hardware portability from the start of development. The fusion and detection pipeline is implemented in a compute-graph representation that is compiled to the target SoC's neural processing units, DSPs, and CPU cores with workload allocation tuned per platform. Model quantisation — converting floating-point weights to 8-bit integer representations — is applied with per-layer sensitivity analysis rather than uniformly, preserving precision in layers where quantisation error is highest while reducing memory and compute load across the majority of the network.
End-to-end latency is measured on target hardware from the first hardware timestamp to the point at which the validated environment model is posted to the control system's shared memory. The 40-millisecond budget is treated as a hard constraint, not a target average, and is verified across the full test scenario suite including worst-case sensor load scenarios.
"A fusion architecture that works on a development workstation is not a production system. The 40-millisecond budget is a hard constraint, not a target average."
Results in adverse-condition testing
The fused pipeline was validated against a structured adverse-condition test suite covering heavy rain at 80 mm/hr, dense fog with visibility below 50 metres, direct solar glare at low sun angle, and partial sensor obscuration from road spray. For each condition, detection recall and false negative rate were measured against the same scenario set run on a camera-only baseline.
Across the full adverse-condition suite, the early-fusion pipeline reduced false negatives by 38% relative to the camera-only baseline and maintained detection recall above the safety threshold in all tested conditions where the camera-only system fell below it. The improvement was largest in rain and fog scenarios, where radar's weather penetration provides the largest compensating advantage, and smallest in glare scenarios, where radar and lidar together provide partial but not full compensation for camera degradation in direct sunlight.
Latency on the target SoC measured 36.2 milliseconds at median and 39.1 milliseconds at the 99th percentile across the full test load, within the 40-millisecond hard constraint. The tracking layer maintained object continuity through occlusion intervals of up to 18 frames — 600 milliseconds at 30 Hz camera rate — before dropping a track, which covered the full range of occlusion scenarios in the test suite without a single false track termination.
What this demonstrates
Sensor fusion at the edge is not a feature that can be bolted onto an existing single-sensor ADAS stack. It requires temporal synchronisation infrastructure at the hardware level, a fusion architecture committed to before the detection models are trained, and a validation suite that specifically tests the adverse conditions where fusion provides its largest advantage over any individual sensor. Done correctly, it produces an environment model that is categorically more reliable under real-world conditions than any single modality can achieve — and that reliability is what the vehicle's control system requires to make safety decisions it can be held accountable for.
The stack described here is the same foundation that underlies Botron Dynamics' broader ADAS and autonomous systems work across passenger, commercial, and off-highway platforms. The principles — early fusion, hardware timestamping, hard latency constraints, and adverse-condition validation — transfer across platform sizes and sensor suite configurations, adapted to the specific requirements of each deployment context.