Key takeaways
- A model with strong benchmark accuracy can still be unsafe in production — benchmark distributions rarely match real-world driving conditions.
- Sensor fusion across camera, radar, and LIDAR resolves failure modes that no single sensor can solve alone.
- Latency and determinism matter as much as accuracy: a correct detection delivered too late is operationally a missed detection.
- Edge cases — not average-case performance — are what determine whether a perception stack is safe to deploy.
The problem nobody talks about until it's too late
Object detection for ADAS is sold internally on the strength of a benchmark score. A model hits 95% mAP on a curated dataset, the team celebrates, and the assumption is that the hard part is done. The finance and programme case usually passes comfortably at this point: the model works, the roadmap says integrate and ship.
What rarely makes it into that programme case is the gap between a benchmark dataset and the actual distribution of conditions a vehicle will encounter over a ten-year service life. In our experience building perception stacks across passenger, commercial, and off-highway platforms, the operational risk of object detection systems is consistently underestimated — often by a wide margin. The model performs well in the lab. Whether it performs well at 3am in freezing fog, with a sensor partially obscured by road spray, is a different question entirely.
"The model performs well in the lab. Whether it performs well at 3am in freezing fog, with a sensor partially obscured by road spray, is a different question entirely."
Why benchmark accuracy doesn't translate to road safety
The root cause is almost always the same: teams validate detection models against datasets that are easier than the real world, and against latency budgets that don't reflect the deployed compute platform. Each new scenario — a cyclist partially occluded by a parked van, a deer at the tree line in low light, a plastic bag blowing across three lanes — exposes a gap between what the training distribution covered and what the road actually contains. Over time, the gap between validated performance and real performance becomes the difference between a system that intervenes correctly and one that doesn't intervene at all.
Three root causes
When we audit perception stacks moving from prototype to production, we consistently find three root causes of real-world performance gaps:
| Root cause | Why it happens |
|---|---|
| Dataset distribution mismatch | Training and validation data skew toward clear weather, daylight, and common object classes. No benchmark covers everything the fleet will actually see. |
| Single-sensor blind spots | Camera-only systems struggle in glare, fog, and low contrast. Radar-only systems struggle to classify stationary objects. Few teams fuse sensors early enough to catch this before integration testing. |
| Latency treated as a software afterthought | Models are optimised for accuracy on a development GPU, then ported late to automotive-grade silicon with a fraction of the compute, where real-time constraints force last-minute compromises. |
What a production-grade detection pipeline actually means
A production ADAS object detection system is not a single model. It is a pipeline — sensor inputs, fusion logic, model inference, tracking, and a safety layer — engineered so that the system fails predictably rather than silently. The word "predictably" is important: the goal is not zero failure, which is not achievable, but failure modes that are bounded, detectable, and handled gracefully by the rest of the vehicle's safety architecture.
Definition
A perception stack is the full chain from raw sensor data to a validated list of tracked objects with class, position, velocity, and confidence — engineered so that every downstream safety decision can trust what it's given, or knows when not to.
In practice this means:
- A sensor fusion layer that combines camera, radar, and (where fitted) LIDAR data, so that the weaknesses of one modality are covered by the strengths of another rather than left exposed.
- A model deployment pipeline that includes quantisation and hardware-in-the-loop testing on the actual target silicon, not just the development workstation.
- A scenario and edge-case test suite — built from real fleet data, simulation, and adversarial scenario generation — that runs on every model candidate before it is considered for release.
- A runtime monitoring layer that tracks detection confidence, sensor health, and degraded-mode triggers, producing evidence that feeds directly into functional safety audits.
Designing for the edge cases before the first model ships
The highest-leverage moment is before the first model is frozen for integration. Teams that invest in scenario coverage and fusion architecture before model development begins consistently report fewer late-stage integration failures and a shorter path through validation. The cost of retrofitting sensor fusion or a proper scenario test suite onto a camera-only system that's already in vehicle integration is almost always higher than designing for it upfront.
We have seen programmes attempt to add radar fusion or LIDAR support after a camera-only model is already deep into homologation testing. It is not impossible, but it requires re-validation of the full pipeline, significant rework of the fusion and tracking logic, and schedule pressure that tends to compress the very testing that catches edge cases.
"Designing the fusion architecture and scenario coverage before the model is frozen saves months of rework during integration and homologation."
What to prioritise in the perception design phase
Not everything needs to be finalised before the first model trains. The minimum viable perception foundation — the decisions that are very expensive to change later — includes:
| Decision | Why it's expensive to change later |
|---|---|
| Sensor suite and placement | The number, type, and mounting position of cameras, radar units, and LIDAR. Changing this later means re-running calibration, re-collecting data, and retraining. |
| Fusion architecture | Whether fusion happens early (raw sensor data) or late (post-detection, combining outputs). This decision shapes every model and pipeline component built afterward. |
| Target compute platform | The automotive-grade SoC the model must ultimately run on, including its memory, power, and real-time constraints. Models trained without this in mind often need substantial rework to fit the latency budget. |
| Scenario and edge-case taxonomy | The catalogue of weather, lighting, occlusion, and object-class scenarios the system must handle, agreed with safety and validation teams before data collection begins. |
Measuring real-world detection performance
Perception performance is sometimes difficult to validate because the failures that matter most are rare by design — the near-misses you want to never see in the fleet. We recommend instrumenting the following metrics from the start of development so you can demonstrate readiness with evidence rather than a single benchmark number:
- Recall and precision broken out by weather, lighting, and occlusion condition — not just an aggregate score
- End-to-end latency from sensor capture to actuation decision, measured on target hardware
- False negative rate on the specific edge-case scenario suite, tracked across every model release
- Degraded-mode trigger rate — how often the system correctly recognises it cannot make a confident detection and hands control back appropriately
Programmes that build sensor fusion and scenario testing in from the start typically see false negative rates on edge-case scenarios drop substantially release over release, with far fewer surprises surfacing during track and on-road validation.
Getting started
If your programme is early in perception development and hasn't yet committed to a fusion architecture or scenario taxonomy, the best time to lock those decisions in is now, before models are trained against a single sensor's data. If integration is already underway with a camera-only system, start by cataloguing the specific scenarios where it underperforms and prioritise fusion or scenario coverage work against those, rather than attempting a full architectural rebuild.
Perception teams are not just a model-accuracy function. They are the layer that determines whether the rest of the vehicle's safety case holds up in conditions nobody tested for in the demo. The programmes we see succeeding at production ADAS are those that treat edge-case coverage as a first-class engineering deliverable, with dedicated ownership, a growing scenario library, and a commitment to testing the conditions that don't show up on a leaderboard.