Key takeaways
- ZTR's core innovation is a streaming reconstruction pipeline that converts monocular or stereo video feeds into navigable 3D scene representations in real time — without requiring pre-captured depth data or structure-from-motion post-processing on a separate workstation.
- Heat-signature overlays are integrated at the rendering layer rather than composited as a separate image pass, allowing thermal and optical data to share a unified depth-aware scene graph and produce geometrically consistent overlays at any viewing angle.
- Night-vision mode in ZTR is not a post-processing filter applied to the output image — it is a separate low-light-optimised reconstruction path with its own feature detection and depth estimation parameters, tuned for the noise characteristics of low-photon sensor data.
- The rendering pipeline was designed from the outset for constrained compute environments — forward operating bases, field research stations, and vehicle-mounted systems — and does not require dedicated GPU infrastructure beyond what is available in ruggedised edge hardware.
The problem: live video is flat, but operational environments are not
Video feeds from cameras deployed in defence, exploration, and field research contexts carry spatial information that a flat display discards. A vehicle-mounted camera watching a building perimeter captures angular parallax across frames that encodes the three-dimensional geometry of the scene. A drone feed over an unfamiliar terrain contains depth cues that a trained observer reads intuitively but that are lost when the feed is passed through as a 2D stream to a remote operator. The cost of discarding that information is that operators make decisions about a three-dimensional world from a representation that collapses it into two dimensions — introducing the class of spatial reasoning errors that 3D situational awareness is designed to prevent.
ZTR — short for Zero-Latency Terrain Reconstruction — is Botron Dynamics' answer to this problem. It is a rendering pipeline that takes live video as its input and produces an interactive 3D scene as its output, in real time, on hardware that can be deployed in the same field environments where the cameras operate. This whitepaper describes the architecture of that pipeline: how it extracts depth from video, how it manages the scene representation under streaming update conditions, how heat-signature and night-vision modalities are integrated, and what the rendering pipeline does to deliver interactive frame rates on constrained hardware.
"A flat video feed collapses a three-dimensional operational environment into two dimensions. ZTR reconstructs what the display discards — in real time, on hardware that goes where the cameras go."
Depth extraction from live video
The depth extraction stage is the foundation of the ZTR pipeline. Every downstream component — scene reconstruction, overlay integration, rendering — depends on a continuous, low-latency stream of per-frame depth estimates derived from the incoming video. The architecture supports three depth estimation modes, selected based on the available camera hardware and the latency and accuracy requirements of the deployment context.
In stereo mode, ZTR processes synchronised left and right camera feeds through a disparity estimation network that produces dense depth maps at the input frame rate. Stereo depth is the highest-accuracy mode and is the default for static installations and vehicle-mounted rigs where a calibrated stereo pair can be maintained. In monocular mode — used when only a single camera feed is available — ZTR uses a learned monocular depth estimator trained on a dataset spanning the scene types encountered in defence and field research deployments: structured and unstructured terrain, built environments, low-light and thermally cluttered scenes. Monocular depth is inherently scale-ambiguous, and ZTR resolves the scale ambiguity using sensor fusion with available inertial and GPS data when present, or through cross-frame scale consistency constraints when it is not. The third mode — LiDAR-assisted — uses sparse LiDAR returns to anchor the depth estimate from the video stream, producing dense depth maps with absolute scale accuracy in environments where LiDAR-equipped platforms are deployed.
Scene representation under streaming update conditions
A 3D scene reconstructed from a live video stream is not a static model that is built once and rendered repeatedly. It is a continuously updated representation that must incorporate new depth observations as each frame arrives, propagate camera pose estimates as the viewpoint moves, and discard or revise geometry that is inconsistent with recent observations — all without introducing the frame-to-frame discontinuities that would make the scene unusable for real-time navigation and operator decision making.
ZTR represents the reconstructed scene as a volumetric signed distance field maintained on a sparse voxel grid. New depth observations are integrated into the SDF using a truncated signed distance function update that weights incoming measurements by estimated confidence — a function of depth estimation mode, disparity magnitude, and the angular incidence of the observation on the reconstructed surface. Geometry that has not been observed for a configurable time window is marked stale and progressively de-weighted, allowing the scene representation to age out geometry from regions no longer in the camera's field of view without producing hard boundaries in the rendered output.
Why a volumetric SDF rather than a point cloud or mesh
Point clouds accumulate observations directly but produce representations that are difficult to render at interactive frame rates without significant post-processing, and that do not support the smooth surface interpolation needed for overlay integration. Meshes produced by reconstruction pipelines require explicit topology management that becomes expensive under high-frequency streaming updates. A volumetric SDF supports smooth surface extraction at any desired resolution via marching cubes or equivalent, integrates new observations in constant time per voxel regardless of total scene complexity, and provides a natural representation for the occupancy queries used in collision avoidance and path planning components that consume ZTR's output downstream.
Heat-signature overlays: depth-aware thermal integration
Thermal imaging is operationally critical in a range of ZTR deployment contexts — personnel detection in low-visibility environments, equipment temperature monitoring in field research, and thermal anomaly detection in infrastructure inspection. The naive approach to combining thermal and optical data is to composite the thermal image over the optical image as a 2D overlay, keyed by pixel position in the output frame. This approach breaks the moment the operator's viewpoint deviates from the original camera angle: the thermal overlay drifts relative to the geometry it is supposed to annotate because it was composited in screen space rather than registered to the 3D scene.
ZTR integrates thermal data into the scene graph rather than the output image. The thermal camera's intrinsic parameters and its extrinsic relationship to the optical camera are calibrated at deployment time, and incoming thermal frames are projected into the volumetric scene representation using the same depth-registered projection applied to optical frames. Each scene voxel carries both optical appearance attributes and a thermal signature value derived from the integrated thermal observations. When the scene is rendered from any viewpoint, the thermal overlay is computed from the scene's thermal attributes in 3D and projected into the output image — remaining geometrically consistent with the reconstructed geometry regardless of the operator's viewing angle.
| Modality | Integration point | Scene representation | Viewpoint consistency |
|---|---|---|---|
| Optical (RGB) | Depth-registered projection into SDF voxel grid | Per-voxel colour and confidence attributes | Full — rendered from scene geometry at any angle |
| Thermal (heat-signature) | Calibrated projection into scene graph, same pipeline as optical | Per-voxel thermal signature value, blended with optical | Full — thermal overlay rendered from 3D scene, not composited in 2D |
| Night-vision (low-light) | Separate low-light reconstruction path; output merged into shared SDF | Separate depth and appearance estimates merged into unified scene | Full — reconstructed geometry is viewpoint-independent |
| LiDAR (depth anchor) | Sparse depth prior injected into depth estimation before SDF update | Absolute-scale depth constraint propagated through SDF | Full — scale-anchored geometry consistent across all views |
Night-vision mode: a separate reconstruction path, not a filter
Low-light video presents a qualitatively different signal to a depth estimation and scene reconstruction pipeline than daylight optical imagery. Sensor noise is higher, fine-grained texture that depth estimation networks rely on for correspondence matching is absent or degraded, and the photometric consistency assumptions that underlie most learned depth estimators break down in scenes illuminated by mixed artificial sources with strong shadows and specular reflections. Applying a night-vision filter to the output of a pipeline tuned for daylight imagery does not address these problems — it changes the appearance of the rendered output while leaving the underlying depth estimates as degraded as they would be without it.
ZTR implements night-vision mode as a separate reconstruction path with its own depth estimation network and feature detection parameters. The low-light depth estimator was trained on a dataset of night and low-illumination sequences with ground truth depth from LiDAR-equipped capture rigs, covering the range of artificial illumination conditions — infrared illuminators, vehicle headlights, starlight, and total darkness with thermal-only imaging — encountered in defence and field research deployments. The feature detection parameters are tuned for the noise characteristics of low-photon CMOS sensors rather than for the high-SNR characteristics assumed by standard optical flow and correspondence matching implementations. The output of the low-light reconstruction path is merged into the same shared scene representation as the daylight path, allowing the system to transition between modes without scene discontinuity as illumination conditions change.
"Night-vision in ZTR is not a colour grade applied to the rendered frame. It is a separate reconstruction path trained on the noise characteristics of low-photon sensors — the geometry it produces is as accurate as the daylight path, in the dark."
The rendering pipeline: interactive frame rates on edge hardware
The ZTR rendering pipeline is designed for deployment on ruggedised edge computing hardware — systems that can be carried into a forward operating base, mounted in a vehicle, or installed in a field research station — not for the dedicated GPU infrastructure of a rear-echelon visualisation centre. The target hardware envelope for ZTR is a ruggedised single-board computer or compact embedded system with a mid-range GPU or equivalent neural processing unit, with a power budget consistent with battery or vehicle power operation.
Achieving interactive frame rates within this constraint required a tiered rendering architecture. The primary rendering pass operates on a coarse voxel resolution that supports full scene coverage at low computational cost. A detail enhancement pass is applied selectively to regions of the scene within the operator's current focus area — determined by viewing direction and proximity — using a higher-resolution local surface extraction. The overlay integration stage — combining optical, thermal, and any active annotation layers — is implemented as a single unified shader pass that avoids the multiple render-target switches that would otherwise dominate the GPU's time budget on the target hardware class. The result is a pipeline that maintains the 30 frames per second minimum interactive update rate across all depth estimation modes and modality combinations validated during field testing, with end-to-end latency from frame ingestion to scene output below 80 milliseconds.
Tiered rendering: why coarse-first is correct for operational visualisation
Operational use of a 3D situational awareness tool is not the same as inspection of a photogrammetric model. An operator using ZTR to monitor a perimeter or navigate unfamiliar terrain needs high-fidelity geometry in the region they are actively looking at, and is not harmed by coarser geometry in peripheral regions they are not currently evaluating. Tiered rendering allocates GPU budget to where the operator's attention is directed, rather than distributing it uniformly across a scene where most of the detail is outside the current decision focus. The detail enhancement pass is fast enough that it completes within the same frame interval as the coarse pass on the target hardware, meaning the operator never sees the coarse intermediate state in normal operation.
Deployment contexts: defence, exploration, and field research
ZTR was designed for three primary deployment contexts, each of which shaped a distinct set of requirements that the architecture had to satisfy simultaneously rather than sequentially. Defence deployments drove the requirements for low-latency night-vision reconstruction, heat-signature overlay accuracy at arbitrary viewing angles, and the ability to operate on vehicle-mounted or man-portable hardware without external network connectivity. Exploration deployments — unmanned surface vehicles, aerial survey platforms, and remotely operated vehicles — drove the requirements for monocular depth estimation in environments where a stereo rig cannot be maintained, and for the scene representation to remain coherent as the camera platform moves continuously through an environment with no prior map.
Field research deployments drove the requirements for multi-modality integration and the accuracy of the thermal overlay registration, which in research contexts must be sufficient to support measurement and analysis rather than merely visualisation. A field ecologist using ZTR to monitor animal heat signatures in dense vegetation needs thermal overlays that are registered to the reconstructed geometry accurately enough to associate detections with specific locations in the scene — a requirement that is more demanding than the positional accuracy needed for operator situational awareness in defence applications. The architecture's decision to integrate thermal data into the scene graph rather than compositing it in 2D was driven directly by this requirement.
What ZTR demonstrates about real-time 3D from video
The architectural choices described in this whitepaper — volumetric SDF scene representation, depth-aware thermal and night-vision integration, tiered rendering for constrained hardware — are not independent optimisations applied to a conventional video-to-3D pipeline. They are a set of mutually reinforcing decisions that make it possible to satisfy the latency, accuracy, and hardware constraints of field-deployed operational visualisation simultaneously. A pipeline that integrates thermal in 2D is simpler to implement but cannot satisfy the overlay accuracy requirements of research deployments. A pipeline that targets dedicated GPU infrastructure can use a simpler rendering architecture but cannot be deployed where ZTR is needed.
ZTR is part of Botron Dynamics' broader visualisation and situational awareness work, drawing on the same depth estimation and scene representation foundations that underlie our digital twin infrastructure and our autonomous systems perception stack. The reconstruction and rendering components described here are available as an integrated platform for defence, exploration, and field research operators, with deployment and integration support from the Botron engineering team.