Production-Grade Neural Camera–LiDAR Calibration

End-to-end 6-DoF online calibration—from 500+ GB of multimodal sensor data to TensorRT deployment in an onboard C++ system

PRODUCTION ML · AUTONOMOUS SYSTEMS · END-TO-END OWNERSHIP

I owned the end-to-end development and production deployment of a learning-based system for online 6-DoF camera–LiDAR extrinsic calibration. The project spans the complete ML lifecycle: fleet-scale sensor data processing, geometric validation, multimodal model development, controlled evaluation, ONNX/TensorRT optimization, and integration into an onboard C++ runtime.

500+ GBmultimodal sensor data
50K+quality-controlled pairs
0.31°fleet validation error
<0.6°held-out vehicle error
01 · CHALLENGECalibration drift breaks sensor agreement

Even a small camera–LiDAR offset corrupts projection geometry and propagates error into downstream perception.

02 · APPROACHLearn correction from paired evidence

Fuse RGB appearance with rasterized depth and intensity while preserving camera identity and 3D geometric constraints.

03 · OUTCOMEClose the loop onboard

From quality-gated fleet logs to TensorRT inference, robust aggregation, health diagnostics, and sensor-rig generation.

Problem & operating context

Camera–LiDAR misalignment degrades the geometric consistency required by downstream perception. Manual calibration is costly to repeat across a production fleet and difficult to monitor continuously. This system turns calibration into an automated, observable onboard capability: it selects trustworthy sensor observations, estimates an extrinsic correction, aggregates evidence across frames, evaluates calibration health, and produces an updated sensor configuration.

System architecture

Fleet sensor logs→ Synchronization & quality gates→ RGB–LiDAR fusion→ 6-DoF correction→ Robust aggregation→ Sensor-rig update
01
DATA & GEOMETRY

Convert raw fleet logs into trusted pairs

  • Built a scalable pipeline processing 500+ GB of camera, LiDAR, vehicle-state, and odometry logs into synchronized, reproducible training and evaluation data.
  • Implemented temporal alignment, LiDAR motion compensation, virtual-camera rectification, deterministic point selection, and session-level dataset partitioning.
  • Added automated gates for motion state, illumination, exposure, blur, point density, depth distribution, and calibration consistency.
  • Detected and rejected stale sensor configurations with errors approaching 10°, preventing corrupted geometry from entering training and deployment evaluation.
  • Eliminated pose-dependent field-of-view leakage and aligned the Python training pipeline with the production C++ preprocessing contract.
02
GEOMETRIC FORMULATION

Learn a correction—not an absolute pose

The calibration task is formulated as residual estimation on the rigid-motion group. An imperfect initialization renders the LiDAR observation; supervision asks the network to predict the transformation that returns that observation to the true camera geometry.

THE GEOMETRIC CONTRACT

Each camera–LiDAR extrinsic is represented as a homogeneous transform. The current estimate renders LiDAR evidence into the camera frame; the network learns the correction, not an absolute pose:

\[ T=\begin{bmatrix}R&t\\0&1\end{bmatrix}\in SE(3),\qquad R\in SO(3) \] \[ R^{\mathsf T}R=I,\qquad \det(R)=1 \] \[ p_C^{\mathrm{init}}=T_{\mathrm{init}}p_L,\qquad \tilde{u}\sim K\,p_C^{\mathrm{init}} \] \[ \begin{aligned} \Delta T_{\mathrm{gt}}&=T_{\mathrm{gt}}T_{\mathrm{init}}^{-1}\\ T_{\mathrm{final}}&=\Delta T_{\mathrm{pred}}T_{\mathrm{init}}\approx T_{\mathrm{gt}} \end{aligned} \]
ROTATION\[\Delta R_{\mathrm{gt}}=R_{\mathrm{gt}}R_{\mathrm{init}}^{\mathsf T}\]
TRANSLATION\[\Delta t_{\mathrm{gt}}=t_{\mathrm{gt}}-\Delta R_{\mathrm{gt}}t_{\mathrm{init}}\]

Training perturbs the initialization while holding the true sensor geometry fixed. Every perturbation therefore creates a new projected depth image and a mathematically consistent SE(3) correction target.

03
MODEL ARCHITECTURE

Fuse appearance with 3D structure

The network consumes camera RGB, a rasterized LiDAR inverse-depth/intensity representation, and camera identity. Its design separates global rotational reasoning from the local correspondence needed for translation:

LCCNet architecture: camera and LiDAR preprocessing, dual encoders, camera FiLM, cross-attention, rotation and translation heads, and SE(3) correction composition
Figure 1. LCCNet predicts a residual rigid transform from camera appearance and pose-conditioned LiDAR projection. Local cross-modal correspondence supports translation while coarse global geometry supports rotation.
  • ImageNet-pretrained ResNet-18 RGB encoder and an independent depth + intensity LiDAR branch with GroupNorm.
  • Camera-conditioned FiLM modulation, enabling one network to model multiple camera viewpoints without duplicating the backbone.
  • Bidirectional RGB↔LiDAR cross-attention at higher spatial resolution for local geometric correspondence.
  • A coarse/global rotation branch and a local-plus-context translation branch for full 6-DoF correction.
  • Quaternion normalization, geodesic rotation, robust translation, and 3D point-consistency objectives that supervise the transform in both parameter and geometric space.
Synthetic LCCNet qualitative comparison: the predicted calibration aligns the projected LiDAR scan pattern with the camera scene, the mis-calibrated input shows a coherent rotational offset, and the reference alignment restores the geometry
Figure 2. A fully synthetic qualitative visualization of the recovery contract. The middle row applies a coherent rigid perturbation to the LiDAR projection; the predicted result restores the structured scan pattern toward the reference alignment. No proprietary data, vehicle imagery, or hardware is shown.
04
EVALUATION

Separate capture range from operating accuracy

CONTROLLED PERTURBATIONup to 10°rotational disturbance
CORRECTED RESIDUAL≈0.5°controlled recovery
SUB-1°recovery envelope

Controlled perturbation evaluation demonstrated recovery from rotational disturbances of up to 10° to approximately 0.5° residual error. This capture range is reported separately from operating-point accuracy: the system reached 0.31° / 3.33 cm on fleet validation and demonstrated sub-degree cross-vehicle generalization, including below 0.6° rotation error on held-out vehicle evaluation.

FLEET VALIDATION0.31°rotation error
TRANSLATION3.33 cmvalidation error
HELD-OUT VEHICLE<0.6°rotation error
05
PRODUCTION SYSTEM

Ship the same geometry into C++

I productionized the model through PyTorch → ONNX → TensorRT → C++ and integrated it into the onboard autonomy stack. The runtime includes:

  • Camera–LiDAR timestamp synchronization and point-cloud motion compensation.
  • Shared virtual-camera preprocessing across training and deployment.
  • Deterministic depth/intensity rasterization and camera-conditioned inference.
  • Iterative correction and robust multi-frame MAD outlier rejection.
  • Per-camera calibration health classification and operational diagnostics.
  • Safe relative/absolute sensor-pose updates and complete sensor-rig generation.
  • Hardware-compatible TensorRT engine selection, inference timing, and recoverable result logging.

Engineering and research depth

Beyond headline accuracy, I built a diagnostic framework for understanding model behavior under domain shift. Controlled ablations and geometric probes isolated bias-dominated error, synthetic-border shortcuts, augmentation-distribution mismatch, and pose-attractor behavior. These findings guided data gates, model selection, and deployment decisions while eliminating expensive but ineffective directions such as naive resolution scaling and variance-only multi-frame aggregation.

Technology

Python · C++ · PyTorch · OpenCV · ONNX · TensorRT · CUDA · Distributed Training · Protobuf · Docker · Sensor Fusion · 3D Geometry

COMPANION SYSTEMLiLiNet · Neural LiDAR–LiDAR CalibrationView case study →

Selected details are summarized at a high level to respect the confidentiality of production systems and fleet operations.