Production-Grade Neural Camera–LiDAR Calibration

Designed, trained, and deployed camera–LiDAR calibration that reduced rotation error by 95% on an unseen vehicle

PRODUCTION ML · AUTONOMOUS SYSTEMS · END-TO-END OWNERSHIP

I designed, trained, and deployed a 6-DoF camera–LiDAR calibration network with dual RGB/LiDAR encoders, camera-conditioned FiLM, cross-modal attention, and geometry-aware losses. On an unseen vehicle, the deployed model reduced rotation error by 95% (2.95° → 0.14°) and reprojection error by 89% (22.4 → 2.4 px). As sole technical owner, I also built its data pipeline, distributed training, and ONNX → TensorRT → onboard C++ integration.

95%less rotation error · unseen vehicle
6-DoFlearned residual correction
89%less reprojection error · unseen vehicle
~13×training acceleration
01 · CHALLENGECalibration drift breaks sensor agreement

Even a small camera–LiDAR offset corrupts projection geometry and propagates error into downstream perception.

02 · APPROACHDesign the network around sensor geometry

Dual encoders, camera-conditioned FiLM, and learned cross-modal correspondence feed separate rotation and translation heads.

03 · OUTCOMEImprove calibration on unseen data

The deployed model reduced rotation error by 95% and reprojection error by 89% on an unseen vehicle.

Problem & operating context

Camera–LiDAR misalignment degrades the geometric consistency required by downstream perception. Manual calibration is costly to repeat across a production fleet and difficult to monitor continuously. This system turns calibration into an automated, observable onboard capability: it selects trustworthy sensor observations, estimates an extrinsic correction, aggregates evidence across frames, evaluates calibration health, and produces an updated sensor configuration.

System architecture

Fleet sensor logs→ Synchronization & quality gates→ RGB–LiDAR fusion→ 6-DoF correction→ Robust aggregation→ Sensor-rig update
01
GEOMETRIC FORMULATION

Learn a correction—not an absolute pose

The calibration task is formulated as residual estimation on the rigid-motion group. An imperfect initialization renders the LiDAR observation; supervision asks the network to predict the transformation that returns that observation to the true camera geometry.

THE GEOMETRIC CONTRACT

Each camera–LiDAR extrinsic is represented as a homogeneous transform. The current estimate renders LiDAR evidence into the camera frame; the network learns the correction, not an absolute pose:

\[ T=\begin{bmatrix}R&t\\0&1\end{bmatrix}\in SE(3),\qquad R\in SO(3) \] \[ R^{\mathsf T}R=I,\qquad \det(R)=1 \] \[ p_C^{\mathrm{init}}=T_{\mathrm{init}}p_L,\qquad \tilde{u}\sim K\,p_C^{\mathrm{init}} \] \[ \begin{aligned} \Delta T_{\mathrm{gt}}&=T_{\mathrm{gt}}T_{\mathrm{init}}^{-1}\\ T_{\mathrm{final}}&=\Delta T_{\mathrm{pred}}T_{\mathrm{init}}\approx T_{\mathrm{gt}} \end{aligned} \]
ROTATION\[\Delta R_{\mathrm{gt}}=R_{\mathrm{gt}}R_{\mathrm{init}}^{\mathsf T}\]
TRANSLATION\[\Delta t_{\mathrm{gt}}=t_{\mathrm{gt}}-\Delta R_{\mathrm{gt}}t_{\mathrm{init}}\]

Training perturbs the initialization while holding the true sensor geometry fixed. Every perturbation therefore creates a new projected depth image and a mathematically consistent SE(3) correction target.

02
MODEL ARCHITECTURE

Fuse appearance with 3D structure

I designed and trained a dual-branch network that consumes camera RGB, a rasterized LiDAR inverse-depth/intensity representation, and camera identity. Its architecture learns cross-modal correspondence while giving rotation and translation different paths through the fused geometry:

LCCNet architecture: camera and LiDAR preprocessing, dual encoders, camera FiLM, cross-attention, rotation and translation heads, and SE(3) correction composition
Figure 1. LCCNet predicts a residual rigid transform from camera appearance and pose-conditioned LiDAR projection. Local cross-modal correspondence supports translation while coarse global geometry supports rotation.
  • Modality-specific encoders. An ImageNet-pretrained ResNet-18 extracts RGB appearance, while an independent GroupNorm LiDAR branch encodes projected inverse depth and intensity before fusion.
  • Camera-conditioned FiLM. Learned feature modulation carries camera identity into the network, allowing one model to serve different camera viewpoints and projection geometries.
  • Cross-modal attention. Bidirectional RGB↔LiDAR attention at 1/8-scale features learns local appearance-to-geometry relationships that support correspondence-sensitive pose estimation.
  • Separate pose heads. Coarse/global features support rotation; a local-plus-context path supports translation. The network predicts a normalized quaternion and translation vector for a full 6-DoF residual.
  • Geometry-aware learning. Rotation and robust translation objectives are coupled with 3D point consistency, tying the predicted correction to the alignment it produces in physical space.
Figure 2. A fully synthetic qualitative visualization of the recovery contract. The middle row applies a coherent rigid perturbation to the LiDAR projection; the predicted result restores the structured scan pattern toward the reference alignment. No proprietary data, vehicle imagery, or hardware is shown.
03
DISTRIBUTED TRAINING RUNTIME

Scale the experiment—not the operational risk

I treated training throughput as an end-to-end systems problem, not a model-only optimization. The original pipeline spent most of its roughly 60-hour run time repeatedly executing CPU-heavy point-cloud geometry, projection, and perturbation work. I moved reusable computation out of the hot path and repackaged the dataset into sequential TAR shards for streaming access, reducing a representative single-accelerator run to approximately 20 hours. I then built a world-size-configurable PyTorch DistributedDataParallel runtime that brought the same experiment class below 5 hours—an approximately 13× reduction in iteration time from the initial baseline.

EXPERIMENT TURNAROUNDRemove host-side waste, then scale the optimized path
≈13×end-to-end reduction
01 · INITIAL PIPELINE≈60 hoursCPU-heavy geometry, projection, and perturbation repeatedly executed online
02 · DATA-PATH OPTIMIZATION≈20 hoursreusable computation precomputed · sequential TAR-sharded streaming
03 · DISTRIBUTED RUNTIME<5 hoursworld-size-configurable DDP · deterministic shard ownership
04
DATA & GEOMETRY

Convert raw fleet logs into trusted pairs

The training dataset contains approximately 100K camera–LiDAR frames, with RGB rectified to shared virtual-camera intrinsics and LiDAR rendered as inverse depth plus intensity. The data platform is shared with LiLiNet; camera-specific synchronization and projection determine which observations are useful for this model.

  • Freeze the rig used for each label, using either the recording-time configuration or an explicitly revised calibration. Internal geometric consistency checks reject corrupted configurations before they enter training.
  • Pair camera and LiDAR observations on sensor timestamps. Per-log measurements account for different trigger behavior across compute platforms and track drift within a recording.
  • Retain the field-of-view margin needed for perturbed-prior reprojection, filter insufficient projected point support, and preserve frame-time motion metadata.
  • Deduplicate views using pose changes and select frames before downloading. Camera downloads were approximately halved in the measured collection batch.
  • Preserve validation/test partitions when adding training data; replace recollected versions of a session explicitly so old and new labels do not silently mix.

The shared collector was rebuilt after measuring a drifting publication delay of approximately 126 ms. In one measured vehicle, camera matching windows narrowed from ±100 ms to ±4 ms without frame loss; in a separate drifting recording, usable-frame coverage rose from 8% to 100%. These measurements describe the tested logs and motivate sensor-clock matching across the platform.

05
AUTOMATED DATA COLLECTION

One collection platform for both calibration systems

I designed and implemented a shared Airflow / Kubernetes collection platform for camera–LiDAR and LiDAR–LiDAR learning, then worked with infrastructure and operations teams to review and launch it in two regions. A single streaming read of each drive log produces both datasets, with one independently retryable Pod per log.

The October collection streamed 17+ TB of raw drive logs through single-pass extraction, producing ~600 GB of camera–LiDAR and LiDAR–LiDAR calibration data. These are shared collection totals for both models; curated training-set sizes are reported separately for each model.

Shared calibration pipeline: stream 17+ TB of raw logs once to produce approximately 600 GB of data for both calibration models, then select training samples
Shared data platform. One log read serves both models. Sensor timing, calibration-label provenance, and sample selection connect collection to reproducible model iteration.
RAW LOGS STREAMED17+ TBsingle-pass extraction
PAIRED DATA PRODUCED~600 GBcombined output for both models
COLLECTION IMAGE20 GB → 600 MBself-contained runtime
  • Orchestration and recovery. A scanner discovers uncollected logs and triggers bounded batches. Access and write-permission checks run before collection; failed logs retry independently, and output manifests report actual stored results. Production runs are triggered on demand; daily scanning is implemented and available.
  • Sensor-clock synchronization. Header-only measurements estimate each log’s trigger offsets and timing jitter. Segmented offsets follow drift during a recording; delayed matching aligns sensor data, CAN, and poses at the frame timestamp rather than at publication or write time.
  • Traceable geometric labels. Each sequence records the calibration rig and its source, clock selection, timing measurements, and quality thresholds. I also recalibrated fleet vehicles and used the revised rigs for recollection or explicit relabeling.
  • Select useful views before downloading. Pose-aware deduplication limits repeated viewpoints and per-session dominance while retaining collected source data in object storage. Calibration labels and train/validation assignments remain traceable through dataset indexes.

Timing measurements across 16 fleet logs also exposed stale camera pairing, incorrect timing gates, and missing-sensor failure paths in the onboard calibration stack. I documented five issues and proposed fixes for the responsible onboard team.

06
EVALUATION

Connect geometric learning to runtime behavior

The deployed model improved both calibration accuracy on an unseen vehicle and fleet performance:

Evaluation Rotation error Reprojection error
Unseen vehicle 2.95° → 0.14° (−95%) 22.4 → 2.4 px (−89%)
Fleet evaluation −44% −50%

The unseen-vehicle result uses the August deployed model. Fleet results use the current deployed model against nominal extrinsics, with 57% lower translation error as well. Export and runtime checks verify preprocessing, numerical parity, and pose composition before onboard integration.

October model iterations and offline comparison The **1002 production release** was retrained after the new collector expanded the training dataset by **16%**. I exported the selected model to ONNX, converted it to TensorRT, and deployed it through the existing onboard interface.
Version / experiment Rotation error Translation error Status and measurement scope
Previous production model 0.49° 4.2 cm Reference on 2,824 validation frames
1002 release 0.41° 3.4 cm Deployed; same 2,824-frame validation comparison
October candidate 0.42° → 0.31° — Separate same-vehicle, different-session comparison; selected checkpoint, not yet exported or deployed
The release comparison measures **offline validation improvement over the previous production model**; the fleet figures above compare the current model against nominal extrinsics. Training and validation contain different sessions from the same vehicle pool. The newer candidate also incorporates expanded data and revised labels, so its improvement reflects the combined update.

Data-quality iteration. A port of the onboard depth-distribution gate is implemented and pending merge and collection-image activation. In the measured data, it rejects 7.9% of frames; those frames carried 2.2× the rotation error of the retained group. Training with the filtered data is in progress. The collection speed limit is still being evaluated.

07
PRODUCTION SYSTEM

Ship the same geometry into C++

I productionized the model through PyTorch → ONNX → TensorRT → C++ and integrated it into the onboard autonomy stack. The runtime includes:

  • Camera–LiDAR timestamp synchronization and point-cloud motion compensation.
  • Shared virtual-camera preprocessing across training and deployment.
  • Deterministic depth/intensity rasterization and camera-conditioned inference.
  • Iterative correction and robust multi-frame MAD outlier rejection.
  • Per-camera calibration health classification and operational diagnostics.
  • Safe relative/absolute sensor-pose updates and complete sensor-rig generation.
  • Hardware-compatible TensorRT engine selection, inference timing, and recoverable result logging.

Engineering and research depth

Beyond headline accuracy, I built a diagnostic framework for understanding model behavior under domain shift. Controlled ablations and geometric probes isolated bias-dominated error, synthetic-border shortcuts, augmentation-distribution mismatch, and pose-attractor behavior. These findings guided data gates, model selection, and deployment decisions while eliminating expensive but ineffective directions such as naive resolution scaling and variance-only multi-frame aggregation.

Technology

Python · C++ · PyTorch · PyTorch DistributedDataParallel · NCCL · Distributed Sampling · TAR-Sharded Streaming · Geometry Precomputation · Global Metric Reduction · OpenCV · ONNX · TensorRT · CUDA · Protobuf · Docker · Airflow · Kubernetes · Sensor-Clock Synchronization · Data Provenance · Sensor Fusion · 3D Geometry

COMPANION SYSTEMLiLiNet · Neural LiDAR–LiDAR CalibrationView case study →

Selected details are summarized at a high level to respect the confidentiality of production systems and fleet operations.