LiLiNet: Production Neural LiDAR–LiDAR Calibration

Built and deployed LiDAR–LiDAR calibration from scratch, reducing fleet rotation error by 63%

PRODUCTION ML · 3D GEOMETRY · END-TO-END OWNERSHIP

I built a LiDAR–LiDAR calibration network from scratch with shared GroupNorm spherical encoders, circular azimuth correlation, and sensor-conditioned rotation/translation heads. The network reduced mean rotation error by 83% (4.86° → 0.83°) in testing. I deployed it onboard, reducing fleet rotation error by 63% (0.93° → 0.35°). As sole technical owner, I also built its data pipeline, distributed training, model export, and C++ integration.

63%less fleet rotation error
83%less rotation error · testing
96%evaluated frames improved
~20Ktraining frames
01 · CHALLENGEOne rig, multiple LiDAR viewpoints

Small rotational offsets fragment the unified point cloud and quietly degrade the geometry consumed by downstream perception.

02 · APPROACHMake displacement explicit

Project both clouds into a shared spherical representation, then expose angular misalignment through circular azimuth correlation.

03 · OUTCOMEReduce fleet calibration error

Onboard deployment reduced rotation error by 63%, from 0.93° to 0.35°, with 96% of evaluated frames improved.

Problem & operating context

Cross-LiDAR misalignment reduces the geometric consistency of the unified point cloud consumed by downstream perception. Repeating manual calibration for every sensor rig is expensive, while silent drift and incorrect initialization are difficult to diagnose operationally. LiLiNet turns this into an observable production capability: begin with an imperfect extrinsic estimate, select trustworthy synchronized observations, recover the relative pose, aggregate evidence across frames, classify calibration health, and generate an updated sensor rig.

System architecture

Synchronized LiDAR logs→ Initial extrinsics→ Spherical rasterization→ Azimuth correlation→ Robust aggregation→ Sensor-rig update
01
GEOMETRIC FORMULATION

Estimate a residual on SE(3)

The initial extrinsics define how the two point clouds should meet in a shared frame. LiLiNet is a local refiner around an existing rig, targeting approximately ±2–3° residual rotation. Rotation quality is measured intrinsically on SO(3). The training model includes a translation head, while production applies rotation only: blind-spot LiDAR ranging-scale bias makes the translation estimate unsuitable for rig write-back.

RESIDUAL CALIBRATION ON SE(3)

The supervision target is a residual rigid-body correction. Data diversity and geometric probes are used to distinguish image/point-cloud evidence from vehicle- or sensor-specific pose shortcuts:

\[ T=\begin{bmatrix}R&t\\0&1\end{bmatrix}\in SE(3),\qquad R\in SO(3) \] \[ \begin{aligned} \Delta T_{\mathrm{gt}}&=T_{\mathrm{gt}}T_{\mathrm{init}}^{-1}\\ T_{\mathrm{final}}&=\Delta T_{\mathrm{pred}}T_{\mathrm{init}}\approx T_{\mathrm{gt}} \end{aligned} \]

Rotation is evaluated by the geodesic distance on SO(3), not by element-wise matrix error:

\[ d_{SO(3)}(R_1,R_2)=\cos^{-1}\!\left(\frac{\operatorname{tr}(R_1^{\mathsf T}R_2)-1}{2}\right) \]
02
MODEL ARCHITECTURE

Make angular displacement explicit

I designed LiLiNet around the angular structure of cross-LiDAR misalignment. Each synchronized point cloud becomes a full-azimuth 256 × 1536 spherical raster with range statistics, density, intensity, and overlap channels. Shared features and explicit correlation then expose displacement before the pose heads estimate a correction:

LiLiNet architecture: paired LiDAR inputs, spherical rasterization, shared encoders, circular azimuth correlation, pose heads, robust aggregation, and sensor-rig update
Figure 1. LiLiNet converts both point clouds into a shared full-azimuth representation, exposes rotational displacement through circular correlation, and estimates a residual pose from local and global geometry.
  • Shared spherical encoders. ResNet-18-style encoders with GroupNorm use shared weights to place main- and blind-spot-LiDAR observations in a comparable feature space.
  • Circular azimuth correlation. Wrap-around cross-correlation makes angular displacement explicit across the 360° raster, providing the rotation branch with a matching signal.
  • Task-specific geometry. The rotation head combines correlation with global context. The translation branch combines local 1/8-scale and coarse 1/32-scale features to preserve both correspondence detail and broader geometry.
  • Sensor-conditioned pose heads. Learned sensor-position conditioning captures heterogeneous viewpoints; normalized-quaternion and translation outputs parameterize the residual transform.
  • Geometric objectives and training. SO(3) rotation, robust translation, trimmed point consistency, and yaw/tilt auxiliary objectives supervise alignment. EMA, learned loss weighting, and distributed training support optimization.
Figure 2. A fully synthetic LiDAR–LiDAR alignment visualization. The mis-calibrated input exposes the rigid-pose error as duplicated walls, vehicles, curbs, and poles; predicted calibration collapses the two colored clouds back toward the reference geometry. No proprietary point clouds, environments, or sensor hardware is shown.
03
DISTRIBUTED TRAINING RUNTIME

Treat scale, evaluation, and recovery as one system

LiLiNet hardens multi-accelerator training beyond basic gradient synchronization. Its PyTorch DistributedDataParallel / NCCL runtime combines vehicle-aware weighted sampling, deterministic per-rank execution, duplicate-free distributed validation, globally reduced metrics, optional synchronized batch normalization, and rank-coordinated operational controls. Strict resume contracts preserve optimizer, scheduler, learned-loss, raw-model, and EMA state so interrupted or extended experiments continue without silently changing model semantics. Multi-accelerator execution increased experimental capacity, while end-to-end profiling localized the remaining throughput ceiling to CPU-side point-cloud loading, spherical rasterization, and nearest-neighbor correspondence construction.

PRODUCTION-HARDENED TRAINING SYSTEMDeterministic scale-out with a profiled data path
PROFILEDend-to-end throughput
01 · LAUNCHRank-aware executiontorchrun · NCCL/Gloo backend · process-local seeding
02 · SAMPLEDistribution-aware shardsvehicle-weighted training · no-padding evaluation
03 · OPTIMIZESynchronized model stateDDP gradients · optional SyncBatchNorm · learned-loss sync
04 · RECOVERStrict experiment continuityglobal metrics · raw/EMA checkpoints · coordinated stop control
04
DATA & GEOMETRY

Build trustworthy supervision

The training dataset contains approximately 20K paired LiDAR frames, matching blind-spot observations with the main LiDAR. Data quality is central: stable vehicle/sensor pose priors can become shortcuts, while ranging-scale errors can contaminate geometric labels.

  • Apply sensor-clock pairing and frame-time pose association; reject excessive translation or yaw over the pairing interval.
  • Collect at ≤0.5 m/s with steering and motion limits because this collection path does not perform motion compensation.
  • Select views by speed bin and pose change, with a 600-row per-session cap to limit repetitive data and session dominance.
  • A measured selection batch shrank from 171K to 13K frames, freeing 404 GB of local storage while retaining the collected data in object storage for future selection rules.
  • Audit point-cloud overlays and rig consistency before admitting revised labels. Numerical alignment scores support diagnosis, with visual point-cloud review providing the final geometric check.
05
AUTOMATED DATA COLLECTION

One collection platform for both calibration systems

I designed and implemented a shared Airflow / Kubernetes collection platform for camera–LiDAR and LiDAR–LiDAR learning, then worked with infrastructure and operations teams to review and launch it in two regions. A single streaming read of each drive log produces both datasets, with one independently retryable Pod per log.

The October collection streamed 17+ TB of raw drive logs through single-pass extraction, producing ~600 GB of camera–LiDAR and LiDAR–LiDAR calibration data. These are shared collection totals for both models; curated training-set sizes are reported separately for each model.

Shared calibration pipeline: stream 17+ TB of raw logs once to produce approximately 600 GB of data for both calibration models, then select training samples
Shared data platform. One log read serves both models. Sensor timing, calibration-label provenance, and sample selection connect collection to reproducible model iteration.
RAW LOGS STREAMED17+ TBsingle-pass extraction
PAIRED DATA PRODUCED~600 GBcombined output for both models
COLLECTION IMAGE20 GB → 600 MBself-contained runtime
  • Orchestration and recovery. A scanner discovers uncollected logs and triggers bounded batches. Access and write-permission checks run before collection; failed logs retry independently, and output manifests report actual stored results. Production runs are triggered on demand; daily scanning is implemented and available.
  • Sensor-clock synchronization. Header-only measurements estimate each log’s trigger offsets and timing jitter. Segmented offsets follow drift during a recording; delayed matching aligns sensor data, CAN, and poses at the frame timestamp rather than at publication or write time.
  • Traceable geometric labels. Each sequence records the calibration rig and its source, clock selection, timing measurements, and quality thresholds. I also recalibrated fleet vehicles and used the revised rigs for recollection or explicit relabeling.
  • Select useful views before downloading. Pose-aware deduplication limits repeated viewpoints and per-session dominance while retaining collected source data in object storage. Calibration labels and train/validation assignments remain traceable through dataset indexes.

Timing measurements across 16 fleet logs also exposed stale camera pairing, incorrect timing gates, and missing-sensor failure paths in the onboard calibration stack. I documented five issues and proposed fixes for the responsible onboard team.

06
EVALUATION

Reduce rotation error in testing and deployment

The deployed model reduced fleet rotation error by 63% (0.93° → 0.35°) and improved 96% of evaluated frames. Testing also demonstrated recovery from larger initial misalignment:

Evaluation Rotation error Error reduction
Deployed model · fleet evaluation 0.93° → 0.35° 63%
Larger-misalignment testing 4.86° → 0.83° 83%

Fleet figures evaluate the deployed model; the larger-misalignment result comes from a separate stress-test version.

Next-model evaluation An October candidate reduced rotation error by **76% (0.96° → 0.23°)** in a separate evaluation. It uses expanded data and revised calibration labels and has not been deployed. Label reliability and LiDAR ranging-scale errors remain under investigation.
07
PRODUCTION SYSTEM

Carry the geometry contract onto the vehicle

I productionized the full path through PyTorch → ONNX → TensorRT → C++ and integrated it into the onboard autonomy stack. The runtime includes:

  • Stable tensor contracts and numerical parity tooling across Python, ONNX, TensorRT, and C++ preprocessing.
  • Hardware-aware TensorRT engine selection across onboard compute variants, with fail-closed compatibility and shape checks.
  • Automatic module startup, synchronized keyframe collection, vehicle-motion gating, and geometric BEV-quality screening.
  • Independent per-sensor caches, multi-frame inference, quaternion-aware averaging, and MAD outlier rejection.
  • Calibration-health classification, operational telemetry, parity dumps, recoverable diagnostics, and complete sensor-rig generation.
  • Rotation-only production updates that preserve established translations and handle incompatible sensor-frame conventions explicitly.

Engineering and research depth

I combined network architecture, geometric learning, and controlled model diagnostics to determine why a calibration model succeeds or fails. Controlled probes quantified a projection-mask shortcut, showed that residual error was dominated by stable cross-domain bias rather than frame noise, and demonstrated why simply increasing aggregation or model complexity could not solve new-domain behavior. These findings drove the data gates, noise curriculum, checkpoint-selection protocol, initialization strategy, and deployment decisions while retiring expensive approaches that did not improve the real operating point.

The evaluation protocol also separates session-held-out accuracy from whole-vehicle generalization instead of conflating them—a critical distinction for production ML systems with limited ground-truth calibration data.

Technology

Python · C++ · PyTorch · PyTorch DistributedDataParallel · NCCL · Distributed Sampling · Distributed Evaluation · SyncBatchNorm · EMA Checkpointing · 3D Geometry · Point Clouds · ONNX · TensorRT · CUDA · Eigen · Protobuf · Bazel · Airflow · Kubernetes · Sensor-Clock Synchronization · Data Curation · Sensor Fusion

COMPANION SYSTEMLCCNet · Neural Camera–LiDAR CalibrationView case study →

Selected details are summarized at a high level to respect the confidentiality of production systems and fleet operations.