In-Cabin Occupant Perception System

Front- and rear-seat occupant monitoring built from RTMO detector adaptation, ResNet-18 multi-task learning, iToF reasoning, and temporally fused VLM attributes

INTEGRATED SYSTEM · MULTIMODAL PERCEPTION · REAL-VEHICLE EVALUATION

As technical owner, I designed and delivered front- and rear-seat occupant monitoring, beginning with RTMO person/keypoint adaptation and a ResNet-18 multi-task occupancy/posture model, then integrating iToF seating-position reasoning, VLM attributes, temporal fusion, and confidence-aware vehicle messaging. My work spans neural-model design and training, targeted detector fine-tuning, data construction, and integration with the upstream vehicle software.

99.3%occupancy F1
95.8%posture Macro-F1
89.2%seat-depth accuracy
1.4 cmstable height MAE

The system covers front and rear seats. The results section documents the task-specific protocols behind these benchmarks, alongside separate rear-seat detector-adaptation results.

01 · CHALLENGEFollow occupant state through difficult transitions

Forward leaning, occlusion, entering and leaving the vehicle, and seat changes demand reliable detection and consistent seat-level state.

02 · APPROACHTrain the visual models, then integrate their evidence

Fine-tune RTMO for cabin-specific failures, train a shared-encoder multi-task classifier, and connect their outputs to physical signals and temporal state.

03 · RESULTQuantify the system on vehicle recordings

Measure occupancy, posture, seating position, and attributes against reviewed human labels, then publish a confidence-aware vehicle message.

Problem & operating context

Occupant perception must associate a person with a seat, identify forward-leaning, upright, or reclined posture, estimate where an upright occupant is sitting, and maintain useful attributes over time. A missed person can suppress everything downstream; an old attribute estimate can become incorrect as soon as occupants change seats.

I treated data construction, model adaptation, temporal state, sensor interpretation, and vehicle integration as parts of one system. Development used AI coding agents, including collaboration with the upstream development agent, to support implementation and integration; I owned the technical decisions, model work, and evaluation.

01
DETECTOR ADAPTATION

Recover forward-leaning occupants before classifying them

RTMO-s performs one-stage person detection and predicts body keypoints; hip keypoints support seat association. The production detector’s difficult cases were occupants folded forward: some produced no candidate, while others fell below the confidence gate. I fine-tuned the detector for front and rear cabin views, using a targeted annotation and training workflow around those failure modes:

  • Selected missed detections, already-detected forward leans, and upright/reclined controls from training sessions.
  • Created and reviewed 349 cabin frames across front and rear views, with tight visible-body boxes and hip keypoints for seat association; converted the labels into COCO-compatible training data.
  • Fine-tuned RTMO-s with cabin examples mixed with COCO person data, retaining general person-detection behavior while adapting to cabin viewpoints.
  • Exported through ONNX with the existing input/output and in-graph NMS contract, then evaluated the exported model through the production seat-assignment logic.

On the detector’s reserved evaluation sessions, rear-seat forward-lean confirmation increased from 72.1% to 98.6% (106/147 → 145/147). Front-seat model detection with correct hip-based seat assignment reached 108/111; the unchanged bounding-box-height rule limited confirmed outputs to 88/111. This diagnosis separated model recovery from a remaining downstream policy constraint.

These detector-adaptation results use a separate protocol from the front-seat integrated-system results below.

02
POSTURE MODEL & DATA

Learn posture with explicit seat and label semantics

I designed and trained a shared-encoder multi-task network over fixed seat ROIs. An ImageNet-pretrained ResNet-18 trunk feeds a binary occupancy head and a three-class posture head for forward lean, upright, and reclined states. A masked objective combines occupancy BCE with posture cross-entropy only on occupied, visually judgeable samples. Inference uses explicit occupancy and image-quality gating to preserve the meaning of the outputs.

  • Built versioned annotation rules, browser-based labeling/review tools, deterministic manifests, and burst-aware sampling.
  • Isolated recording days for training and evaluation, and restricted propagated labels to training.
  • Addressed class imbalance, seat-identity shortcuts, and augmentation/inference mismatch through diagnostics and training-set BatchNorm recalibration.
  • Completed nine-session out-of-fold evaluation over 6,337 manually labeled seat observations, including 3,072 valid posture labels: pooled posture Macro-F1 0.8714, occupancy F1 0.9820. This earlier model study measures recording-day variation and has its own dataset and protocol.
03
SENSOR & TEMPORAL FUSION

Turn individual observations into a consistent occupant state

Seating position. The system combines iToF distance and seat-track position to distinguish front-edge seating from the middle/back region. It produces this estimate when the occupant is upright, with explicit confidence and validity fields.

VLM attributes. Repeated attribute readings are fused in a sliding window for the same occupant. Entry and exit reset the accumulated state, so the next occupant starts a new estimate. The evaluation tracks how age and height estimates change as observations accumulate, as well as their stable-window error.

Vehicle context. Per-seat perception is combined with seat position, backrest angle, seatbelt status, door state, and gear signals in one output schema. Each estimated field carries confidence and validity so consuming software can interpret uncertain or unavailable results.

04
SYSTEM ARCHITECTURE

Connect complementary evidence at the seat level

Occupant perception system: cabin images feed RTMO detection, seat-level posture classification and VLM attributes; iToF and seat signals support seating-position reasoning; temporal fusion and confidence rules produce one vehicle message per seat
System architecture. Person detection, posture, seating position, and attributes provide distinct evidence. Per-seat state, temporal fusion, and confidence/validity rules connect their outputs to the vehicle interface. The diagram contains no proprietary imagery or cabin geometry.
05
REAL-VEHICLE RESULTS

Measure each capability under its operating conditions

The October 16, 2026 engineering report evaluates driver and front-passenger seats from the front OMS camera, including entry/exit, seat changes, empty seats, and multiple postures. Ground truth was manually labeled after interval sampling and grouping similar consecutive frames. Cross-annotator agreement was 96% for posture and 97% for occupancy, with disagreements reviewed.

Capability Result Evaluation condition
Occupancy 98.7% accuracy · 98.5% recall · 99.3% F1 Frame-level person-detection output against occupied/empty labels
Posture 96.3% accuracy · 95.8% Macro-F1 Three classes on ground-truth occupied samples, evaluated independently of occupancy detection
Forward lean 94.7% recall · 97.3% F1 Posture classification; errors were toward upright
Seating position 89.2% accuracy Front-edge vs. middle/back; output coverage 94.5% on upright samples
VLM gender 98.0% accuracy Individual VLM replies
Height 1.4 cm MAE · 94.3% within ±4 cm Fused estimate after the observation window fills
Age 4.2 years MAE · 78.6% within ±6 years Fused estimate after the observation window fills

Temporal accumulation substantially improved the measured attribute estimates: from 1–2 readings to a full window, height within ±4 cm rose from 38.7% to 94.3%, and age within ±6 years rose from 11.3% to 78.6%. Weight remained a weaker estimate at 8.5 kg MAE, making it a target for further work with physical priors.

06
VEHICLE INTEGRATION

Deliver one interpretable message per seat

The unified message carries occupancy, posture, seating position, occupant attributes, and vehicle-provided signals. Confidence and validity accompany the estimates, while timestamps and sequence numbers support downstream consumption. The quantification report includes a recorded output example from the integrated system.

This work extended my ownership from model training and ONNX contracts into upstream integration, per-seat state management, temporal aggregation, and real-vehicle quantitative evaluation. Further iteration focuses on difficult detector cases, upright/reclined boundaries, iToF thresholds across body types, and stronger physical constraints for attribute estimation.

Technology

Python · PyTorch · RTMO · MMPose · ResNet · ONNX · Computer Vision · Keypoint Detection · iToF · VLM Integration · Temporal Fusion · Vehicle Signals · Quantitative Evaluation

SELECTED WORKReturn to the complete project portfolioView all projects →

Architecture and evaluation results are summarized at a high level. No proprietary cabin imagery, raw recordings, internal endpoints, or vehicle identifiers are included.