Vision-Based Occupant State Monitoring

ACTIVE DEVELOPMENT · COMPUTER VISION · SAFETY-CRITICAL PERCEPTION

I am developing an end-to-end visual perception system for per-seat occupant state understanding from fisheye cabin cameras. The current baseline jointly estimates seat occupancy and coarse torso pose, while the surrounding engineering work turns raw recording sessions into reproducible supervision, separates evaluation sessions from training data, and maintains a stable interface for downstream integration.

Because the project is still active, this page emphasizes the technical contract and engineering decisions rather than publishing provisional headline accuracy.

Activedevelopment status

2 headsoccupancy + torso pose

Session-levelheld-out evaluation

ONNXintegration interface

01 · CHALLENGEUnderstand every seat from distorted cabin views

Fisheye imagery, occlusion, posture variation, class imbalance, and temporally correlated recordings make apparently simple classification difficult to evaluate honestly.

02 · APPROACHBuild one reproducible seat-level perception contract

Version fixed regions of interest, isolate recording sessions, share visual features across seats, and couple model quality with deterministic data and evaluation checks.

03 · CURRENT PHASEStabilize labels, metrics, and integration boundaries

Expand label coverage, freeze the evaluation protocol, test failure modes across seeds, and validate the exported model contract before reporting final results.

Problem & operating context

An occupant-monitoring model must convert wide-angle cabin imagery into a stable state estimate for each defined seat. The hard part is not only recognizing people. A usable system must preserve seat identity, avoid temporal leakage between train and validation data, remain interpretable under class imbalance, and expose predictable outputs to the consuming software stack.

The project therefore treats data construction, model behavior, evaluation, and integration as one system rather than optimizing an isolated image classifier.

System architecture

Fisheye RGB sessions→ Seat-level labels→ Versioned ROI crops→ Shared visual encoder→ Multi-task heads→ Gated state output

DATA CONTRACT

Turn correlated recordings into trustworthy supervision

Defined a seat-centric label schema that keeps occupancy and torso state explicit instead of hiding application semantics inside folder names or preprocessing code.
Built deterministic sample manifests from recording sessions, with fixed seat regions and reproducible mapping from frame metadata to model inputs.
Used session-isolated splits so visually adjacent frames from the same recording burst cannot inflate held-out performance.
Designed burst-aware sampling and dataset audits to expose duplicate frames, sparse labels, class skew, missing files, and inconsistent annotations before training.
Kept preprocessing configuration versioned so the same crop geometry and normalization contract can be reproduced during evaluation and export.

MODEL ARCHITECTURE

Share perception while preserving seat-level outputs

The current visual baseline crops each configured seat region from a fisheye frame and passes every crop through a shared ResNet-18 encoder. Two task heads operate on the same representation: a binary occupancy head and a three-class torso-pose head. Deterministic output gating prevents an unoccupied seat from producing a semantically misleading pose state.

Figure 1. Current RGB baseline. A shared encoder learns reusable occupant appearance features across seat crops, while task-specific heads preserve distinct occupancy and posture objectives. The illustration is schematic and contains no proprietary imagery or cabin geometry.

For seat region (s), the model can be summarized as

SEAT-LEVEL MULTI-TASK CONTRACT

\[ h_s=f_\theta\!\left(\operatorname{Crop}(I,\mathcal{R}_s)\right),\qquad p_s^{\mathrm{occ}}=\sigma(w_o^{\mathsf T}h_s),\qquad p_s^{\mathrm{pose}}=\operatorname{softmax}(W_p h_s) \] \[ \mathcal{L}=\lambda_o\,\mathcal{L}_{\mathrm{occ}}+ \lambda_p\,\mathbb{1}[y_s^{\mathrm{occ}}=1],\mathcal{L}_{\mathrm{pose}} \]

The pose objective is meaningful only for occupied seats. Making this dependency explicit keeps the training target and downstream output semantics aligned.

TRAINING SYSTEM

Make experiments repeatable, not anecdotal

Centralized dataset, augmentation, optimizer, schedule, seed, and checkpoint configuration so experiments can be reconstructed from artifacts rather than memory.
Added deterministic execution controls and multi-seed runs to distinguish architecture changes from initialization noise.
Tracked task-specific losses and class behavior separately, preventing a strong aggregate number from masking a collapsed occupancy or posture class.
Preserved best-checkpoint selection and export metadata as part of the experiment contract.
Structured the repository around data preparation, training, evaluation, and export boundaries so changes can be audited without tracing an ad hoc notebook workflow.

EVALUATION

Test the failure modes that matter operationally

The evaluation plan goes beyond aggregate accuracy. It reports per-task confusion matrices and per-class behavior, checks whether every class is actually predicted, and separates directional occupancy errors because a missed occupant and a false occupied seat have different operational consequences.

Session-held-out evaluation, multi-seed variation, degeneration checks, and label-quality audits are treated as release gates. Final quantitative results will be published only after the dataset and evaluation contract are frozen.

LEAKAGE CONTROLSession splitrecording groups remain isolated

ROBUSTNESSMulti-seedvariance is measured explicitly

SAFETY VIEWDirectional errorsfailure costs remain visible

MODEL INTERFACE

Design for integration from the beginning

The baseline is exportable through PyTorch → ONNX with a documented tensor contract for preprocessing, seat ordering, occupancy probability, pose scores, and deterministic gating. This boundary is intentionally narrow: the learned model estimates visual state, while seat configuration, policy thresholds, and application-specific decisions remain explicit in the consuming system.

Current integration work focuses on numerical parity, stable input shapes, reproducible preprocessing, and clear failure behavior when inputs or configuration violate the expected contract.

CURRENT ROADMAP

Freeze the evidence before publishing the result

The next milestones are to expand and audit label coverage, finalize the held-out session set, complete multi-seed comparison of the visual baselines, validate upstream frame and ROI assumptions, and lock the exported interface. Once those contracts are stable, this page can graduate from an active research case study to a result-led project with defensible headline metrics.

Technology

Python · PyTorch · ResNet · OpenCV · Fisheye Vision · Multi-Task Learning · Experiment Tracking · ONNX · Reproducible Evaluation

SELECTED WORKReturn to the complete project portfolioView all projects →

Architecture and workflow are summarized at a high level. No proprietary cabin imagery, customer data, vehicle counts, hardware identifiers, or unpublished performance values are shown.