At the 11th IEEE International Smart Cities Conference (ISC2 2025), held in Patras, Greece, AviSense.AI presented the paper: “A Holistic Perception System of Internal and External Monitoring for Ground Autonomous Vehicles: The Au…

At the 11th IEEE International Smart Cities Conference (ISC2 2025), held in Patras, Greece, AviSense.AI presented the paper: “A Holistic Perception System of Internal and External Monitoring for Ground Autonomous Vehicles: The AutoTRUST Paradigm.” Developed under the EU-funded Horizon Europe project AutoTRUST, this work introduces a fully integrated perception and interaction architecture for autonomous electric vehicles, combining multi-modal sensing, deep unrolling for LiDAR enhancement, and edge-deployed language models for explainable, human-centric automation.

External Perception Layer AviSense designed a LiDAR-based deep unrolling super-resolution module that upgrades sparse 16-channel LiDAR scans into dense 64-channel equivalents, enabling high-fidelity semantic segmentation from cost-efficient sensors. The model operates as a ROS2 modular pipeline on NVIDIA Jetson Orin, achieving real-time performance (8–9 FPS) with TensorRT-optimized inference and full on-vehicle autonomy.

Internal Perception Layer The in-cabin system integrates multi-camera driver and occupant monitoring, sound event detection, and smart environmental sensing:

  • Driver distraction detection: MoViNet-A3 (89% accuracy).
  • Facial emotion analysis: ResEmoteNet (72.9% on AffectNet-7).
  • Occupant identification: FaceNet (96.5% accuracy).
  • Acoustic monitoring: YAMNet + MLP classifier (94% macro accuracy).
  • Environmental sensing: CO₂, PM2.5, VOCs, and thermal comfort indices visualized in real time via Grafana dashboards.

Virtual Assistant Layer A quantized Llama-3-8B-Instruct large language model is deployed at the edge via NanoLLM and TVM optimization, enabling context-aware driver interaction fully offline. Integrated with NVIDIA Riva for speech recognition and Piper for neural text-to-speech, the assistant enables real-time, privacy-preserving communication inside the vehicle.

The complete system was deployed and validated on the platforms of AviSense.AI and ALKE Vehicles, demonstrating real-time performance, energy efficiency, and explainable operation across both internal and external perception layers.

Full paper: https://arxiv.org/abs/2508.17969

Demo video: https://www.youtube.com/watch?v=zt9rJj7tlPQ

Partners: AviSense.AI (Patras Science Park, Greece) CERTH – Information Technologies Institute (ITI) KIOS Research and Innovation Center of Excellence, University of Cyprus ALKE Electric Vehicles, Padova, Italy