Design and Implementation of Multi-Sensor Perception System for Autonomous Vehicle

Abstract

This paper provides an in-depth exploration of a design and implementation of multi-sensor perception system for autonomous vehicle (AV). The proposed system integrates a host laptop runs the CARLA simulator to produce a configurable urban scene with traffic, pedestrians, and virtual sensors, namely six RGB Cameras and a LiDAR (Light Detection and Ranging). The sensor streams are sent via ROS2 middleware, which communicates over TCP/IP to an NVIDIA Jetson AGX Xavier, enabling online perception with reliable communication. Image data are processed with YOLOv8 (You Only Look Once) for object detection and the results are shown as a mosaic that merges the six camera views. LiDAR point clouds are processed using a PointPillars-based pipeline and visualized in a bird’s-eye-view (BEV) format. Experimental results provide real-time visualizations of detected objects and LiDAR spatial structures, supporting the feasibility of the architecture for simulation-based autonomous vehicle research and rapid prototyping of embedded perception systems. Critically, this work demonstrates real-time multi-sensor autonomous vehicle perception on affordable edge hardware, making the technology accessible to startups and research institutions. This represents a fundamental shift in autonomous vehicle deployment and removing traditional barriers.