Steering Angle-Guided Multimodal Fusion Lane Detection for Autonomous Driving

Yan Gong, Xinyu Zhang#, Jianli Lu, Xinmin Jiang, Zichen Wang, Hao Liu, Zhiwei Li, Li Wang, Qingshan Yang, Xingang Wu

published on IEEE Transactions on Intelligent Transportation Systems

Code ｜ Cite

• We are the first work to introduce steering angle in-formation into lane detection, which adopts LIA-CAM to fuse steering angle features with image features, allowing the network to exploit the implicit relationship between the two to assist lane detection in adverse lighting environments.

TCLaneNet: Task-Conditioned Lane Detection Network Driven by Vibration Information

Yan Gong, Xinmin Jiang, Lu Wang#, Lisheng Xu, Jianli Lu, Hao Liu, Lei Lin, Xinyu Zhang

published on IEEE Transactions on Intelligent Vehicles

Code ｜ Cite

• To deal with occlusion and low-light conditions in the lane detection, we first propose to utilize the vibration signals generated when vehicles pass over the vibration marking lines as supervision for lane occlusion prediction, whose features are then used to adaptively adjust the weights of lane detection network to improve its performance.

SIFDriveNet: Speed and Image Fusion for Driving Behavior Classification Network

Yan Gong*, Jianli Lu*; Wenzhuo Liu*; Zhiwei Li#; Xinmin Jiang; Xin Gao; Xingang Wu

published on IEEE Transactions on Computational Social Systems

Code ｜ Cite

• We first introduce a 2D image with detailed roadside information and convert speeds into a 2D spectrogram using short-time Fourier transform (STFT) to represent their time-frequency characteristics, unifying the data space of image and speed information.
• A tensor fusion method based on weight decomposition is proposed to fully fuse the vectors of the two modalities, achieving leading performance on UAH-DriveSet and distracted driving multimodal dataset.

SkipcrossNets: Adaptive Skip-cross Fusion for Road Detection

Yan Gong, Xinyu Zhang#, Hao Liu, Xinming Jiang, Zhiwei Li, Xin Gao, Lei Lin, Dafeng Jin, Jun Li, Huaping Liu

published on Automotive Innovation

Cite

• To address the challenges of pedestrian detection under insufficient nighttime illumination, we propose a Feature Aggregation Module (FAM) that adaptively captures cross-channel and cross-dimensional information interactions between two modes.
• The proposed FAM module is embedded into a dual-stream network adapted from YOLOv5. The advantages of FANet are its small size (15 MB) and fast processing speed (8 ms per frame).

A Feature Aggregation Network for Multispectral Pedestrian Detection

Yan Gong, Lu Wang#, Lisheng Xu

published on Applied Intelligence

Code ｜ Cite

• To address the challenges of pedestrian detection under insufficient nighttime illumination, we propose a Feature Aggregation Module (FAM) that adaptively captures cross-channel and cross-dimensional information interactions between two modes.
• The proposed FAM module is embedded into a dual-stream network adapted from YOLOv5. The advantages of FANet are its small size (15 MB) and fast processing speed (8 ms per frame).

Multi-Modal Fusion Technology Based on Vehicle Information: A Survey

Xinyu Zhang, Yan Gong#, Jianli Lu, Jiayi Wu, Zhiwei Li, Dafeng Jin, Jun Li

published on IEEE Transactions on Intelligent Vehicles

Cite

• To address the challenges of pedestrian detection under insufficient nighttime illumination, we propose a Feature Aggregation Module (FAM) that adaptively captures cross-channel and cross-dimensional information interactions between two modes.
• The proposed FAM module is embedded into a dual-stream network adapted from YOLOv5. The advantages of FANet are its small size (15 MB) and fast processing speed (8 ms per frame).

Oblique Convolution: A Novel Convolution Idea for Redefining Lane Detection

Xinyu Zhang, Yan Gong#, Jianli Lu, Zhiwei Li, Shixiang Li, Shu Wang, Wenzhuo Liu, Li Wang, Jun Li

published on IEEE Transactions on Intelligent Vehicles

Code ｜ Cite

• To address the challenges of pedestrian detection under insufficient nighttime illumination, we propose a Feature Aggregation Module (FAM) that adaptively captures cross-channel and cross-dimensional information interactions between two modes.
• The proposed FAM module is embedded into a dual-stream network adapted from YOLOv5. The advantages of FANet are its small size (15 MB) and fast processing speed (8 ms per frame).

GLMDriveNet: Global–local Multimodal Fusion Driving Behavior Classification Network

Wenzhuo Liu, Yan Gong, Guoying Zhang#, Jianli Lu, Yunlai Zhou, Junbin Liao

published on Engineering Applications of Artificial Intelligence

Code ｜ Cite

• We propose a Global-Local Multimodal Fusion Driving Behavior Classification Network (GLMDriveNet), which accurately classifies driver behavior into normal driving, aggressive driving, and fatigued driving, achieving state-of-the-art performance on the public UAH dataset.
• A Global-Local Interaction Channel Attention Module (GLI-CAM) is introduced to extract effective features from roadside images and vehicle speed spectrograms, while a Multi-Scale Feature Representation Fusion Module (MS-FRFM) integrates high-scale and low-scale information, assigning varying importance to different modalities to enhance the network’s focus on useful information.

OpenMPD: An Open Multimodal Perception Dataset for Autonomous Driving

Xinyu Zhang*, Zhiwei Li#*, Yan Gong*, Dafeng Jin, Jun Li, Li Wang, Yanzhang Zhu, Huaping Liu

published on IEEE Transactions on Vehicular Technology

Cite

• A multimodal perception benchmark for challenging examples is presented for the first time. Compared to existing datasets, OpenMPD places greater emphasis on complex urban scenarios, such as overexposed or dark environments, crowded areas, unstructured roads, and intersections.
• Our data collection vehicle is equipped with 6 cameras and 4 LiDARs, capturing multimodal data with a 360-degree view. We also utilize a 128-beam LiDAR to provide high-resolution point clouds for a better understanding of the 3D environment and sensor fusion. More details can be found on http://www.openmpd.com/.

Just Ask One More Time! Self-Agreement Improves Reasoning of Language Models in (Almost) All Scenarios

Lei Lin*, Jiayi Fu*, Pengli Liu, Qingyang Li, Yan Gong*, Junchen Wan, Fuzheng Zhang, Zhongyuan Wang, Di Zhang, Kun Gai

published on The 62nd Annual Meeting of the Association for Computational Linguistics

Cite

• To address the issues of repetitiveness and local optimality caused by naive greedy decoding in chain-of-thought (CoT) prompting, we propose a general ensemble optimization method. This method is applicable to nearly all types of input problem formats and scenarios where the reasoning path answers may be known or unknown.
• Our method demonstrates outstanding performance and exceptional generalization capabilities across six public reasoning benchmarks.

Multi-modal attention guided real-time lane detection

Xinyu Zhang#*, Yan Gong*, Zhiwei Li, Xuan Liu, Shuyue Pan and Jun Li

published on IEEE International Conference on Advanced Robotics and Mechatronics

Cite

• We propose an effective real-time model for lane detection, using a fusion strategy to compensate for the limitations of single mode detection and applying multi-frame input to solve the practical problems such as vehicles obstruction and mark degradation.

SparseDet: A Simple and Effective Framework for Fully Sparse LiDAR-based 3D Object Detection

Lin Liu, Ziying Song, Qiming Xia, Feiyang Jia, Caiyan Jia, Lei Yang, Yan Gong, Hongyu Pan

published on IEEE Transactions on Geoscience and Remote Sensing

Cite

• SparseDet introduces a novel approach to LiDAR-based sparse 3D object detection by using sparse queries as object proxies, enhancing contextual information aggregation through its Local Multi-scale Feature Aggregation (LMFA) and Global Feature Aggregation (GFA) modules.
• Experiments show that SparseDet outperforms the previous best sparse detector, VoxelNeXt, by 2.2% mAP and achieves a faster inference speed of 13.5 FPS on nuScenes, while also surpassing the classical FSDV2 method in both accuracy and speed.

Tvg-reid: Transformer-based vehicle-graph re-identification

Zhiwei Li, Xinyu Zhang#, Chi Tian, Xin Gao, Yan Gong, Jiani Wu, Guoying Zhang, Jun Li, and Huaping Liu

published on IEEE Transactions on Intelligent Vehicles

Cite

• We propose a backbone network for vehicle re-identification that leverages CNN and Transformer feature extraction advantages, achieving superior performance by extracting detailed image features, while a vehicle knowledge graph transfer network enhances information correlation across different vehicle types.