Comfort Improvement for Autonomous Vehicles using Reinforcement Learning with In-situ Human Feedback
2022-01-0974
03/29/2022
- Event
- Content
- Autonomous vehicles have become a driving force to transform our existing transportation system in the near future and will help improve a lot of driving performance such as driving safety, congestion, emissions. Despite that passengers’ comfort in autonomous vehicle (AV) has been an essential topic, there are insufficient methods to significantly increase AVs’ ride comfort or adapt humans to accept AVs. Ride comfort in AVs is dramatically different from that in a traditional vehicle. Traditionally, researchers have investigated ergonomic factors such as seat vibrations and noise. However, the introduction of automated driving functions would lead researchers to shift toward vehicle control factors. In order to make AVs comfortable for humans, we need to know human’s expectations on AVs’ comfort. General approaches to model human expected driving behaviors can be categorized into analytical approaches and heuristic approaches. However, all existing approaches require humans to drive the vehicle to generate demonstrations, which can be unrealistic for autonomous vehicles. In this paper, we propose a reinforcement learning (RL) based approach to learn human expectations on autonomous vehicles without a need of human demonstrations but just pressing force information from humans. The goal of reinforcement learning is to learn policies for decision-makings by optimizing a cumulative future reward function. Different from existing reinforcement learning approaches that use some calculated criteria like safety or efficiency as the reward, we directly integrate the real-time in-situ human pressing force into the reward as a representation of human comfort. In addition, it is known that reinforcement learning may sometimes learn unrealistically high action values because it includes a maximization step over estimated action values, which tends to prefer overestimated to underestimated values. To address this, we have integrated some extra constraints and penalties in reinforcement learning in order to avoid such unrealistic actions. It is also known that human’s expected driving behaviors may not always be correct. To address this, we propose an adaptation of human to autonomy process through some human-vehicle interfaces to provide real-time feedback to human in order to correct their improper expectations. The proposed approaches are implemented and tested with human-involved experiments using real-time driving simulator. The results showed that the trained automated RL driving agent could increase the ride comfort for the corresponding participant. Simultaneously, the human participants could acquire warning information when they are having improper feedback effectively. Such information successfully helped the participants to develop a better expectation during automated driving rides.
- Citation
- Xiang, J., and Guo, L., "Comfort Improvement for Autonomous Vehicles using Reinforcement Learning with In-situ Human Feedback ," SAE Technical Paper 2022-01-0974, 2022, .