RISK-CONSTRAINED SPATIO-TEMPORAL ATTENTION PPO FOR LANE-CHANGING DECISIONS IN MIXED TRAFFIC
Keywords:
Mixed traffic, Lane changing, Proximal policy optimization, Risk-constrained reinforcement learning, Spatio-temporal attention, SUMO, Autonomous drivingAbstract
Lane-changing decisions in mixed traffic must reconcile task completion, surrounding-vehicle interaction, and safety under partial observability. A vanilla Proximal Policy Optimization (PPO) policy that flattens neighboring-vehicle features tends to obscure which vehicles are relevant to the current maneuver, while directly adding a risk penalty to the reward does not provide an explicit budget for cumulative risk. This paper develops a reproducible three-lane Eclipse SUMO environment and proposes a risk-constrained spatio-temporal attention PPO method. The method represents the ego vehicle and up to eight neighboring vehicles as traffic-agent tokens, applies multi-head attention within each frame, aggregates four historical frames with a gated recurrent unit (GRU), and introduces a separate cost critic with a Lagrangian risk budget. The experimental protocol compares vanilla PPO, attention-enabled PPO, risk-constrained PPO, and the full model under identical traffic, reward, and optimizer settings across three training seeds and 20 evaluation episodes per seed. Across the resulting 240 evaluation episodes no policy produced a recorded collision, so the policies were separated by task completion and surrogate risk rather than by collision counts. The full model reached the target lane in 76.7% of its episodes with a mean episode risk cost of 0.040, compared with 46.7% and 0.247 for vanilla PPO, at a mean speed 0.42 m/s lower; attention and the risk constraint each improved completion modestly on their own, and only their combination produced a gain whose direction was consistent across all three seeds. The results support the modular design of the method, and they also show that the simulator safety backstop, rather than the collision count, sets the limit on what this evaluation can claim about safety.References
[1] Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. DOI: 10.48550/arXiv.1707.06347.
[2] Shi T, Wang P, Cheng X, et al. Driving decision and control for automated lane change behavior based on deep reinforcement learning. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), 2019: 2895-2900. DOI: 10.1109/ITSC.2019.8917392.
[3] Li G, Yang Y, Li S, et al. Decision making of autonomous vehicles in lane change scenarios: Deep reinforcement learning approaches with risk awareness. Transportation Research Part C: Emerging Technologies, 2022, 134: 103452. DOI: 10.1016/j.trc.2021.103452.
[4] Yao X, Du Z, Sun Z, et al. Cooperative lane-changing in mixed traffic: a deep reinforcement learning approach. Transportmetrica A: Transport Science, 2024, 22(1). DOI: 10.1080/23249935.2024.2343048.
[5] Lu C, Lu H, Chen D, et al. Human-like decision making for lane change based on the cognitive map and hierarchical reinforcement learning. Transportation Research Part C: Emerging Technologies, 2023, 156: 104328. DOI: 10.1016/j.trc.2023.104328.
[6] Zhang S, Zhuang W, Li B, et al. Integration of planning and deep reinforcement learning in speed and lane change decision-making for highway autonomous driving. IEEE Transactions on Transportation Electrification, 2025, 11(1): 521-535. DOI: 10.1109/TTE.2024.3393293.
[7] Wang T, Ma M, Liang S, et al. Robust lane change decision for autonomous vehicles in mixed traffic: A safety-aware multi-agent adversarial reinforcement learning approach. Transportation Research Part C: Emerging Technologies, 2025, 172: 105005. DOI: 10.1016/j.trc.2025.105005.
[8] Deng Z, Hu W, Sun C, et al. Eliminating uncertainty of driver’s social preferences for lane change decision-making in realistic simulation environment. IEEE Transactions on Intelligent Transportation Systems, 2025, 26(2): 1583-1597. DOI: 10.1109/TITS.2024.3512784.
[9] Yang K, Tang X, Qiu S, et al. Towards robust decision-making for autonomous driving on highway. IEEE Transactions on Vehicular Technology, 2023, 72(9): 11251-11263. DOI: 10.1109/TVT.2023.3268500.
[10] Wang S, Wang Z, Jiang R, et al. A multi-agent reinforcement learning-based longitudinal and lateral control of CAVs to improve traffic efficiency in a mandatory lane change scenario. Transportation Research Part C: Emerging Technologies, 2024, 158: 104445. DOI: 10.1016/j.trc.2023.104445.
[11] Achiam J, Held D, Tamar A, et al. Constrained policy optimization. In: Advances in Neural Information Processing Systems, 2017, 30.
[12] Dai J, Ji J, Yang L, et al. Augmented proximal policy optimization for safe reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37(6): 7288-7295. DOI: 10.1609/aaai.v37i6.25888.
[13] Gao Z, Hao H, Gao F, et al. Constrained reinforcement-learning-enabled policies with augmented Lagrangian for cooperative intersection management. IEEE Internet of Things Journal, 2025, 12(5): 5396-5411. DOI: 10.1109/JIOT.2024.3487854.
[14] Zhang X, Peng Y, Luo B, et al. Model-based safe reinforcement learning with time-varying constraints: Applications to intelligent vehicles. IEEE Transactions on Industrial Electronics, 2024, 71(10): 12744-12753. DOI: 10.1109/TIE.2023.3317853.
[15] Lin K, Li Y, Chen S, et al. Motion planner with fixed-horizon constrained reinforcement learning for complex autonomous driving scenarios. IEEE Transactions on Intelligent Vehicles, 2024, 9(1): 1577-1588. DOI: 10.1109/TIV.2023.3273857.
[16] Gao F, Wang X, Fan Y, et al. Constraints driven safe reinforcement learning for autonomous driving decision-making. IEEE Access, 2024, 12: 128007-128023. DOI: 10.1109/ACCESS.2024.3454249.
[17] Zhao W, He T, Chen R, et al. State-wise safe reinforcement learning: A survey. In: Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, 2023: 6814-6822. DOI: 10.24963/ijcai.2023/763.
[18] Wachi A, Shen X, Sui Y. A survey of constraint formulations in safe reinforcement learning. In: Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, 2024: 8262-8271. DOI: 10.24963/ijcai.2024/913.
[19] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. In: Advances in Neural Information Processing Systems, 2017, 30: 5998-6008.
[20] Guo H, Keyvan-Ekbatani M, Xie K. Modeling coupled driving behavior during lane change: A multi-agent Transformer reinforcement learning approach. Transportation Research Part C: Emerging Technologies, 2024, 165: 104703. DOI: 10.1016/j.trc.2024.104703.
[21] Wang J J, Zhang Q C, Zhao D B. Highway lane change decision-making via attention-based deep reinforcement learning. IEEE/CAA Journal of Automatica Sinica, 2022, 9(3): 567-569. DOI: 10.1109/JAS.2021.1004395.
[22] Xu D, Liu P, Li H, et al. Multi-view graph convolution network reinforcement learning for CAVs cooperative control in highway mixed traffic. IEEE Transactions on Intelligent Vehicles, 2024, 9(1): 2588-2599. DOI: 10.1109/TIV.2023.3297310.
[23] Liu H, Huang Z, Mo X, et al. Augmenting reinforcement learning with transformer-based scene representation learning for decision-making of autonomous driving. IEEE Transactions on Intelligent Vehicles, 2024, 9(3): 4405-4421. DOI: 10.1109/TIV.2024.3372625.
[24] Mozaffari S, Alipour Sormoli M R, Koufos K, et al. Multimodal manoeuvre and trajectory prediction for automated driving on highways using transformer networks. IEEE Robotics and Automation Letters, 2023, 8(10): 6123-6130. DOI: 10.1109/LRA.2023.3301720.
[25] Ashfaq F, Ghoniem R M, Jhanjhi N Z, et al. Using dual attention BiLSTM to predict vehicle lane changing maneuvers on highway dataset. Systems, 2023, 11(4): 196. DOI: 10.3390/systems11040196.
[26] Cho K, van Merriënboer B, Gülçehre Ç, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014. DOI: 10.48550/arXiv.1406.1078.
[27] Guo H, Xie K, Keyvan-Ekbatani M. Modeling driver’s evasive behavior during safety-critical lane changes: Two-dimensional time-to-collision and deep reinforcement learning. Accident Analysis & Prevention, 2023, 186: 107063. DOI: 10.1016/j.aap.2023.107063.
[28] Choi J H, Kim S. Predictive risk-aware reinforcement learning for autonomous vehicles using safety potential. Electronics, 2025, 14(22): 4446. DOI: 10.3390/electronics14224446.
[29] Jiao Y, Calvert S C, van Cranenburgh S, et al. Learning collision risk proactively from naturalistic driving data at scale. Nature Machine Intelligence, 2026, 8(3): 337-350. DOI: 10.1038/s42256-026-01189-w.
[30] Miqdady T, de Oña R, Casas J, et al. Studying traffic safety during the transition period between manual driving and autonomous driving: A simulation-based approach. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(6): 6690-6710. DOI: 10.1109/TITS.2023.3241970.
[31] Schuurmans M, Katriniok A, Meissen C, et al. Safe, learning-based MPC for highway driving under lane-change uncertainty: A distributionally robust approach. Artificial Intelligence, 2023, 320: 103920. DOI: 10.1016/j.artint.2023.103920.
[32] Eclipse SUMO. TraCI documentation. 2026. https://sumo.dlr.de/docs/TraCI/.
[33] Eclipse SUMO. Change vehicle state: TraCI documentation. 2026. https://sumo.dlr.de/docs/TraCI/Change_Vehicle_State.html.
[34] Yang Z, Zheng Z, Kim J, et al. Eco-driving strategies using reinforcement learning for mixed traffic in the vicinity of signalized intersections. Transportation Research Part C: Emerging Technologies, 2024, 165: 104683. DOI: 10.1016/j.trc.2024.104683.
[35] Mamond A W, Kundroo M, Yoo S E, et al. FLDQN: Cooperative multi-agent federated reinforcement learning for solving travel time minimization problems in dynamic environments using SUMO simulation. Sensors, 2025, 25(3): 911. DOI: 10.3390/s25030911.
[36] Jo Y, Oh C. A lane change strategy to enhance traffic safety in the coexistence of autonomous vehicles and manual vehicles. Journal of Advanced Transportation, 2024, 2024(1): 6126204. DOI: 10.1155/2024/6126204.