A THREE-CHANNEL NEURAL NETWORK DETECTION FRAMEWORK INTEGRATING SCENE, OBJECT, AND BEHAVIOR ANALYSIS
Keywords:
Three-channel neural network, Scene understanding, Object detection, Behavior analysis, Multimodal fusion, Cross-attention mechanism, Visual intelligenceAbstract
Three neural channels running side by side are the core of the detection framework presented here, and they let scene context, object recognition and behavior analysis be read together instead of one after another. Conventional pipelines either keep the three apart or chain them in a fixed order, and in both cases the dependencies among environmental context, object semantics and agent behavior drop out. Our pathways are the Scene Context Channel (SCC), which performs hierarchical spatial feature extraction; the Object Recognition Channel (ORC), built around a multiscale attention mechanism; and the Behavior Analysis Channel (BAC), which handles temporal dynamic modeling. A cross-attention fusion module joins them, learning adaptive weights that govern how much each channel draws on the others. Static images and video sequences are both accepted, and the three analysis dimensions are inferred jointly in real time. For validation we ran the method over several benchmark datasets covering surveillance, autonomous driving and human-computer interaction. Set against single-channel baselines and current multi-task architectures, it improves markedly on joint “scene-object-behavior” recognition and reaches state-of-the-art accuracy. Generalization holds across a range of environmental conditions, and the margin widens precisely where one would predict, namely in ambiguous scenes that no single modality can resolve. The ablations support the three-channel design as well, since each channel supplies information the others lack, and together they raise overall detection accuracy by 12-18% relative to isolated processing.References
[1] Yitzhak N, M E Y, Simon K, et al. Hippocampal sharp-wave ripples linked to visual episodic recollection in humans. Science (New York, N.Y.), 2019, 365(6454): eaax1030. DOI: 10.1126/science.aax1030.
[2] Haley P S, Surinach A D, Nietz K A, et al. Cortex-wide characterization of decision-making neural dynamics during spatial navigation. Nature Communications, 2026, 17(1): 4482. DOI: 10.1038/s41467-026-71074-y.
[3] Qian X, Gao S, Deng W, et al. Improving Oriented Object Detection by Scene Classification and Task-Aligned Focal Loss. Mathematics, 2024, 12(9). DOI: 10.3390/math12091343.
[4] Darrell T. Introduction to the CVIU special issue on “Parts and Attributes: Mid-level representation for object recognition, scene classification, and object detection”. Computer Vision and Image Understanding, 2015, 138: 85. DOI: 10.1016/j.cviu.2015.07.001.
[5] Vaca-Castano G, Das S, Sousa P J, et al. Improved Scene Identification and Object Detection on Egocentric Vision of Daily Activities. Computer Vision and Image Understanding, 2016, 156: 92-103. DOI: 10.1016/j.cviu.2016.10.016.
[6] Cao J, Hou G, Lv J, et al. MFDSGC: A Novel Pedestrian Crossing Intention Prediction Method for Autonomous Vehicles. Advanced Engineering Informatics, 2026, 74(PD): 104810. DOI: 10.1016/j.aei.2026.104810.
[7] Bovesi A, Calabretto A, Stroppa A, et al. The autonomous vehicle dilemma: passenger(s) versus pedestrian(s). Behavior & Information Technology, 2025, 44(6): 1155-1168. DOI: 10.1080/0144929x.2024.2433037.
[8] Ningbo C, Liying Z, Qiaowen B. MPC-Based Method for Intersection Control in Mixed Traffic Environments with Autonomous Vehicles, Human-Driven Vehicles, and Pedestrians. Journal of Transportation Engineering, Part A: Systems, 2025, 151(3). DOI: 10.1061/jtepbs.teeng-8371.
[9] Subramanian K, Thomas L, Sahin M, et al. Supporting Human–Robot Interaction in Manufacturing with Augmented Reality and Effective Human–Computer Interaction: A Review and Framework. Machines, 2024, 12(10): 706. DOI: 10.3390/machines12100706.
[10] Tim W, Alvika G, Jacob T, et al. Confronting barriers to human-robot cooperation: balancing efficiency and risk in machine behavior. iScience, 2021, 24(1). DOI: 10.1016/j.isci.2020.101963.
[11] Santocildes G A, Vazquez I J, Eguiluz A. Enhancing Robot Behavior with EEG, Reinforcement Learning, and Beyond: A Review of Techniques in Collaborative Robotics. Applied Sciences, 2024, 14(14): 6345. DOI: 10.3390/app14146345.
[12] Talha M A, Junejo N I. Dynamic Scene Understanding Using Temporal Association Rules. Image and Vision Computing, 2014, 32(12): 1102-1116. DOI: 10.1016/j.imavis.2014.08.010.
[13] Fooladgar F, Kasaei S. A survey on indoor RGB-D semantic segmentation: from hand-crafted features to deep convolutional neural networks. Multimedia Tools and Applications: An International Journal, 2020, 79(1): 4499-4524. DOI: 10.1007/s11042-019-7684-3.
[14] Xie G, Shi B, Su Y, et al. Research on the Vanishing Point Detection Method Based on an Improved Lightweight AlexNet Network for Narrow Waterway Scenarios. Journal of Marine Science and Engineering, 2024, 12(5). DOI: 10.3390/jmse12050765.
[15] Mohsen M, Safar M, Mozhgan A, et al. Scenario-Based Discrimination of Common Grapevine Varieties Using In-Field Hyperspectral Data in Western Iran. International Journal of Applied Earth Observation and Geoinformation, 2019, 80: 26-37. DOI: 10.1016/j.jag.2019.04.002.
[16] Ma X Y, Chen S, Tan H K. Experimental study of steel beam-column joints with novel slot-bolted connections under a central-column-removal scenario. Journal of Constructional Steel Research, 2024, 214: 108481. DOI: 10.1016/j.jcsr.2024.108481.
[17] Raju N S A, Geetha G, Subashini B, et al. HYDRA-XAI dual-backbone disaster scene recognition using ResNet50-Swin transformer feature fusion, explainable evidence, and an operational recommender. Scientific Reports, 2026. DOI: 10.1038/s41598-026-59683-5.
[18] Jiang Y, Zeng Y, Tang J, et al. A multi-view vision transformer integrated with progressive transfer learning for accurate identification of prenatal fetal coarctation of the aorta. Engineering Applications of Artificial Intelligence, 2026, 181(P6): 115654. DOI: 10.1016/j.engappai.2026.115654.
[19] Sathish R, Raja M D S, Arul E. A quality-gated hybrid Viola–Jones pipeline for efficient face detection under computational constraints. Journal of Real-Time Image Processing, 2026, 23(3): 101. DOI: 10.1007/s11554-026-01878-0.
[20] Kamera S, Challa D, Bwalya H, et al. A web-enabled real-time multimodal emotion detection system integrating BERT text embeddings and CNN-based speech and facial models. Signal, Image and Video Processing, 2026, 20(9): 496. DOI: 10.1007/s11760-026-05550-9.
[21] Kharismawati E D, Kazic T. Maize Seedling Detection Dataset (MSDD): A Curated High-Resolution RGB Dataset for Maize Seedling Detection and Benchmarking with YOLOv9, YOLO11, YOLOv12, and Faster-RCNN. Agronomy, 2026, 16(16): 1605. DOI: 10.3390/agronomy16161605.
[22] Choi Y J, Han M J. Deep Learning (Fast R-CNN)-Based Evaluation of Rail Surface Defects. Applied Sciences, 2024, 14(5). DOI: 10.3390/app14051874.
[23] Wang K, Huang Y, Qi Y. TLDD-YOLO: An Improved YOLO for Transmission Line Component and Defect Detection. Electronics, 2026, 15(4): 757. DOI: 10.3390/electronics15040757.
[24] Donohoe P S, Alege P F, Delhom D C. Toward Multi-Class Detection and Localization of Contamination in Cotton Fields: Comparing EfficientDet, Faster R-CNN, ResNet50, SSD, MobileNet V2, and YOLOv11. Computers and Electronics in Agriculture, 2026, 247: 111690. DOI: 10.1016/j.compag.2026.111690.
[25] Mohammed A, Ibrahim M H, Omar M N. Optimizing RetinaNet anchors using differential evolution for improved object detection. Scientific Reports, 2025, 15(1): 20101. DOI: 10.1038/s41598-025-02888-x.
[26] Imran A, Li J. X-MCB-Net: An Explainable Multimodal Cyberbullying Detection Network Using Swin Transformer. Information Processing and Management, 2027, 64(2PA): 105172. DOI: 10.1016/j.ipm.2026.105172.
[27] Wang S, Yang W. DG-DETR: Density-Guided Dynamic Query Detection Transformer for Efficient Tiny Object Detection. Image and Vision Computing, 2026, 175: 106190. DOI: 10.1016/j.imavis.2026.106190.
[28] Kaushik P, Kukreja V, Jain E, et al. Oral Tumor Detection and Localization Using Detection Transformer (DETR) with Multiscale Feature Extraction and Self-Supervised Learning. Array, 2026, 30: 100825. DOI: 10.1016/j.array.2026.100825.
[29] Phan H, Vu N, Nguyen V, et al. Action recognition based on motion of oriented magnitude patterns and feature selection. IET Computer Vision, 2018, 12(5): 735-743. DOI: 10.1049/iet-cvi.2017.0282.
[30] Chiman K, Bence B. Enhancing Small Moving Target Detection Performance in Low-Quality and Long-Range Infrared Videos Using Optical Flow Techniques. Remote Sensing, 2020, 12(24): 4024. DOI: 10.3390/rs12244024.
[31] Mostafazadeh R, Zabihi M, Mobaser A J. Variability in Time and Space: Flow Regimes and Seasonality Indices in the Mountainous Rivers of Northern Iran. River Research and Applications, 2025, 41(9): 1874-1887. DOI: 10.1002/rra.70016.
[32] Zou L, Jia J, Ye Y, et al. STA-Fault3D: A Lightweight 3D Seismic Fault Detection Network Based on Spatial–Temporal Asymmetric Convolution Set. Applied Sciences, 2025, 15(22): 12153. DOI: 10.3390/app152212153.
[33] Xuewen X, Lu Z, Hongrun W, et al. Violence Detection with a Two-Stream Neural Network Based on C3D. International Journal of Cognitive Informatics and Natural Intelligence (IJCINI), 2021, 15(4): 1-17. DOI: 10.4018/ijcini.287601.
[34] Flávio S, Dalila D, S F M, et al. Weakness Evaluation on In-Vehicle Violence Detection: An Assessment of X3D, C2D, and I3D against FGSM and PGD. Electronics, 2022, 11(6): 852. DOI: 10.3390/electronics11060852.
[35] Alalwan N, Alblehai F, Bayatti A H A, et al. Parallel Spatio-Temporal Slow-Fast Model for Behavior Detection in AI-Enhanced Classrooms: A Vision-Based Approach. Journal of Circuits, Systems and Computers, 2024, 34(6). DOI: 10.1142/s0218126625501385.
[36] Alsawadi M, Kenawy E S E, Rio M. Using BlazePose on Spatio-Temporal Graph Convolutional Networks for Action Recognition. Computers, Materials & Continua, 2022, 74(1): 19-36. DOI: 10.32604/cmc.2023.032499.
[37] Zexu D, Guoliang Z, Weijiang L, et al. Spatio-Temporal Transformer for Online Video Understanding. Journal of Physics: Conference Series, 2022, 2171(1). DOI: 10.1088/1742-6596/2171/1/012020.
[38] Li Q, Li X, He X, et al. Identification of Microchannel Gas–Liquid Two-Phase Flow Patterns Based on the Timesformer Framework. Flow Measurement and Instrumentation, 2026, 111: 103456. DOI: 10.1016/j.flowmeasinst.2026.103456.
[39] Hu Z, Zhang S, Liu F. Video tampering detection with a forgery trace-aware SwinTransformer. Neurocomputing, 2026, 675: 132995. DOI: 10.1016/j.neucom.2026.132995.
[40] Hermida P, Santos D M E. Adding compact parameter blocks to multimodal transformers to detect harmful memes. Engineering Applications of Artificial Intelligence, 2024, 137(PA): 109136. DOI: 10.1016/j.engappai.2024.109136.
[41] Hongjie W, Junkai L, Tengsheng J, et al. AttentionMGT-DTA: A multimodal drug-target affinity prediction model using graph transformers and attention mechanisms. Neural Networks, 2024, 169: 623-636. DOI: 10.1016/j.neunet.2023.11.018.
[42] Li H, Zhuang P, Su Y, et al. Toward Generalizable and Robust Image Tampering Localization with Multi-Task Learning and Contrastive Learning. Expert Systems with Applications, 2025, 270: 126492. DOI: 10.1016/j.eswa.2025.126492.
[43] Qiao L, Li C, Cheng Z, et al. Reading order detection in visually-rich documents with multi-modal layout-aware relation prediction. Pattern Recognition, 2024, 150: 110314. DOI: 10.1016/j.patcog.2024.110314.
[44] Song J, Hao Y, Zhai H, et al. Multi-Scale Hierarchical Context-Aware Survival Prediction Network Based on Whole-Slide Images. Biomedical Signal Processing and Control, 2025, 110(PA): 108159. DOI: 10.1016/j.bspc.2025.108159.
[45] Cen J, Si W, Liu X, et al. Intelligent Fault Diagnosis Method Based on Data Generation and Long-Patch Vision Transformer under Small Samples. Applied Intelligence, 2025, 55(7): 650. DOI: 10.1007/s10489-025-06535-w.
[46] Zhang X, Wu D, Li J, et al. Infrared and Visible Image Fusion via Lightweight Semantic Prior Encoding and Cross-Attention Fusion. Sensors, 2026, 26(13): 4300. DOI: 10.3390/s26134300.
[47] Li Y, Zhang W, Lv S, et al. YOLOv11-CAFM Model for Pavement Distress Detection and Optimization Using Ground-Penetrating Radar Images. Construction and Building Materials, 2025, 48: 141907. DOI: 10.1016/j.conbuildmat.2025.141907.
[48] Gunawardhana M, Zolek N. ResNext-based U-Net for Sonomammogram Segmentation. Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference, 2025: 1-7. DOI: 10.1109/embc58623.2025.11253306.
[49] Hu L, Zhou X, Ruan J, et al. ASPP+-LANet: A Multi-Scale Context Extraction Network for Semantic Segmentation of High-Resolution Remote Sensing Images. Remote Sensing, 2024, 16(6). DOI: 10.3390/rs16061036.
[50] Zhou S, Qiu J, Solanki A. Improved SSD Using Deep Multi-Scale Attention Spatial–Temporal Features for Action Recognition. Multimedia Systems, 2021, 28(6): 1-9. DOI: 10.1007/s00530-021-00831-4.
[51] Xu T, Zhang G, Ruan Y, et al. An Improved YOLOv8 by Fusing a Coordinate Attention Mechanism and a Bidirectional Feature Pyramid Network for Identifying Power Repair Vehicles in the Cable Terminal Field. International Journal of Parallel, Emergent and Distributed Systems, 2026, 41(5): 675-692. DOI: 10.1080/17445760.2025.2518139.
[52] Zhu Y, Yu Q, Xu Z. An Optimized Watermelon Scion Leaf Segmentation Model Based on the Hungarian Algorithm and Information Theory. Electronics, 2025, 14(3): 620. DOI: 10.3390/electronics14030620.
[53] Wulff D, Ernst F. Real-time deformable structure tracking in 3D ultrasound sequences using deformable convolutional layers. Computers in Biology and Medicine, 2025, 186: 109671. DOI: 10.1016/j.compbiomed.2025.109671.
[54] Beram R S, Lalchhanhima R, Singh R K. MsFuNet: A Novel Deep Learning Framework for Robust Mammogram Analysis Using Context-Aware Feature Pyramid Networks and Cross-Attention Feature Fusion. Evolving Systems, 2026, 17(2): 58. DOI: 10.1007/s12530-026-09825-x.
[55] Jianbo Z, Mingfeng G, Zhimin C, et al. CRCNet: Global-local context and multi-modality cross-attention for polyp segmentation. Biomedical Signal Processing and Control, 2023, 83. DOI: 10.1016/j.bspc.2023.104593.
[56] Li Z, Shou D. Gesture recognition with adaptive-weight-based residual Multihead Cross Attention fusion based on multi-level feature information. Information Fusion, 2025, 115: 102789. DOI: 10.1016/j.inffus.2024.102789.
[57] Matsushita Y, Tamura A, Masuda T, et al. Deep learning-based automated identification of papillary bladder cancer from endoscopic still images using SE-ResNeXt-50. World Journal of Urology, 2026, 44(1): 566. DOI: 10.1007/s00345-026-06694-6.