ADAPTIVE CROSS-MODAL ALIGNMENT AND ORTHOGONAL TASK DECOUPLING FOR MULTITASK MULTIMODAL SUMMARIZATION
Keywords:
Multimodal summarization, Multitask learning, Orthogonal representation decoupling, Cross-modal alignment, Shared-private representation, Negative transferAbstract
Producing a reliable multimodal summary requires a model to determine which claims are supported across language and vision and then express that evidence coherently. Training only for generation leaves useful cross-modal supervision untapped, yet indiscriminate parameter sharing across auxiliary tasks can merge transferable semantics with task-dependent cues and thereby induce negative transfer. Alignment is made still harder by visually salient background regions that have little bearing on the desired summary. To address these difficulties, we develop a single framework for multimodal summarization, image-text matching, and image-text retrieval. Visual tokens are first weighted through text-conditioned semantic attention; a learned gate then suppresses unhelpful visual responses before fusion. The aligned sequence is decomposed into one common representation and three private, task-oriented representations. Their redundancy is controlled by a Frobenius-norm orthogonality penalty, optimized jointly with the generation, matching, and retrieval objectives. The implementation couples a T5-base language encoder with CLIP ViT-B/16 and maintains 768-dimensional features in both streams. On MSMO, the full system records ROUGE-1, ROUGE-2, ROUGE-L, and BLEU values of 46.95, 22.03, 44.42, and 18.71, respectively, exceeding the single-task and multitask comparators included in the reported evaluation. Removing adaptive alignment, the shared-private decomposition, or the orthogonal term lowers ROUGE-L by 2.91, 2.08, and 0.87 points. Taken together, the findings indicate that text-directed visual filtering and an explicit division between reusable and task-specific information are both consequential for grounded summary generation.References
[1] Zhu J, Li H, Liu T, et al. MSMO: Multimodal summarization with multimodal output. In: Conference on Empirical Methods in Natural Language Processing, 2018: 4154-4164.
[2] Salzmann M, Ek C H, Urtasun R, et al. Factorized orthogonal latent spaces. In: International Conference on Artificial Intelligence and Statistics. PMLR, 2010: 701-708.
[3] Bousmalis K, Trigeorgis G, Silberman N, et al. Domain separation networks. In: Advances in Neural Information Processing Systems, 2016.
[4] Hu J, Kundu S, Poria S, et al. UniMS: A unified framework for multimodal summarization with knowledge distillation. In: AAAI Conference on Artificial Intelligence, 2022.
[5] Li H, Zhu J, Zhang C, et al. Aspect-aware multimodal summarization for Chinese e-commerce products. In: AAAI Conference on Artificial Intelligence, 2020: 8188-8195.
[6] Zhang L, Wei H, Cheng J, et al. Hierarchical cross-modal attention for multimodal summarization. IEEE Transactions on Multimedia, 2023, 25: 1230-1241.
[7] Wang Z, Yu J, Yu A W K, et al. SimVLM: Simple visual language model pretraining with weak supervision. In: International Conference on Learning Representations, 2022.
[8] Chung J Y, Lei J, Tan H, et al. Unifying vision-and-language tasks via text generation. In: International Conference on Machine Learning. PMLR, 2021: 1931-1942.
[9] Liu N, Joty S. UniSumm: Unified contrastive learning for multimodal summarization. In: Annual Meeting of the Association for Computational Linguistics, 2023.
[10] Liu Y, Zhu W. Unified multi-modal learning for cross-task generalization. In: IEEE Conference on Computer Vision and Pattern Recognition, 2023: 1821-1830.
[11] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. In: Advances in Neural Information Processing Systems, 2017: 5998-6008.
[12] Raffel C, Shazeer N, Roberts A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020, 21(1): 5485-5551.
[13] Chen J, He X, Guo Y, et al. CLIP-It! Language-guided video summarization. In: Advances in Neural Information Processing Systems, 2021.
[14] Lu J, Batra D, Parikh D, et al. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Advances in Neural Information Processing Systems, 2019.
[15] Li J, Li D, Xiong C, et al. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning, 2022.
[16] Vedantam R, Zitnick C L, Parikh D. CIDEr: Consensus-based image description evaluation. In: IEEE Conference on Computer Vision and Pattern Recognition, 2015: 4566-4575.
[17] Wei Y, Zhang L, Wang Y. Cross-modal information fusion for multimedia summarization: A survey. Journal of Computer Science and Technology, 2023, 38(1): 1-20.
[18] Jégou Y, Johnson J, Douze H. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 2019, 7(3): 535-547.
[19] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. PMLR, 2021: 8748-8763.
[20] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16×16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations, 2021.
[21] Lin C Y. ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out, 2004: 74-81.
[22] Papineni K, Roukos S, Ward T, et al. BLEU: A method for automatic evaluation of machine translation. In: Annual Meeting of the Association for Computational Linguistics, 2002: 311-318.
[23] Loshchilov I, Hutter F. Decoupled weight decay regularization. In: International Conference on Learning Representations, 2019.