EmoXFormer: Human-Cognition-Inspired Multimodal Emotion Recognition from Disjoint Modality Datasets

초록

Most multimodal emotion recognition (MER) assumes temporally aligned and co-recorded modalities, yet deployed systems often observe text, audio, and vision from separate pipelines where sample-level correspondence is unavailable. We study Disjoint Modality Learning (DML), where modality specialists are trained on independent corpora and combined at inference without pairing, identity linkage, or synchronization. We propose EmoXFormer, an alignment-free fusion model that couples strong unimodal encoders (RoBERTa, Wav2Vec2, ViT) with Cross-Gated Context Attention (CGCA) to exchange cross-modal context while suppressing off-context activations under missing or mismatched evidence. To enable reproducible evaluation, we introduce a Unified Disjoint Test (UDT) that harmonizes MELD (text), RAVDESS (audio), and FER2013 (vision) into a shared seven-class label space. On UDT-U (unimodal aggregate under a shared head), EmoXFormer achieves 66.9% micro accuracy. On UDT-3M (congruent triads from disjoint corpora), fusion reaches 79.2%, outperforming the best unimodal baseline (63.9%) by +15.3 points. Results show that robust multimodal decision fusion is feasible even when cross-modal correspondence is absent.

키워드