Domain Knowledge vs. Complex Architectures in Emotion Recognition

A recent study has questioned the effectiveness of complex attention mechanisms, such as Transformers, in multimodal emotion recognition when applied to small datasets. The research, based on the EAV dataset, compares different model architectures, including baseline Transformers, novel factorized attention mechanisms, and improved CNN baselines.

Surprising Results

The results indicate that models based on sophisticated attention mechanisms tend to underperform compared to simpler baselines. In particular, models with factorized attention showed a performance drop of 5% to 13% due to overfitting and the destruction of pre-trained features. In contrast, targeted and domain knowledge-based modifications proved more effective. For example, adding delta MFCCs to the audio CNN improved accuracy from 61.9% to 65.56%, while using frequency-domain features for EEG led to a 7.62% increase compared to the original baseline.

Vision Transformer and Pre-training

The baseline Vision Transformer achieved an accuracy of 75.30%, surpassing the original ViViT result thanks to domain-specific pre-training. The use of delta features for vision led to a further improvement of 1.28% compared to the original CNN.

Implications

These results suggest that, for small-scale emotion recognition, domain knowledge and proper implementation can outperform architectural complexity. For those evaluating on-premise deployments, there are trade-offs to consider, and AI-RADAR offers analytical frameworks on /llm-onpremise to evaluate these aspects.