Meme Sentiment Analysis — Dual Cross-Attention (V1)
Author: EsferSami
Task: Multilingual Multimodal Meme Sentiment Analysis
Dataset: MemoSen — Kaggle
Version: V1 — Frozen Encoders (Pure Pretrained Baseline)
Overview
MemoSen is a multilingual multimodal meme sentiment dataset containing Bengali, English, and code-mixed meme captions paired with meme images. Each sample has a sentiment label: Positive, Negative, or Neutral.
This model resolves cross-modal conflict between meme text and image using a dual cross-attention framework. Meme understanding is challenging because text and image often conflict — especially in sarcastic, ironic, or culturally specific memes.
Model Architecture
| Component | Details |
|---|---|
| Text Encoder | XLM-RoBERTa base (frozen) |
| Image Encoder | EVA02-CLIP-L/14 from QuanSun (frozen) |
| Feature Dim | 768 projected to 512 |
| Fusion | Dual Cross-Attention + Adaptive Gating |
| Conflict Detection | Cosine similarity + MLP |
| Reliability Weighting | Per-modality sigmoid scoring |
| Classifier | MLP (512 -> 256 -> 128 -> 3) |
Dataset
Dataset: MemoSen on Kaggle
| Split | Ratio |
|---|---|
| Train | 70% |
| Validation | 15% |
| Test | 15% |
Classes: Positive, Negative, Neutral
Languages: Bengali, English, Code-mixed (Banglish)
Training Configuration
| Parameter | Value |
|---|---|
| Epochs | 50 (early stopped at 26) |
| Optimizer | AdamW (lr=1e-4) |
| Scheduler | CosineAnnealingLR |
| Loss | Focal Loss + Class Weights |
| Early Stopping | Patience = 7 |
| Batch Size | 32 |
| Max Text Length | 128 tokens |
| Image Size | 224 x 224 |
| GPU | T4 (Kaggle) |
Results (Validation Set)
| Metric | Score |
|---|---|
| Macro-F1 | 0.5584 |
Full test set metrics will be updated after evaluation.
Main metric: Macro-F1
Training Curves
Notes
- V1 uses fully frozen encoders — only fusion and classification layers are trained
- Trainable parameters are limited to adaptation, cross-attention, conflict detection, reliability weighting, gated fusion, and MLP classifier
- V2 will experiment with unfreezing encoders and architecture modifications
Citation
If you use MemoSen dataset, please refer to the original dataset on Kaggle:
https://www.kaggle.com/datasets/arifkaggle979/memosen-dataset