| --- |
| language: |
| - bn |
| - en |
| license: mit |
| tags: |
| - multimodal |
| - sentiment-analysis |
| - meme |
| - cross-attention |
| - xlm-roberta |
| - eva-clip |
| - bengali |
| - multilingual |
| datasets: |
| - memosen |
| metrics: |
| - f1 |
| - accuracy |
| --- |
| |
| # Meme Sentiment Analysis — Dual Cross-Attention (V1) |
|
|
| **Author:** EsferSami |
| **Task:** Multilingual Multimodal Meme Sentiment Analysis |
| **Dataset:** [MemoSen — Kaggle](https://www.kaggle.com/datasets/arifkaggle979/memosen-dataset) |
| **Version:** V1 — Frozen Encoders (Pure Pretrained Baseline) |
|
|
| --- |
|
|
| ## Overview |
|
|
| MemoSen is a multilingual multimodal meme sentiment dataset containing Bengali, English, and code-mixed meme captions paired with meme images. Each sample has a sentiment label: Positive, Negative, or Neutral. |
|
|
| This model resolves cross-modal conflict between meme text and image using a dual cross-attention framework. Meme understanding is challenging because text and image often conflict — especially in sarcastic, ironic, or culturally specific memes. |
|
|
| --- |
|
|
| ## Model Architecture |
|
|
| | Component | Details | |
| |-----------------------|---------------------------------------| |
| | Text Encoder | XLM-RoBERTa base (frozen) | |
| | Image Encoder | EVA02-CLIP-L/14 from QuanSun (frozen) | |
| | Feature Dim | 768 projected to 512 | |
| | Fusion | Dual Cross-Attention + Adaptive Gating| |
| | Conflict Detection | Cosine similarity + MLP | |
| | Reliability Weighting | Per-modality sigmoid scoring | |
| | Classifier | MLP (512 -> 256 -> 128 -> 3) | |
|
|
| --- |
|
|
| ## Dataset |
|
|
| Dataset: [MemoSen on Kaggle](https://www.kaggle.com/datasets/arifkaggle979/memosen-dataset) |
|
|
| | Split | Ratio | |
| |------------|-------| |
| | Train | 70% | |
| | Validation | 15% | |
| | Test | 15% | |
|
|
| **Classes:** Positive, Negative, Neutral |
| **Languages:** Bengali, English, Code-mixed (Banglish) |
|
|
| --- |
|
|
| ## Training Configuration |
|
|
| | Parameter | Value | |
| |------------------|--------------------------| |
| | Epochs | 50 (early stopped at 26) | |
| | Optimizer | AdamW (lr=1e-4) | |
| | Scheduler | CosineAnnealingLR | |
| | Loss | Focal Loss + Class Weights| |
| | Early Stopping | Patience = 7 | |
| | Batch Size | 32 | |
| | Max Text Length | 128 tokens | |
| | Image Size | 224 x 224 | |
| | GPU | T4 (Kaggle) | |
|
|
| --- |
|
|
| ## Results (Validation Set) |
|
|
| | Metric | Score | |
| |----------|------------------| |
| | Macro-F1 | 0.5584| |
|
|
| > Full test set metrics will be updated after evaluation. |
| > Main metric: Macro-F1 |
|
|
| --- |
|
|
| ## Training Curves |
|
|
|  |
|
|
| --- |
|
|
| ## Notes |
|
|
| - V1 uses fully frozen encoders — only fusion and classification layers are trained |
| - Trainable parameters are limited to adaptation, cross-attention, conflict detection, reliability weighting, gated fusion, and MLP classifier |
| - V2 will experiment with unfreezing encoders and architecture modifications |
|
|
| --- |
|
|
| ## Citation |
|
|
| If you use MemoSen dataset, please refer to the original dataset on Kaggle: |
| https://www.kaggle.com/datasets/arifkaggle979/memosen-dataset |
|
|
