---
language:
- bn
- en
license: mit
tags:
- multimodal
- sentiment-analysis
- meme
- cross-attention
- xlm-roberta
- eva-clip
- bengali
- multilingual
datasets:
- memosen
metrics:
- f1
- accuracy
---

# Meme Sentiment Analysis — Dual Cross-Attention (V1)

**Author:** EsferSami  
**Task:** Multilingual Multimodal Meme Sentiment Analysis  
**Dataset:** [MemoSen — Kaggle](https://www.kaggle.com/datasets/arifkaggle979/memosen-dataset)  
**Version:** V1 — Frozen Encoders (Pure Pretrained Baseline)

---

## Overview

MemoSen is a multilingual multimodal meme sentiment dataset containing Bengali, English, and code-mixed meme captions paired with meme images. Each sample has a sentiment label: Positive, Negative, or Neutral.

This model resolves cross-modal conflict between meme text and image using a dual cross-attention framework. Meme understanding is challenging because text and image often conflict — especially in sarcastic, ironic, or culturally specific memes.

---

## Model Architecture

| Component             | Details                               |
|-----------------------|---------------------------------------|
| Text Encoder          | XLM-RoBERTa base (frozen)             |
| Image Encoder         | EVA02-CLIP-L/14 from QuanSun (frozen) |
| Feature Dim           | 768 projected to 512                  |
| Fusion                | Dual Cross-Attention + Adaptive Gating|
| Conflict Detection    | Cosine similarity + MLP               |
| Reliability Weighting | Per-modality sigmoid scoring          |
| Classifier            | MLP (512 -> 256 -> 128 -> 3)          |

---

## Dataset

Dataset: [MemoSen on Kaggle](https://www.kaggle.com/datasets/arifkaggle979/memosen-dataset)

| Split      | Ratio |
|------------|-------|
| Train      | 70%   |
| Validation | 15%   |
| Test       | 15%   |

**Classes:** Positive, Negative, Neutral  
**Languages:** Bengali, English, Code-mixed (Banglish)

---

## Training Configuration

| Parameter        | Value                    |
|------------------|--------------------------|
| Epochs           | 50 (early stopped at 26) |
| Optimizer        | AdamW (lr=1e-4)          |
| Scheduler        | CosineAnnealingLR        |
| Loss             | Focal Loss + Class Weights|
| Early Stopping   | Patience = 7             |
| Batch Size       | 32                       |
| Max Text Length  | 128 tokens               |
| Image Size       | 224 x 224                |
| GPU              | T4 (Kaggle)              |

---

## Results (Validation Set)

| Metric   | Score            |
|----------|------------------|
| Macro-F1 | 0.5584|

> Full test set metrics will be updated after evaluation.  
> Main metric: Macro-F1

---

## Training Curves

![Training Curves](training_curves_v1.png)

---

## Notes

- V1 uses fully frozen encoders — only fusion and classification layers are trained
- Trainable parameters are limited to adaptation, cross-attention, conflict detection, reliability weighting, gated fusion, and MLP classifier
- V2 will experiment with unfreezing encoders and architecture modifications

---

## Citation

If you use MemoSen dataset, please refer to the original dataset on Kaggle:  
https://www.kaggle.com/datasets/arifkaggle979/memosen-dataset