Update with 51K dataset - 98.6% accuracy

README.md

@@ -1,242 +1,65 @@
 ---
-license: apache-2.0
 language:
 - en
-- es
-- fr
-- de
-- zh
 - ja
-- ko
-- ar
-- pt
-- ru
-- hi
+- tr
 - multilingual
-library_name: peft
+license: apache-2.0
 tags:
-- text-classification
 - feedback-detection
 - user-satisfaction
 - mmbert
-- modernbert
-- multilingual
 - lora
 - peft
-- vllm-semantic-router
+base_model: jhu-clsp/mmBERT-base
 datasets:
 - llm-semantic-router/feedback-detector-dataset
 metrics:
 - accuracy
 - f1
-base_model: jhu-clsp/mmBERT-base
-pipeline_tag: text-classification
-model-index:
-- name: mmbert-feedback-detector-lora
-  results:
-  - task:
-      type: text-classification
-      name: User Feedback Classification
-    dataset:
-      name: feedback-detector-dataset
-      type: llm-semantic-router/feedback-detector-dataset
-    metrics:
-    - type: accuracy
-      value: 0.9689
-      name: Accuracy
-    - type: f1
-      value: 0.9688
-      name: F1 Macro
 ---
 
-# mmBERT Feedback Detector (LoRA Adapter)
-
-A **multilingual** 4-class user feedback classifier built on [jhu-clsp/mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base). This is the **LoRA adapter** version for parameter-efficient fine-tuning and inference.
+# mmBERT Feedback Detector (LoRA)
 
-## Model Description
+A multilingual 4-class feedback classification model fine-tuned with LoRA on mmBERT-base.
 
-This model uses **LoRA (Low-Rank Adaptation)** for efficient fine-tuning, reducing trainable parameters to just **1.09%** of the full model while maintaining high performance.
+## Model Performance
 
-For the merged version (ready for direct inference), see [llm-semantic-router/mmbert-feedback-detector-merged](https://huggingface.co/llm-semantic-router/mmbert-feedback-detector-merged).
-
-### Labels
-
-| Label | ID | Description |
-|-------|-----|-------------|
-| `SAT` | 0 | User is satisfied with the response |
-| `NEED_CLARIFICATION` | 1 | User needs more explanation or clarification |
-| `WRONG_ANSWER` | 2 | User indicates the response is incorrect |
-| `WANT_DIFFERENT` | 3 | User wants alternative options or different response |
-
-## Performance
-
-| Metric | Score |
+| Metric | Value |
 |--------|-------|
-| **Accuracy** | 96.89% |
-| **F1 Macro** | 96.88% |
-| **F1 Weighted** | 96.88% |
-
-### Per-Class Performance
-
-| Class | F1 Score |
-|-------|----------|
-| SAT | 100.0% |
-| NEED_CLARIFICATION | 99.7% |
-| WRONG_ANSWER | 94.0% |
-| WANT_DIFFERENT | 93.8% |
+| **Accuracy** | 98.63% |
+| **F1 Macro** | 97.94% |
+| F1 SAT | 100.0% |
+| F1 NEED_CLARIFICATION | 99.7% |
+| F1 WRONG_ANSWER | 96.2% |
+| F1 WANT_DIFFERENT | 95.9% |
 
-## LoRA Configuration
+## Labels
 
-| Parameter | Value |
-|-----------|-------|
-| **Rank (r)** | 16 |
-| **Alpha** | 32 |
-| **Target Modules** | query, key, value, dense |
-| **Dropout** | 0.1 |
-| **Trainable Params** | 3.38M (1.09%) |
-| **Total Params** | 310.9M |
+- **SAT** (0): User is satisfied
+- **NEED_CLARIFICATION** (1): User needs more information  
+- **WRONG_ANSWER** (2): System gave incorrect response
+- **WANT_DIFFERENT** (3): User wants something different
 
-## Multilingual Support
+## Training
 
-Thanks to mmBERT's multilingual pretraining (256k vocabulary, 100+ languages), this model achieves excellent cross-lingual transfer:
-
-| Language | Accuracy |
-|----------|----------|
-| 🇺🇸 English | 100% |
-| 🇪🇸 Spanish | 100% |
-| 🇫🇷 French | 100% |
-| 🇩🇪 German | 100% |
-| 🇨🇳 Chinese | 100% |
-| 🇯🇵 Japanese | 100% |
-| 🇰🇷 Korean | 100% |
-| 🇸🇦 Arabic | 100% |
+- **Base Model**: jhu-clsp/mmBERT-base
+- **Dataset**: 51,694 examples (llm-semantic-router/feedback-detector-dataset)
+- **LoRA**: rank=32, alpha=64
+- **Epochs**: 5
+- **Batch Size**: 64
 
 ## Usage
 
-### With PEFT
-
-```python
-from peft import PeftModel, PeftConfig
-from transformers import AutoModelForSequenceClassification, AutoTokenizer
-import torch
-
-# Load base model and LoRA adapter
-base_model_name = "jhu-clsp/mmBERT-base"
-adapter_name = "llm-semantic-router/mmbert-feedback-detector-lora"
-
-# Load tokenizer
-tokenizer = AutoTokenizer.from_pretrained(adapter_name)
-
-# Load base model
-base_model = AutoModelForSequenceClassification.from_pretrained(
-    base_model_name,
-    num_labels=4,
-    trust_remote_code=True
-)
-
-# Load LoRA adapter
-model = PeftModel.from_pretrained(base_model, adapter_name)
-model.eval()
-
-# Classify feedback
-text = "Thanks, that's exactly what I needed!"
-inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
-
-with torch.no_grad():
-    outputs = model(**inputs)
-    probs = torch.softmax(outputs.logits, dim=-1)
-    pred = probs.argmax().item()
-
-labels = ["SAT", "NEED_CLARIFICATION", "WRONG_ANSWER", "WANT_DIFFERENT"]
-print(f"Prediction: {labels[pred]} ({probs[0][pred]:.1%})")
-```
-
-### Merge and Save
-
 ```python
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
 from peft import PeftModel
-from transformers import AutoModelForSequenceClassification, AutoTokenizer
 
-# Load and merge
-base_model = AutoModelForSequenceClassification.from_pretrained(
-    "jhu-clsp/mmBERT-base",
-    num_labels=4,
-    trust_remote_code=True
-)
+base_model = AutoModelForSequenceClassification.from_pretrained("jhu-clsp/mmBERT-base", num_labels=4)
 model = PeftModel.from_pretrained(base_model, "llm-semantic-router/mmbert-feedback-detector-lora")
-merged_model = model.merge_and_unload()
-
-# Save merged model
-merged_model.save_pretrained("./merged_model")
 tokenizer = AutoTokenizer.from_pretrained("llm-semantic-router/mmbert-feedback-detector-lora")
-tokenizer.save_pretrained("./merged_model")
-```
-
-### Continue Fine-tuning
-
-```python
-from peft import PeftModel, get_peft_model, LoraConfig
-from transformers import AutoModelForSequenceClassification
-
-# Load existing adapter
-base_model = AutoModelForSequenceClassification.from_pretrained(
-    "jhu-clsp/mmBERT-base",
-    num_labels=4,
-    trust_remote_code=True
-)
-model = PeftModel.from_pretrained(base_model, "llm-semantic-router/mmbert-feedback-detector-lora")
 
-# Continue training with your data
-# model.train()
-# ... training code ...
+inputs = tokenizer("Thank you, that was helpful!", return_tensors="pt")
+outputs = model(**inputs)
+label = outputs.logits.argmax(-1).item()
 ```
-
-## Training Details
-
-- **Base Model**: [jhu-clsp/mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base)
-- **Method**: LoRA (Low-Rank Adaptation)
-- **LoRA Rank**: 16
-- **LoRA Alpha**: 32
-- **Learning Rate**: 2e-5
-- **Batch Size**: 32
-- **Epochs**: 5
-- **Max Length**: 512
-- **Dataset**: [llm-semantic-router/feedback-detector-dataset](https://huggingface.co/datasets/llm-semantic-router/feedback-detector-dataset)
-
-## Use Cases
-
-- **Conversational AI**: Understand if users are satisfied with chatbot responses
-- **Customer Support**: Route dissatisfied users to human agents
-- **Quality Monitoring**: Track response quality across languages
-- **Feedback Analysis**: Categorize user feedback automatically
-- **Continued Fine-tuning**: Adapt to domain-specific feedback patterns
-
-## Advantages of LoRA
-
-- **Storage Efficient**: Only ~13MB adapter vs ~1.2GB full model
-- **Fast Training**: Train in minutes on consumer GPUs
-- **Composable**: Stack with other adapters
-- **Base Model Updates**: Benefit from base model improvements
-
-## Related Models
-
-- [llm-semantic-router/mmbert-feedback-detector-merged](https://huggingface.co/llm-semantic-router/mmbert-feedback-detector-merged) - Merged version
-- [llm-semantic-router/mmbert-intent-classifier-lora](https://huggingface.co/llm-semantic-router/mmbert-intent-classifier-lora) - Intent classification
-- [llm-semantic-router/mmbert-fact-check-lora](https://huggingface.co/llm-semantic-router/mmbert-fact-check-lora) - Fact checking
-- [llm-semantic-router/mmbert-jailbreak-detector-lora](https://huggingface.co/llm-semantic-router/mmbert-jailbreak-detector-lora) - Security
-
-## Citation
-
-```bibtex
-@misc{mmbert-feedback-detector-lora,
-  title={mmBERT Feedback Detector LoRA Adapter},
-  author={vLLM Semantic Router Team},
-  year={2025},
-  publisher={Hugging Face},
-  url={https://huggingface.co/llm-semantic-router/mmbert-feedback-detector-lora}
-}
-```
-
-## License
-
-Apache 2.0

adapter_config.json

@@ -13,7 +13,7 @@
   "layers_pattern": null,
   "layers_to_transform": null,
   "loftq_config": {},
-  "lora_alpha": 32,
+  "lora_alpha": 64,
   "lora_bias": false,
   "lora_dropout": 0.1,
   "megatron_config": null,
@@ -24,14 +24,14 @@
   ],
   "peft_type": "LORA",
   "qalora_group_size": 16,
-  "r": 16,
+  "r": 32,
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
+    "attn.Wqkv",
     "attn.Wo",
     "mlp.Wi",
-    "mlp.Wo",
-    "attn.Wqkv"
+    "mlp.Wo"
   ],
   "target_parameters": null,
   "task_type": "SEQ_CLS",

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1d96dddc3fd68845012d5c21af03edce7c5d61ec0d9a8553bb2711025e5148de
-size 13551024
+oid sha256:1df0ab37e51dd11727ccdcbbdd040a4a05d0f5f4cf66b0296b5cfd54d1e01afd
+size 27067968

training_config.json

@@ -21,8 +21,8 @@
   "base_model": "jhu-clsp/mmBERT-base",
   "max_length": 512,
   "use_lora": true,
-  "lora_rank": 16,
-  "lora_alpha": 32,
+  "lora_rank": 32,
+  "lora_alpha": 64,
   "class_weights": [
     1.0,
     1.0,
@@ -30,17 +30,17 @@
     1.0
   ],
   "metrics": {
-    "eval_loss": 0.11521816998720169,
-    "eval_accuracy": 0.9688755020080321,
-    "eval_f1_macro": 0.9688249790009527,
-    "eval_f1_weighted": 0.9688249790009527,
-    "eval_f1_SAT": 1.0,
-    "eval_f1_NEED_CLARIFICATION": 0.996996996996997,
-    "eval_f1_WRONG_ANSWER": 0.9403578528827038,
-    "eval_f1_WANT_DIFFERENT": 0.9379450661241099,
-    "eval_runtime": 1.454,
-    "eval_samples_per_second": 1370.016,
-    "eval_steps_per_second": 22.008,
+    "eval_loss": 0.060757871717214584,
+    "eval_accuracy": 0.9819095477386934,
+    "eval_f1_macro": 0.9729965135266696,
+    "eval_f1_weighted": 0.9818680631789307,
+    "eval_f1_SAT": 0.999664767013074,
+    "eval_f1_NEED_CLARIFICATION": 0.996,
+    "eval_f1_WRONG_ANSWER": 0.9474747474747475,
+    "eval_f1_WANT_DIFFERENT": 0.9488465396188566,
+    "eval_runtime": 1.2344,
+    "eval_samples_per_second": 2418.084,
+    "eval_steps_per_second": 19.442,
     "epoch": 5.0
   }
 }