Create README.md

README.md

@@ -0,0 +1,109 @@
+---
+datasets:
+- stanfordnlp/imdb
+metrics:
+- perplexity
+base_model:
+- EuroBERT/EuroBERT-210m
+pipeline_tag: fill-mask
+tags:
+- art
+---
+# Data Card for EuroBERT-210m-finetuned-imdb
+
+## Model Overview
+
+- **Model Name**: EuroBERT-210m-finetuned-imdb
+- **Base Model**: EuroBERT-210m
+- **Fine-tuned On**: IMDb dataset
+- **Task**: Masked Language Modeling (MLM)
+- **Training Objective**: Minimize Perplexity
+
+## Dataset Details
+
+- **Dataset Used**: IMDb
+- **Dataset Version**: Default version from `datasets` library
+- **Dataset Source**: Hugging Face `datasets`
+- **Training Split**: `train`
+- **Evaluation Split**: `test`
+
+## Training & Evaluation
+
+### Training Process
+- The model was fine-tuned for three epochs using PyTorch and Hugging Face's `transformers` library.
+- The optimizer and learning rate scheduler were set up within the `accelerate` framework.
+
+### Evaluation Metrics
+- The model was evaluated using **Perplexity (PPL)** on the test set.
+- Results:
+  - **Epoch 0**: PPL = 12.63
+  - **Epoch 1**: PPL = 9.35
+  - **Epoch 2**: PPL = 8.12
+
+## Model Usage
+
+### Inference
+The model can be used for masked token prediction using the following script:
+
+```python
+import torch
+from transformers import AutoModelForMaskedLM, AutoTokenizer
+
+def predict_masked_sentence(sentence, mask_token="<|mask|>"):
+    """
+    Predicts top-1 tokens for all mask tokens in a sentence and returns the reconstructed text.
+    
+    Args:
+        sentence (str): Input sentence with mask tokens (e.g., "The movie was [MASK]!").
+        mask_token (str, optional): Token used as mask in the input sentence. Defaults to "<|mask|>".
+    
+    Returns:
+        str: Sentence with all mask tokens replaced by top-1 predictions.
+    """
+    model_checkpoint = "milanvelinovski/EuroBERT-210m-finetuned-imdb"
+    model = AutoModelForMaskedLM.from_pretrained(model_checkpoint, trust_remote_code=True)
+    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, trust_remote_code=True)
+
+    sentence_with_model_mask = sentence.replace(mask_token, "<|mask|>")
+    inputs = tokenizer(sentence_with_model_mask, return_tensors="pt")
+    token_logits = model(**inputs).logits
+
+    mask_token_indices = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]
+    top_tokens = [torch.topk(token_logits[0, idx, :], 1).indices.item() for idx in mask_token_indices]
+    
+    text_parts = sentence.split(mask_token)
+    final_text = text_parts[0] + ''.join(tokenizer.decode([token]) + text_parts[i+1] for i, token in enumerate(top_tokens))
+    
+    return final_text
+
+text = "The protagonist's journey was <|mask|>, filled with <|mask|> obstacles that made the ending feel <|mask|>."
+final_text = predict_masked_sentence(text)
+print(final_text)
+```
+
+## Libraries Used
+
+| Library      | Version  |
+|-------------|----------|
+| datasets    | 3.3.1    |
+| transformers| 4.49.0   |
+| evaluate    | 0.4.3    |
+| accelerate  | 1.2.1    |
+| torch       | 2.5.1+cu121 |
+
+## Model Limitations
+- The model is primarily trained for masked language modeling and may not generalize well to other NLP tasks.
+- The perplexity scores indicate that further fine-tuning or hyperparameter optimization might improve performance.
+- Model predictions are constrained by the IMDb dataset and may not generalize well to other domains.
+
+## Citation
+If you use this model, please cite:
+```
+@misc{EuroBERT-210m-finetuned-imdb,
+  author = {Milan Velinovski},
+  title = {EuroBERT-210m-finetuned-imdb},
+  year = {2025},
+  publisher = {Hugging Face},
+  url = {https://huggingface.co/milanvelinovski/EuroBERT-210m-finetuned-imdb}
+}
+```