Model Card for ai-ml-t-tes3-tradexy-data-dsr1-1.5b

This is a fine-tuned version of the facebook/opt-1.3b model using the LoRA (Low-Rank Adaptation) technique. The model has been trained on a dataset focused on Ayurveda and the concept of doshas (Vata, Pitta, Kapha). Compared to the previous model (ai-ml-t-tes2-dftopcat-data-dsr1-1.5b), this version incorporates improved training parameters and adjustments to enhance coherence, reduce repetition, and minimize inaccuracies. However, challenges remain in achieving depth and specificity about Vata, Pitta, and Kapha doshas .

---

Model Details

Model Description

This model is a fine-tuned adaptation of the facebook/opt-1.3b base model, optimized for generating explanations related to Ayurveda and doshas. It uses the LoRA technique to reduce computational costs while maintaining performance. The training data consists of instructional prompts and corresponding outputs that explain Ayurvedic concepts like doshic constitution, balance, and their influence on health.

Compared to the previous model (ai-ml-t-tes2-dftopcat-data-dsr1-1.5b), this version demonstrates:

• Improved Coherence: Responses are more structured and logical.
• Reduced Repetition: Adjustments to generation parameters (e.g., repetition_penalty) have further minimized redundant phrases.
• Fewer Inaccuracies: Misinterpretations (e.g., "doshas as disease-causing elements") are less frequent but still present.

However, the model still struggles with depth and specificity, particularly in explaining Vata, Pitta, and Kapha doshas in detail. Additionally, it occasionally introduces unrelated content or formatting artifacts.

• Developed by: tradexy
• Model type: Causal Language Model (Fine-Tuned)
• Language(s): English
• License: MIT License
• Finetuned from model: facebook/opt-1.3b

Model Sources

• Repository: tradexy/ai-ml-t-tes3-tradexy-data-dsr1-1.5b
• Dataset: Abhaykoul/Ancient-Indian-Wisdom

---

Uses

Direct Use

The model can be used to generate responses to questions about Ayurveda, particularly focusing on doshas and their role in health. It is suitable for educational purposes, answering FAQs, or providing introductory insights into Ayurvedic principles.

Downstream Use

The model can be integrated into applications like chatbots, virtual assistants, or educational platforms that focus on alternative medicine and wellness.

Out-of-Scope Use

The model is not designed for medical diagnosis, treatment recommendations, or generating content outside the scope of Ayurveda. Misuse or reliance on the model for critical health decisions is strongly discouraged.

---

Bias, Risks, and Limitations

Known Limitations

• While the model shows improvements over the previous version, it still occasionally generates repetitive or nonsensical phrases.
• Responses lack depth and specificity about Vata, Pitta, and Kapha doshas compared to expert-level explanations.
• The model sometimes introduces inaccuracies (e.g., misinterpreting doshas as "disease-causing elements") due to limitations in training data or fine-tuning.
• Unrelated content or formatting artifacts occasionally appear in responses.

Improvements Over Previous Model

• Increased Training Steps: Extended training steps (max_steps=1500) have allowed the model to better internalize nuances from the dataset.
• Adjusted LoRA Parameters: Higher values for r (32) and lora_alpha (64) provided greater capacity for adaptation.
• Smaller Learning Rate: A reduced learning rate (5e-5) ensured finer adjustments during training, reducing inaccuracies.
• Larger Validation Set: Increased validation set size (100 examples) improved evaluation metrics.
• Optimized Generation Parameters: Adjusted temperature, top_k, and repetition_penalty for better coherence and relevance.

Recommendations

• Use post-processing techniques to filter out irrelevant or inaccurate statements.
• Fine-tune the model further with more diverse and high-quality training data.
• Experiment with even larger base models (e.g., facebook/opt-6.7b) for improved performance

.

---

How to Get Started with the Model

To use this model, follow these steps:

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel, PeftConfig
import torch

# Load the base model
base_model = AutoModelForCausalLM.from_pretrained(
    "facebook/opt-1.3b",  # Original base model
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load the LoRA configuration and adapter
peft_config = PeftConfig.from_pretrained("tradexy/ai-ml-t-tes3-tradexy-data-dsr1-1.5b")
model = PeftModel.from_pretrained(base_model, "tradexy/ai-ml-t-tes3-tradexy-data-dsr1-1.5b")

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained("tradexy/ai-ml-t-tes3-tradexy-data-dsr1-1.5b")
tokenizer.pad_token = tokenizer.eos_token

# Generate text
def generate_text(prompt, max_new_tokens=500):
    inputs = tokenizer(prompt, return_tensors="pt").to('cuda')
    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=0.3,  # Balanced creativity and coherence
            top_k=25,         # Focus on higher-probability tokens
            top_p=0.87,       # Consistent quality
            repetition_penalty=1.3,  # Prevents repetition without over-penalizing
            use_cache=True
        )
    return tokenizer.decode(output[0], skip_special_tokens=True)

# Test the model
prompt = "Ayurveda emphasizes the balance between doshas. How can understanding our doshic constitution promote better health?"
output = generate_text(prompt)
print(output)