Model Card for Mistral-7B-Instruct-v0.1-Unsloth-MedicalQA

This is a medical question-answering model fine-tuned for healthcare domain

Foundation Model: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1
Dataset: https://huggingface.co/datasets/Laurent1/MedQuad-MedicalQnADataset_128tokens_max

The model has been fine-tuned using CUDA-enabled GPU hardware with optimized training through Unsloth.

Model Details

The model is based upon the foundation model: Mistral-7B-Instruct-v0.1.
It has been tuned with Supervised Fine-tuning Trainer using the Unsloth optimization framework for faster and more efficient training.

Libraries

• unsloth
• transformers
• torch
• trl
• peft
• einops
• bitsandbytes
• datasets

Training Configuration

Model Parameters

• max_sequence_length = 2048
• load_in_4bit = True
• LoRA rank (r) = 32
• lora_alpha = 16
• lora_dropout = 0

Target Modules for LoRA

• q_proj
• k_proj
• v_proj
• o_proj
• gate_proj
• up_proj
• down_proj

Training Hyperparameters

• per_device_train_batch_size = 2
• gradient_accumulation_steps = 16
• warmup_steps = 5
• warmup_ratio = 0.03
• max_steps = 1600
• learning_rate = 1e-4
• weight_decay = 0.01
• lr_scheduler_type = "linear"
• optimizer = "paged_adamw_32bit"

Training Statistics

Hardware Utilization

• Training duration: 10,561.28 seconds (approximately 176.02 minutes)
• Peak reserved memory: 5.416 GB
• Peak reserved memory for training: 0.748 GB
• Peak reserved memory % of max memory: 13.689%
• Peak reserved memory for training % of max memory: 1.891%

Dataset

The model was trained on the MedQuad dataset, which contains medical questions and answers. The training data was processed using a chat template format for instruction-tuning.

Bias, Risks, and Limitations

Users (both direct and downstream) should be aware of the following:

1. This model is intended for medical question-answering but should not be used as a substitute for professional medical advice.
2. The model's responses should be verified by healthcare professionals before making any medical decisions.
3. Generation of plausible yet incorrect medical information remains a possibility.
4. The model's knowledge is limited to its training data and may not cover all medical conditions or recent medical developments.

Usage

The model can be loaded and used with the Unsloth library:

from unsloth import FastLanguageModel
max_seq_length = 2048  # Choose any! We auto support RoPE Scaling internally!
dtype = (
    None  # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
)
model, tokenizer = FastLanguageModel.from_pretrained(
    "bouthros/Mistral-7B-Instruct-v0.1-Unsloth-MedicalQA",
    max_seq_length=2048,
    load_in_4bit=True,
)

Example usage:

messages = [
    {"from": "human", "value": "What are the types of liver cancer?"},
]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to("cuda")

Model Access

The model is available on Hugging Face Hub at: bouthros/Mistral-7B-Instruct-v0.1-Unsloth-MedicalQA

Citation

If you use this model, please cite the original Mistral-7B-Instruct-v0.1 model and the MedQuad dataset.