SentenceTransformer based on llm-semantic-router/mmbert-embed-32k-2d-matryoshka

This is a sentence-transformers model finetuned from llm-semantic-router/mmbert-embed-32k-2d-matryoshka. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: llm-semantic-router/mmbert-embed-32k-2d-matryoshka
Maximum Sequence Length: 32768 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 32768, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    "A 45-year-old woman, gravida 3, para 2, at 18 weeks' gestation comes to the physician for a prenatal visit. Ultrasonography at a previous visit when she was at 12 weeks' gestation showed a hypoplastic nasal bone. Pelvic examination shows a uterus consistent in size with an 18-week gestation. Maternal serum studies show low α-fetoprotein and free estriol concentrations, and increased inhibin A and β-hCG concentrations. Physical examination of the infant after delivery is most likely to show which of the following findings?",
    "A 45-year-old woman, gravida 3, para 2, at 18 weeks' gestation comes to the physician for a prenatal visit. Ultrasonography at a previous visit when she was at 12 weeks' gestation showed a hypoplastic nasal bone. Pelvic examination shows a uterus consistent in size with an 18-week gestation. Maternal serum studies show low α-fetoprotein and free estriol concentrations, and increased inhibin A and β-hCG concentrations. Physical examination of the infant after delivery is most likely to show which of the following findings?\n\nAnswer: Single transverse palmar crease",
    'A 35-year-old woman gravida 2, para 1, comes to the physician for her first prenatal visit. Pregnancy and delivery of her first child were uncomplicated. She is not sure about the date of her last menstrual period. Pelvic examination shows a uterus consistent in size with a 10-week gestation. An ultrasound examination confirms the gestational age and shows one fetus with no indication of multiple gestations. During counseling on pregnancy risks and possible screening and diagnostic tests, the patient states she would like to undergo screening for Down syndrome. She would prefer immediate and secure screening with a low risk to herself and the fetus. Which of the following is the most appropriate next step in management at this time?\n\nAnswer: Cell-free fetal DNA testing',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9805, 0.5273],
#         [0.9805, 1.0000, 0.5430],
#         [0.5273, 0.5430, 1.0000]], dtype=torch.bfloat16)

Training Details

Training Dataset

Unnamed Dataset

Size: 492 training samples
Columns: sentence_0, sentence_1, and sentence_2

Approximate statistics based on the first 492 samples:

	sentence_0	sentence_1	sentence_2
type	string	string	string
details	min: 36 tokens mean: 219.2 tokens max: 791 tokens	min: 49 tokens mean: 227.77 tokens max: 819 tokens	min: 51 tokens mean: 159.34 tokens max: 430 tokens

Samples:

sentence_0	sentence_1	sentence_2
A 48-year-old woman presents to the emergency department because of increasingly severe right upper abdominal pain, fever, and non-bloody vomiting for the last 5 hours. The pain is dull, intermittent, and radiates to her right shoulder. During the past 3 months, she has had recurring abdominal discomfort after meals. The patient underwent an appendectomy more than 30 years ago. She has hypertension, diabetes mellitus type 2, and chronic back pain. She takes bisoprolol, metformin, and ibuprofen daily. She is 171 cm (5 ft 6 in) tall and weighs 99 kg (218 lb). Her BMI is 35.2 kg/m2. She appears uncomfortable and is clutching her abdomen. Her temperature is 38.5°C (101.3°F), pulse is 108/min, and blood pressure is 150/82 mm Hg. Abdominal examination shows right upper quadrant abdominal tenderness and guarding. Upon deep palpation of the right upper quadrant, the patient pauses during inspiration. Laboratory studies show the following: Blood Hemoglobin 13.1 g/dL Leukocyte count 10,900/mm3 P...	A 48-year-old woman presents to the emergency department because of increasingly severe right upper abdominal pain, fever, and non-bloody vomiting for the last 5 hours. The pain is dull, intermittent, and radiates to her right shoulder. During the past 3 months, she has had recurring abdominal discomfort after meals. The patient underwent an appendectomy more than 30 years ago. She has hypertension, diabetes mellitus type 2, and chronic back pain. She takes bisoprolol, metformin, and ibuprofen daily. She is 171 cm (5 ft 6 in) tall and weighs 99 kg (218 lb). Her BMI is 35.2 kg/m2. She appears uncomfortable and is clutching her abdomen. Her temperature is 38.5°C (101.3°F), pulse is 108/min, and blood pressure is 150/82 mm Hg. Abdominal examination shows right upper quadrant abdominal tenderness and guarding. Upon deep palpation of the right upper quadrant, the patient pauses during inspiration. Laboratory studies show the following: Blood Hemoglobin 13.1 g/dL Leukocyte count 10,900/mm3 P...	A 27-year-old woman presents with acute abdominal pain in her right upper quadrant. The pain came on suddenly while she was eating dinner. After this pain she began feeling dizzy and came to the emergency department. In the ED, her blood pressure is 75/40 mmHg, pulse is 100/minute, and she is afebrile. On physical exam, she feels too light-headed to ambulate. She demonstrates normal bowel sounds with tenderness upon palpation in the right upper quadrant. The patient is deemed too unstable for imaging. An abdominal radiograph and CT are reviewed from a recent previous visit to the ED for mild abdominal pain, and are shown in Figures A and B, respectively. Which of the following specific additional findings in her history supports the most likely diagnosis? Answer: Use of oral contraceptives (OCPs) for birth control
A 42-year-old man presents to his primary care provider for abdominal pain. He reports that for several months he has been experiencing a stabbing pain above the umbilicus during meals. He denies associated symptoms of nausea, vomiting, or diarrhea. The patient’s past medical history is significant for hypertension and hyperlipidemia for which he takes amlodipine and atorvastatin. His family history is significant for lung cancer in his father. The patient is a current smoker with a 20 pack-year smoking history and drinks 3-5 beers per week. Initial laboratory testing is as follows: Serum: Na+: 141 mEq/L K+: 4.6 mEq/L Cl-: 102 mEq/L HCO3-: 25 mEq/L Urea nitrogen: 14 mg/dL Creatinine: 1.1 mg/dL Glucose: 120 mg/dL Calcium: 8.4 mg/dL Alkaline phosphatase: 66 U/L Aspartate aminotransferase (AST): 40 U/L Alanine aminotransferase (ALT): 52 U/L Gastrin: 96 pg/mL (<100 pg/mL) Lipase: 90 U/L (<160 U/L) The patient is started on a proton pump inhibitor without symptomatic improvement after 6 w...	A 42-year-old man presents to his primary care provider for abdominal pain. He reports that for several months he has been experiencing a stabbing pain above the umbilicus during meals. He denies associated symptoms of nausea, vomiting, or diarrhea. The patient’s past medical history is significant for hypertension and hyperlipidemia for which he takes amlodipine and atorvastatin. His family history is significant for lung cancer in his father. The patient is a current smoker with a 20 pack-year smoking history and drinks 3-5 beers per week. Initial laboratory testing is as follows: Serum: Na+: 141 mEq/L K+: 4.6 mEq/L Cl-: 102 mEq/L HCO3-: 25 mEq/L Urea nitrogen: 14 mg/dL Creatinine: 1.1 mg/dL Glucose: 120 mg/dL Calcium: 8.4 mg/dL Alkaline phosphatase: 66 U/L Aspartate aminotransferase (AST): 40 U/L Alanine aminotransferase (ALT): 52 U/L Gastrin: 96 pg/mL (<100 pg/mL) Lipase: 90 U/L (<160 U/L) The patient is started on a proton pump inhibitor without symptomatic improvement after 6 w...	A 42-year-old man comes to his primary care physician complaining of abdominal pain. He describes intermittent, burning, epigastric pain over the past 4 months. He reports that the pain worsens following meals. He had an upper gastrointestinal endoscopy done 2 months ago that showed a gastric ulcer without evidence of malignancy. The patient was prescribed pantoprazole with minimal improvement in symptoms. He denies nausea, vomiting, diarrhea, or melena. The patient has no other medical problems. He had a total knee replacement 3 years ago following a motor vehicle accident for which he took naproxen for 2 months for pain management. He has smoked 1 pack per day since the age 22 and drinks 1-2 beers several nights a week with dinner. He works as a truck driver, and his diet consists of mostly of fast food. His family history is notable for hypertension in his paternal grandfather and coronary artery disease in his mother. On physical examination, the abdomen is soft, nondistended, and ...
A 62-year-old woman comes to the physician in June for a routine check-up. She has chronic back pain and underwent an appendectomy at the age of 27. She is married and has two kids. The patient recently got back from a cruise to Mexico where she celebrated her 40th wedding anniversary. Her last mammogram was 6 months ago and showed no abnormalities. Her last Pap smear was 2 years ago and unremarkable. A colonoscopy 5 years ago was normal. Her mother died of breast cancer last year and her father has arterial hypertension. Her immunization records show that she has never received a pneumococcal or a shingles vaccine, her last tetanus booster was 6 years ago, and her last influenza vaccine was 2 years ago. She drinks 1– 2 alcoholic beverages every weekend. She takes a multivitamin daily and uses topical steroids. She regularly attends water aerobic classes and physical therapy for her back pain. She is 168 cm (5 ft 6 in) tall and weighs 72 kg (160 lb); BMI is 26 kg/m2. Her temperature is...	A 62-year-old woman comes to the physician in June for a routine check-up. She has chronic back pain and underwent an appendectomy at the age of 27. She is married and has two kids. The patient recently got back from a cruise to Mexico where she celebrated her 40th wedding anniversary. Her last mammogram was 6 months ago and showed no abnormalities. Her last Pap smear was 2 years ago and unremarkable. A colonoscopy 5 years ago was normal. Her mother died of breast cancer last year and her father has arterial hypertension. Her immunization records show that she has never received a pneumococcal or a shingles vaccine, her last tetanus booster was 6 years ago, and her last influenza vaccine was 2 years ago. She drinks 1– 2 alcoholic beverages every weekend. She takes a multivitamin daily and uses topical steroids. She regularly attends water aerobic classes and physical therapy for her back pain. She is 168 cm (5 ft 6 in) tall and weighs 72 kg (160 lb); BMI is 26 kg/m2. Her temperature is...	A 66-year-old G3P3 presents with an 8-year-history of back pain, perineal discomfort, difficulty urinating, recurrent malaise, and low-grade fevers. These symptoms have recurred regularly for the past 5–6 years. She also says that there are times when she experiences a feeling of having a foreign body in her vagina. With the onset of symptoms, she was evaluated by a physician who prescribed her medications after a thorough examination and recommended a vaginal pessary, but she was non-compliant. She had 3 vaginal deliveries She has been menopausal since 51 years of age. She does not have a history of malignancies or cardiovascular disease. She has type 2 diabetes mellitus that is controlled with diet and metformin. Her vital signs include: blood pressure 110/60 mm Hg, heart rate 91/min, respiratory rate 13/min, and temperature 37.4℃ (99.3℉). On physical examination, there is bilateral costovertebral angle tenderness. The urinary bladder is non-palpable. The gynecologic examination reve...

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false
}

Training Hyperparameters

Non-Default Hyperparameters

num_train_epochs: 2
multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

do_predict: False
eval_strategy: no
prediction_loss_only: True
per_device_train_batch_size: 8
per_device_eval_batch_size: 8
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1
num_train_epochs: 2
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_ratio: None
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
enable_jit_checkpoint: False
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
use_cpu: False
seed: 42
data_seed: None
bf16: False
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: -1
ddp_backend: None
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
group_by_length: False
length_column_name: length
project: huggingface
trackio_space_id: trackio
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_for_metrics: []
eval_do_concat_batches: True
auto_find_batch_size: False
full_determinism: False
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
use_cache: False
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: round_robin
router_mapping: {}
learning_rate_mapping: {}

Framework Versions

Python: 3.12.10
Sentence Transformers: 5.2.2
Transformers: 5.0.0
PyTorch: 2.7.0+cu128
Accelerate: 1.12.0
Datasets: 4.5.0
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}