philipp-zettl/all-MiniLM-L6-v2-mtg-v2

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("philipp-zettl/all-MiniLM-L6-v2-mtg-v2")
# Run inference
sentences = [
    '{4}, {t}, sacrifice this artifact: target player discards a card for each charge counter on this artifact',
    'Title: Shrine of Limitless Power\nCost: {3}\nType: Artifact\nDesc: At the beginning of your upkeep and whenever you cast a black spell, put a charge counter on this artifact.\n{4}, {T}, Sacrifice this artifact: Target player discards a card for each charge counter on this artifact.',
    'Title: Dawn Charm\nCost: {1}{W}\nColors: W\nType: Instant\nDesc: Choose one —\n• Prevent all combat damage that would be dealt this turn.\n• Regenerate target creature.\n• Counter target spell that targets you.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.6342, -0.0303],
#         [ 0.6342,  1.0000, -0.0325],
#         [-0.0303, -0.0325,  1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 2,132,917 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 100 samples:

  	sentence_0	sentence_1
  type	string	string
  modality	text	text
  details	min: 3 tokens mean: 10.62 tokens max: 43 tokens	min: 25 tokens mean: 66.88 tokens max: 136 tokens

• Samples:

  sentence_0	sentence_1
  colos yearling	Title: Colos Yearling Cost: {2}{R} Colors: R Type: Creature — Goat Beast Desc: Mountainwalk (This creature can't be blocked as long as defending player controls a Mountain.) {R}: This creature gets +1/+0 until end of turn.
  creature — zombie wizard	Title: Prophet of the Scarab Cost: {4}{U} Colors: U Type: Creature — Zombie Wizard Desc: Vigilance When this creature enters, draw cards equal to the number of Zombies you control or the number of Zombie cards in your graveyard, whichever is greater.
  {w}, {t}: target permanent you control gains protection from instant spells and from sorcery spells until end of turn	Title: Devoted Caretaker Cost: {W} Colors: W Type: Creature — Human Cleric Desc: {W}, {T}: Target permanent you control gains protection from instant spells and from sorcery spells until end of turn.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 96
• fp16: True
• per_device_eval_batch_size: 96
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• per_device_train_batch_size: 96
• num_train_epochs: 3
• max_steps: -1
• learning_rate: 5e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 96
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0225	500	0.8076
0.0450	1000	0.4803
0.0675	1500	0.4223
0.0900	2000	0.4065
0.1125	2500	0.3884
0.1350	3000	0.3720
0.1575	3500	0.3705
0.1800	4000	0.3654
0.2025	4500	0.3555
0.2250	5000	0.3534
0.2475	5500	0.3514
0.2701	6000	0.3571
0.2926	6500	0.3438
0.3151	7000	0.3496
0.3376	7500	0.3458
0.3601	8000	0.3406
0.3826	8500	0.3406
0.4051	9000	0.3350
0.4276	9500	0.3376
0.4501	10000	0.3329
0.4726	10500	0.3311
0.4951	11000	0.3305
0.5176	11500	0.3336
0.5401	12000	0.3306
0.5626	12500	0.3239
0.5851	13000	0.3265
0.6076	13500	0.3172
0.6301	14000	0.3240
0.6526	14500	0.3187
0.6751	15000	0.3234
0.6976	15500	0.3231
0.7201	16000	0.3206
0.7426	16500	0.3191
0.7651	17000	0.3193
0.7876	17500	0.3196
0.8102	18000	0.3187
0.8327	18500	0.3124
0.8552	19000	0.3133
0.8777	19500	0.3183
0.9002	20000	0.3076
0.9227	20500	0.3204
0.9452	21000	0.3188
0.9677	21500	0.3196
0.9902	22000	0.3094
1.0127	22500	0.3077
1.0352	23000	0.3112
1.0577	23500	0.3058
1.0802	24000	0.3103
1.1027	24500	0.3123
1.1252	25000	0.3085
1.1477	25500	0.3046
1.1702	26000	0.3071
1.1927	26500	0.3074
1.2152	27000	0.3134
1.2377	27500	0.3059
1.2602	28000	0.3101
1.2827	28500	0.3089
1.3052	29000	0.3027
1.3278	29500	0.3125
1.3503	30000	0.3024
1.3728	30500	0.3034
1.3953	31000	0.3031
1.4178	31500	0.3080
1.4403	32000	0.3095
1.4628	32500	0.3044
1.4853	33000	0.3012
1.5078	33500	0.3113
1.5303	34000	0.3023
1.5528	34500	0.3117
1.5753	35000	0.3078
1.5978	35500	0.3130
1.6203	36000	0.3103
1.6428	36500	0.3023
1.6653	37000	0.2976
1.6878	37500	0.3018
1.7103	38000	0.3040
1.7328	38500	0.2962
1.7553	39000	0.3036
1.7778	39500	0.3040
1.8003	40000	0.3062
1.8228	40500	0.3040
1.8454	41000	0.3046
1.8679	41500	0.2976
1.8904	42000	0.3014
1.9129	42500	0.3041
1.9354	43000	0.3035
1.9579	43500	0.3001
1.9804	44000	0.3012
2.0029	44500	0.3000
2.0254	45000	0.3061
2.0479	45500	0.3035
2.0704	46000	0.2972
2.0929	46500	0.3001
2.1154	47000	0.2968
2.1379	47500	0.2978
2.1604	48000	0.3052
2.1829	48500	0.2998
2.2054	49000	0.2950
2.2279	49500	0.3018
2.2504	50000	0.3004
2.2729	50500	0.2934
2.2954	51000	0.2969
2.3179	51500	0.2968
2.3404	52000	0.3023
2.3629	52500	0.2974
2.3855	53000	0.3037
2.4080	53500	0.3007
2.4305	54000	0.2969
2.4530	54500	0.3055
2.4755	55000	0.2938
2.4980	55500	0.2989
2.5205	56000	0.2913
2.5430	56500	0.2969
2.5655	57000	0.3012
2.5880	57500	0.2974
2.6105	58000	0.2937
2.6330	58500	0.2917
2.6555	59000	0.2998
2.6780	59500	0.2915
2.7005	60000	0.2924
2.7230	60500	0.2926
2.7455	61000	0.2959
2.7680	61500	0.2950
2.7905	62000	0.2950
2.8130	62500	0.2972
2.8355	63000	0.2955
2.8580	63500	0.2934
2.8805	64000	0.2962
2.9031	64500	0.2941
2.9256	65000	0.2910
2.9481	65500	0.2941
2.9706	66000	0.2893
2.9931	66500	0.2953

Training Time

• Training: 2.4 hours

Framework Versions

• Python: 3.13.11
• Sentence Transformers: 5.6.0
• Transformers: 5.14.1
• PyTorch: 2.13.0+cu130
• Accelerate: 1.14.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}