BabyLM
Collection
UT Austin's model submissions to BabyLM challenge. • 7 items • Updated • 1
# Load model directly
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("venkatasg/lil-bevo-x")
model = AutoModelForMaskedLM.from_pretrained("venkatasg/lil-bevo-x", device_map="auto")Quick Links
Lil-Bevo-X
Lil-Bevo-X is UT Austin's submission to the BabyLM challenge, specifically the strict track.
Model training regime:
- 5 epochs on MAESTRO dataset (85M non-language music tokens) combined with strict small dataset.
- 50 epochs of pretraining with sequence length of 128 on strict dataset.
- 150 epochs of pretraining with sequence length of 512 on strict dataset.
- 10 epochs of targeted MLM.
This README will be updated with more details soon.
- Downloads last month
- 5
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="venkatasg/lil-bevo-x")